IBM Support

multitool: Error: Importing oplog(s) from a deleted replica "myreplica.deleted" is not allowed

Troubleshooting


Problem

This technote explains how to resolve an error,[]Importing oplog(s) from a deleted replica "myreplica.deleted" is not allowed.[], that can occur when attempting to import a IBM Rational ClearCase MultiSite (CC) sync packet.

Symptom

An attempt to import a MultiSite sync packets reports the error:


multitool: Error: Importing oplog(s) from a deleted replica "myreplica.deleted" is not allowed.
multitool: Error: Unable to replay oplog entry 2055484: error detected by ClearCase subsystem.

Cause


This error is seen due to APAR IC46845 which addresses problems caused by importing oplogs from a deleted replica. Refer to technote 1257913 Replay of oplogs from a deleted replica may cause divergence for further information about this defect.

Improperly removing a replica can cause divergence as described in the following two scenarios:

  • A replica was removed with chmaster -all -obsolete_replica but the actual VOB was never taken out of production and packets continue to be sent from that replica.
  • A replica was removed at another replica that was not the most up to date replica with respect to the deleted replica's oplogs. For example the deleted replica had sent an update packet to Boston but Houston removed that replica before receiving that update.
If these packets are sent to another replica in the family, it is possible to introduce divergence.

Resolving The Problem



    To troubleshoot this issue gather information about the packet that is failing to import.

    Set to the View in the VOB that is having trouble importing the packet and run:

    multitool lspacket -long -dump <Problem Packet>


    Note: Remove the -long output from the lspacket command to return a short listing of the oplogs contained in the packet. This may be useful if you want to do a quick check to see if the packet does contain any oplogs from the deleted replica. However, the -long -dump output should still be collected for further analysis.

    5: op=mkelem rep=original id=5

    6: op=checkout rep=original id=6

    7: op=checkin rep=removed.deleted id=7




    Check the following:

    First
    Check the lspacket output to see where the packet was created

    Originating replica is: original


    If the packet is directly from the deleted replica, it should not be imported under any circumstances.


    Second

    If it is not from the deleted replica, check the oplogs listed in the output to see if any of those are from the deleted replica.

    Look at the second and third lines of each oplog to see the operation and the replica_oid

    3:

    op= mkreplica

    replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)


    Note: It is important to check for the replica_oid line directly under the op= line because the replica_oid can show up at other places in the output when it is the replica being acted upon such as the mkreplica example in this output.

    3:

    op= mkreplica

    replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)

    oplog_id= 3

    op_time= 13-Jan-09.20:15:01UTC  create_time= 13-Jan-09.20:15:01UTC

    event comment= ""

    data size= 196  data= 0xe8fa10

    ------------

    replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)


    Note: Even though this is a mkreplica operation for the removed replica it can show as .deleted because the information is being read from the local VOB's database and is not a value stored in the oplog.


    If there are no oplogs created at the deleted replica, then it is most likely related to defect RATLC01015886. This was a defect which caused the error to be thrown because the packet reference a deleted replica even though there were no oplogs from that replica.

    The following replicas are referenced by this packet:

      original

        first oplog id is 2

        incarnation is 0

      replica2

        first oplog id is 0

        incarnation is 0

      removed.deleted

        first oplog id is 0

        incarnation is 0


    If this is the case, upgrade to the patches listed in technote 1257913 or call support for assistance with importing this packet.


    Third

    If there are oplogs from the deleted replica in the packet, then you need to determine if it is safe to import these oplogs.

    This will be a challenge since you will need to compare the operation being performed against any existing data at the importing replica and at the replica that currently masters the object being acted upon.

    For example, note this oplog:

    4:

    op= checkout

    replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)

    oplog_id= 6

    op_time= 13-Jan-09.21:03:35UTC  create_time= 13-Jan-09.21:03:35UTC

    data size= 256  data= 0x947480

    ------------

    ckout_ver_oid= bc9330a5.feec47b1.977e.b9:df:5b:86:9e:5b

                   (*object not found*)

    ver_fstat= ino: 0; type: 1; mode: 00

               usid: NT:S-1-5-21-1614895754-879983540-682003330-7752

               gsid: NT:S-1-5-21-1614895754-879983540-682003330-24329

               nlink: 0; size: 0

               atime: Wed Dec 31 19:00:00 1969

               mtime: Tue Jan 13 16:03:35 2009

               ctime: Tue Jan 13 16:03:35 2009

    branch_oid= 4309a518.f4844192.b187.29:b2:8f:91:04:36

                (M:\myview\das-ist-r1\hello.c@@\main)

    pred_ver_oid= aee07c5b.da514b46.8c0a.33:8d:64:d8:c1:07

                  (M:\myview\das-ist-r1\hello.c@@\main\0)


    At the importer determine if hello.c exists. If it does exist, then does it have more versions than what is contained in the packet?

    The good news is that most problem operations would be prevented by the nature of MultiSite.
    For example, if the packet was attempting to create a version that already existed, an "element is not the highest on the branch" error would be seen after attempting the import.
    If it was applying a change to something that had been removed at the new mastering replica it would simply be ignored with a warning "skipping oplog due to missing input". This message is written to standard out and to the oplog at the importing replica.

    In most cases it is safe to import these packets.

    Once it is determined that the packet is safe to import contact IBM Rational Client Support for a workaround.


    What if it is not safe to import the packet?

    If the packet is directly from the deleted replica, the packet should be discarded.

    Note: The data will be lost. It can be recovered from the deleted replica if it is still live. But it is recommended that a replica that has been deleted be removed from the production server as soon as possible.
    Warning: Locking a VOB will not prevent packets from being created.

    If the packet is not directly from the deleted replica and is not safe to import, this means that there is divergence between one or more replicas in the family. If this is the case, it will be necessary to replace one or more replicas.
    The method for determining which replicas to replace can be tricky especially if multiple replicas are affected. The basic steps are to determine which replica or set of replicas will be the least work to preserve. This will include how many replica will need to be replaced as well as how many changes will need to be manually preserved from the replicas being replaced.

    Commands such as cleartool find, lshistory and multitool dumpoplog can be used to assist in determining what data to preserve.

    See the respective man pages for each of these commands in the ClearCase Information Center





    The following is the full lspacket -long -dump output used for the previous examples.

    3-Jan-09.15.15.14_6720

    Packet type: Update

    Packet fragment: 1 of 1

    Packet identifier is: eb6b934c.b7db4199.8892.d1:b9:00:ca:27:ba

    Packet fragment identifier is: 36a5992d.df5f4986.af15.d4:62:23:56:d4:26

    VOB family identifier is: 7c44b975.f66b49f5.a17d.81:a2:94:c2:01:88

    Comment supplied at packet creation is:

    Packet intended for the following targets:

    replica2

    Originating replica is: original

    The following replicas are referenced by this packet:

    original

    first oplog id is 2

    incarnation is 0

    replica2

    first oplog id is 0

    incarnation is 0

    removed.deleted

    first oplog id is 5

    incarnation is 0

    The major packet version is 6, the minor packet version is 0

    Packet is uncompressed.

    3:

    op= mkreplica

    replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)

    oplog_id= 3

    op_time= 13-Jan-09.20:15:01UTC create_time= 13-Jan-09.20:15:01UTC

    event comment= ""

    data size= 196 data= 0xe70008

    ------------

    replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)

    name_p= "removed"

    rptype_oid= 9f8054f6.6d01458b.808b.f3:12:da:91:8c:a9 (unfiltered)

    host_id_str= "myhost"

    preserve= none

    replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)

    oplog id= 2

    replica_oid= 1b72f378.90b24fe9.868a.b8:3e:07:28:01:fa (replica2)

    oplog id= 0

    obj_fstat= ino: 0; type: 0; mode: 00

    usid: NOBODY

    gsid: NOBODY

    nlink: 0; size: 0

    atime: Wed Dec 31 19:00:00 1969

    mtime: Wed Dec 31 19:00:00 1969

    ctime: Wed Dec 31 19:00:00 1969

    incarnation= 01-Jan-70.00:00:00UTC

    4:

    op= checkout

    replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)

    oplog_id= 6

    op_time= 13-Jan-09.21:03:35UTC create_time= 13-Jan-09.21:03:35UTC

    data size= 256 data= 0x947480

    ------------

    ckout_ver_oid= bc9330a5.feec47b1.977e.b9:df:5b:86:9e:5b

    (*object not found*)

    ver_fstat= ino: 0; type: 1; mode: 00

    usid: NT:S-1-5-21-1614895754-879983540-682003330-7752

    gsid: NT:S-1-5-21-1614895754-879983540-682003330-24329

    nlink: 0; size: 0

    atime: Wed Dec 31 19:00:00 1969

    mtime: Tue Jan 13 16:03:35 2009

    ctime: Tue Jan 13 16:03:35 2009

    branch_oid= 4309a518.f4844192.b187.29:b2:8f:91:04:36

    (M:\myview\das-ist-r1\hello.c@@\main)

    pred_ver_oid= aee07c5b.da514b46.8c0a.33:8d:64:d8:c1:07

    (M:\myview\das-ist-r1\hello.c@@\main\0)

    pred_ver_num= 0

    view_hostname_p= "myhost"

    view_host_pname_p= "C:\ccstore\views\myview.vws"

    view_uuid= 3a156be6.56c8451d.a71e.a4:f0:64:bf:cd:22

    reserve= TRUE

    not_master_ok= FALSE

    5:

    op= checkin

    replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)

    oplog_id= 7

    op_time= 13-Jan-09.21:03:53UTC create_time= 13-Jan-09.21:03:53UTC

    version_oid= 74bd02e3.78ea43fe.85d3.2b:8e:6f:e9:ee:58

    (M:\myview\das-ist-r1\hello.c@@\main\1)

    event comment= ""

    data size= 116 data= 0xe89fd8

    ------------

    ver_oid= 74bd02e3.78ea43fe.85d3.2b:8e:6f:e9:ee:58

    (M:\myview\das-ist-r1\hello.c@@\main\1)

    ver_num= 1

    ver_fstat= ino: 0; type: 1; mode: 02

    usid: DONTCARE

    gsid: DONTCARE

    nlink: 0; size: 8

    atime: Wed Dec 31 19:00:00 1969

    mtime: Tue Jan 13 16:03:52 2009

    ctime: Tue Jan 13 16:03:52 2009

    ckout_ver_oid= bc9330a5.feec47b1.977e.b9:df:5b:86:9e:5b

[{"Product":{"code":"SSSH27","label":"Rational ClearCase"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Oplogs","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"2003.06.16;7.0;7.0.0.1;7.0.0.2;7.0.0.3;7.0.0.4;7.0.1;7.0.1.1;7.0.1.2;7.0.1.3;7.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
16 June 2018

UID

swg21366521