Troubleshooting
Problem
This technote explains how to resolve an error,[]Importing oplog(s) from a deleted replica "myreplica.deleted" is not allowed.[], that can occur when attempting to import a IBM Rational ClearCase MultiSite (CC) sync packet.
Symptom
An attempt to import a MultiSite sync packets reports the error:
multitool: Error: Importing oplog(s) from a deleted replica "myreplica.deleted" is not allowed.
multitool: Error: Unable to replay oplog entry 2055484: error detected by ClearCase subsystem.
Cause
This error is seen due to APAR IC46845 which addresses problems caused by importing oplogs from a deleted replica. Refer to technote 1257913 Replay of oplogs from a deleted replica may cause divergence for further information about this defect.
Improperly removing a replica can cause divergence as described in the following two scenarios:
- A replica was removed with chmaster -all -obsolete_replica but the actual VOB was never taken out of production and packets continue to be sent from that replica.
- A replica was removed at another replica that was not the most up to date replica with respect to the deleted replica's oplogs. For example the deleted replica had sent an update packet to Boston but Houston removed that replica before receiving that update.
Resolving The Problem
To troubleshoot this issue gather information about the packet that is failing to import.
Set to the View in the VOB that is having trouble importing the packet and run:
multitool lspacket -long -dump <Problem Packet>
Note: Remove the -long output from the lspacket command to return a short listing of the oplogs contained in the packet. This may be useful if you want to do a quick check to see if the packet does contain any oplogs from the deleted replica. However, the -long -dump output should still be collected for further analysis.
5: op=mkelem rep=original id=5
6: op=checkout rep=original id=6
7: op=checkin rep=removed.deleted id=7
Check the following:
First
Check the lspacket output to see where the packet was created
Originating replica is: original
If the packet is directly from the deleted replica, it should not be imported under any circumstances.
Second
If it is not from the deleted replica, check the oplogs listed in the output to see if any of those are from the deleted replica.
Look at the second and third lines of each oplog to see the operation and the replica_oid
3:
op= mkreplica
replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)
Note: It is important to check for the replica_oid line directly under the op= line because the replica_oid can show up at other places in the output when it is the replica being acted upon such as the mkreplica example in this output.
3:
op= mkreplica
replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)
oplog_id= 3
op_time= 13-Jan-09.20:15:01UTC create_time= 13-Jan-09.20:15:01UTC
event comment= ""
data size= 196 data= 0xe8fa10
------------
replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)
Note: Even though this is a mkreplica operation for the removed replica it can show as .deleted because the information is being read from the local VOB's database and is not a value stored in the oplog.
If there are no oplogs created at the deleted replica, then it is most likely related to defect RATLC01015886. This was a defect which caused the error to be thrown because the packet reference a deleted replica even though there were no oplogs from that replica.
The following replicas are referenced by this packet:
original
first oplog id is 2
incarnation is 0
replica2
first oplog id is 0
incarnation is 0
removed.deleted
first oplog id is 0
incarnation is 0
If this is the case, upgrade to the patches listed in technote 1257913 or call support for assistance with importing this packet.
Third
If there are oplogs from the deleted replica in the packet, then you need to determine if it is safe to import these oplogs.
This will be a challenge since you will need to compare the operation being performed against any existing data at the importing replica and at the replica that currently masters the object being acted upon.
For example, note this oplog:
4:
op= checkout
replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)
oplog_id= 6
op_time= 13-Jan-09.21:03:35UTC create_time= 13-Jan-09.21:03:35UTC
data size= 256 data= 0x947480
------------
ckout_ver_oid= bc9330a5.feec47b1.977e.b9:df:5b:86:9e:5b
(*object not found*)
ver_fstat= ino: 0; type: 1; mode: 00
usid: NT:S-1-5-21-1614895754-879983540-682003330-7752
gsid: NT:S-1-5-21-1614895754-879983540-682003330-24329
nlink: 0; size: 0
atime: Wed Dec 31 19:00:00 1969
mtime: Tue Jan 13 16:03:35 2009
ctime: Tue Jan 13 16:03:35 2009
branch_oid= 4309a518.f4844192.b187.29:b2:8f:91:04:36
(M:\myview\das-ist-r1\hello.c@@\main)
pred_ver_oid= aee07c5b.da514b46.8c0a.33:8d:64:d8:c1:07
(M:\myview\das-ist-r1\hello.c@@\main\0)
At the importer determine if hello.c exists. If it does exist, then does it have more versions than what is contained in the packet?
The good news is that most problem operations would be prevented by the nature of MultiSite.
For example, if the packet was attempting to create a version that already existed, an "element is not the highest on the branch" error would be seen after attempting the import.
If it was applying a change to something that had been removed at the new mastering replica it would simply be ignored with a warning "skipping oplog due to missing input". This message is written to standard out and to the oplog at the importing replica.
In most cases it is safe to import these packets.
Once it is determined that the packet is safe to import contact IBM Rational Client Support for a workaround.
What if it is not safe to import the packet?
If the packet is directly from the deleted replica, the packet should be discarded.
Note: The data will be lost. It can be recovered from the deleted replica if it is still live. But it is recommended that a replica that has been deleted be removed from the production server as soon as possible.
Warning: Locking a VOB will not prevent packets from being created.
If the packet is not directly from the deleted replica and is not safe to import, this means that there is divergence between one or more replicas in the family. If this is the case, it will be necessary to replace one or more replicas.
The method for determining which replicas to replace can be tricky especially if multiple replicas are affected. The basic steps are to determine which replica or set of replicas will be the least work to preserve. This will include how many replica will need to be replaced as well as how many changes will need to be manually preserved from the replicas being replaced.
Commands such as cleartool find, lshistory and multitool dumpoplog can be used to assist in determining what data to preserve.
See the respective man pages for each of these commands in the ClearCase Information Center
The following is the full lspacket -long -dump output used for the previous examples.
3-Jan-09.15.15.14_6720
Packet type: Update
Packet fragment: 1 of 1
Packet identifier is: eb6b934c.b7db4199.8892.d1:b9:00:ca:27:ba
Packet fragment identifier is: 36a5992d.df5f4986.af15.d4:62:23:56:d4:26
VOB family identifier is: 7c44b975.f66b49f5.a17d.81:a2:94:c2:01:88
Comment supplied at packet creation is:
Packet intended for the following targets:
replica2
Originating replica is: original
The following replicas are referenced by this packet:
original
first oplog id is 2
incarnation is 0
replica2
first oplog id is 0
incarnation is 0
removed.deleted
first oplog id is 5
incarnation is 0
The major packet version is 6, the minor packet version is 0
Packet is uncompressed.
3:
op= mkreplica
replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)
oplog_id= 3
op_time= 13-Jan-09.20:15:01UTC create_time= 13-Jan-09.20:15:01UTC
event comment= ""
data size= 196 data= 0xe70008
------------
replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)
name_p= "removed"
rptype_oid= 9f8054f6.6d01458b.808b.f3:12:da:91:8c:a9 (unfiltered)
host_id_str= "myhost"
preserve= none
replica_oid= 719a4a7a.85154029.bea5.bb:b5:3b:be:58:cc (original)
oplog id= 2
replica_oid= 1b72f378.90b24fe9.868a.b8:3e:07:28:01:fa (replica2)
oplog id= 0
obj_fstat= ino: 0; type: 0; mode: 00
usid: NOBODY
gsid: NOBODY
nlink: 0; size: 0
atime: Wed Dec 31 19:00:00 1969
mtime: Wed Dec 31 19:00:00 1969
ctime: Wed Dec 31 19:00:00 1969
incarnation= 01-Jan-70.00:00:00UTC
4:
op= checkout
replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)
oplog_id= 6
op_time= 13-Jan-09.21:03:35UTC create_time= 13-Jan-09.21:03:35UTC
data size= 256 data= 0x947480
------------
ckout_ver_oid= bc9330a5.feec47b1.977e.b9:df:5b:86:9e:5b
(*object not found*)
ver_fstat= ino: 0; type: 1; mode: 00
usid: NT:S-1-5-21-1614895754-879983540-682003330-7752
gsid: NT:S-1-5-21-1614895754-879983540-682003330-24329
nlink: 0; size: 0
atime: Wed Dec 31 19:00:00 1969
mtime: Tue Jan 13 16:03:35 2009
ctime: Tue Jan 13 16:03:35 2009
branch_oid= 4309a518.f4844192.b187.29:b2:8f:91:04:36
(M:\myview\das-ist-r1\hello.c@@\main)
pred_ver_oid= aee07c5b.da514b46.8c0a.33:8d:64:d8:c1:07
(M:\myview\das-ist-r1\hello.c@@\main\0)
pred_ver_num= 0
view_hostname_p= "myhost"
view_host_pname_p= "C:\ccstore\views\myview.vws"
view_uuid= 3a156be6.56c8451d.a71e.a4:f0:64:bf:cd:22
reserve= TRUE
not_master_ok= FALSE
5:
op= checkin
replica_oid= 637e23bd.46074188.b500.5f:cd:30:fe:50:43 (removed.deleted)
oplog_id= 7
op_time= 13-Jan-09.21:03:53UTC create_time= 13-Jan-09.21:03:53UTC
version_oid= 74bd02e3.78ea43fe.85d3.2b:8e:6f:e9:ee:58
(M:\myview\das-ist-r1\hello.c@@\main\1)
event comment= ""
data size= 116 data= 0xe89fd8
------------
ver_oid= 74bd02e3.78ea43fe.85d3.2b:8e:6f:e9:ee:58
(M:\myview\das-ist-r1\hello.c@@\main\1)
ver_num= 1
ver_fstat= ino: 0; type: 1; mode: 02
usid: DONTCARE
gsid: DONTCARE
nlink: 0; size: 8
atime: Wed Dec 31 19:00:00 1969
mtime: Tue Jan 13 16:03:52 2009
ctime: Tue Jan 13 16:03:52 2009
ckout_ver_oid= bc9330a5.feec47b1.977e.b9:df:5b:86:9e:5b
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21366521