Troubleshooting
Problem
There are many reasons that Transfers to the cloud using CPYTOCLD or from the cloud using CPYFRMCLD may fail. These also apply to transfers initiated by BRMS functions. Customers should know where to find most of the information in the cloud, this document will show examples for COS, for other cloud providers, contact the cloud provider.
This document can refer to resource type of AWSS3. AWSS3 resource type is used for COS, Google, Amazon, so don't assume that because the type is AWSS3 that it is amazon. To view or change the ICC resources, use the WRKCFGICC command. There is no OUTPUT(*PRINT), so ask for screen captures when needed.
This document is geared toward transfers to an actual cloud service, not FTP. Some of the errors could apply to FTP as well.
Helpful Documentation:
For many of these issues, there might be helpful information in the Cloud Storage Solutions for i Flight Recorders and communication traces.
For many of these issues, there might be helpful information in the Cloud Storage Solutions for i Flight Recorders and communication traces.
Here is how to collect the Cloud Storage Solutions for i Flight recorders:
https://www.ibm.com/support/pages/node/687525
https://www.ibm.com/support/pages/node/687525
Here is how to collect communication traces:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(TRCCNNIP) SIZE(998000) TCPDTA(*N () () *N 'xxx.xxx.xxx.xxx')
Where xxx.xxx.xxx.xxx is the IP address cloud endpoint (or the reverse proxy IP if you are using one)
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(TRCCNN2) SIZE(998000)
Re-create the failure.
SBMJOB CMD(TRCCNN SET(*OFF) TRCTBL(TRCCNNIP) OUTPUT(*STMF)
TOSTMF('/trccnnip.pcap' *YES)) JOB(TRCCNNIP) JOBQ(QSYSNOMAX)
SBMJOB CMD(TRCCNN SET(*OFF) TRCTBL(TRCCNN2) OUTPUT(*STMF)
TOSTMF('/trccnn2.pcap' *YES)) JOB(TRCCNN2) JOBQ(QSYSNOMAX)
Send in the /trccnnip and /trccnn2 files
Incorrect resource URI:
Specifying the incorrect resource URI can cause a variety of errors. The correct resource URI depends on the cloud provider.
- IBM CLoud - use a the full path. some examples:
Public endpoint (no proxy server required)
s3.us-east.cloud-object-storage.appdomain.cloud
Private endpoint (usually requires properly configured proxy server):
s3.private.us-east.cloud-object-storage.appdomain.cloud - Amazon Cloud:
Resource URI is always s3.amazonaws.com. Do not add anything about the region to the beginning. - Google Cloud:
Resource URI is always storage.googleapis.com. Do not add anything about the region to the beginning. - No other cloud providers are currently supported.
- If using Cloud Storage Solutions Encryption (configured by using the Cloud Storage Solutions Encryption section of the ICC users guide), for all cloud providers, add https:// to the front of the resource URI
Proxy Server:
If using a proxy server between the IBM i and the cloud, see this document:
Using Proxy Server
Using Proxy Server
ICC0118 HTTP Error code 403 - Forbidden:
403 - Forbidden means the cloud has rejected the transfer, this can be for multiple reasons, and there is no way from the IBM i to distinguish which reason the cloud rejected the transfer (not even from the flight recorder), so all will need to be checked. The known reasons are:
1) The most common reason is that the GMT time calculated from the current system time must match the GMT time of the cloud. If the system time or time zone is not correct (within about 10 or 15 minutes), the cloud will not accept the transfer.
For example, US Central is 6 hours behind GMT during regular time (winter months). If the system is in central time zone, rhw QTIMZON system value should be QN0600CST2. Any other QTIMEZON setting may not work for transferring files to the cloud. With these settings, say the system time is 10:00, and it is in the winter months (so the offset is -6). When 6 is added to GMT time, that is 16:00, which matches with the GMT time of the cloud (also 16:00), so the transfer is allowed. If the wrong time zone set, say QN0500EST2, that is an offset of -5, if the current system time is 10:00, add 5 for the offset, now the file being sent to the cloud that has a timestamp of 15:00, however, the current time in the cloud is GMT time 1600, so the transfer fails with the ICC0118 error code 403 Forbidden.
To change the time zone, use the WRKTIMZON command and pick the correct one. There are multiple choices for each time zone, look at the details and make sure you pick the correct one (correct daylight savings dates, etc). For the example above, pick QN0600CST2. Note, when the time zone is changed, it automatically changes both the QTIME, so QTIME would need to be updated to the correct time. For additional help with time zones, check with the work management team.
It will also fail if the time zone is correct, but the system time is off by 10 minutes or more. If the time is off, change the QTIME system value.
2) Incorrect Access Key ID or Secret Access key used in AWSS3 resource.
The Access Key ID and Secret Access Key comes from the cloud provider. For COS, it can be found under Service Credentials:
The Access Key ID and Secret Access Key comes from the cloud provider. For COS, it can be found under Service Credentials:
Customers generally know how to find this information. If they do not know how to find it, they should ask their cloud provider be it COS, Google, or Amazon, or do an internet search for help on how to find the information.
If COS credentials do not list an acces_key_id or secret_access_key, they need to click New Credential, then advanced options, and turn on 'Include HMAC Credential':
Run command WRKCFGICC and take option 2 to re-enter both keys. If using cut or paste for the keys, remove any blanks that get pasted in.
If items 1 & 2 above are not the problem, collect the ICC Flight recorders (see helpful documentation section), if in the flight recorder you find something like this:
|403 Forbidden..X-Clv-Request-Id:|
| 27dc30ed-779f-4179-abae-de7eadf|
|403 Forbidden..X-Clv-Request-Id:|
| 27dc30ed-779f-4179-abae-de7eadf|
The X-CLV-Request-ID indicates the rejection is coming from IBM cloud.
Other cloud providers will likely have different text. The 403 can also come from other things such as network or firewall that will need to be investigated by customers network team. In some cases Communications traces (see helpful documentation section) may help identify where the 403 is coming from. It may be helpful to contact the cloud provider (IBM or other) to find out if they are rejecting the transfer, if they are not seeing the transfer come in, that indicates a network or firewall is rejecting the transfer.
ICC0118 HTTP Error code 404:
This indicates that the Bucket specified in the AWSS3 resource does not match the bucket on the cloud. For COS, the bucket names can be found here:
ICC0118 HTTP Error code 301
Deleting and re-creating the resource may fix this problem.
ICC0069 - Open Server Connection failed Error code 4:
This indicates that the Resource URI in the AWSS3 resource is not correct or is not pingable on the system. The resource URI (portion past the // if there is one) needs to be pingable. For example:
PING RMTSYS('s3.us-east.cloud-object-storage.appdomain.cloud')
PING RMTSYS('s3.us-east.cloud-object-storage.appdomain.cloud')
ICC0069 - Open Server Connection failed Error code 5:
This could also be because the customer is trying to use the private endpoint or direct endpoint. Endpoints can be found here:
Then scroll down to:
Then scroll down to:
Generally, customers should be using the public endpoint. The private and direct endpoints would require a proxy which some customers might be using, such as nginx open source proxy server, which is not supported by IGSC and is outside the scope of this document.
ICC0069 - Open server connection failed Error Code 7 on CPYFRMCLD after transfer fails part way through.
Flight recorders show many 'Select timeout on recv' errors. This is only when copying data from the cloud. Possible solution is CHGTCPA TCPRCVBUF(524288) TCPSNDBUF(524288). As an added benefit, CPYFRMCLD may perform considerably faster with the new buffer sizes.
Another reason this can occur is bad DASD write performance against the load source DASD unit. When a virtual server instance is provisioned in PowerVS the load source is 100 GB by default. After the initial deployment the load source should be changed to match the size of the other DASD units. If the load source is already very full user data can be drained off of the load source and allocation can be ended on the load source if necessary (See : Operating System Disk Balancing Support).
CPDBC99 - Application Identifier is not registered to use SSL
SSL not configured. See section titled Configuring Cloud Storage Solutions file transfer encryption in the users guide:
ICC0116 - Internal S3 transfer error. RC 519 - invalid part error
This is specific to AWSS3 resource types. This indicates that file being transferred is larger than the maximum size allowed. The original limit was 50gb, with PTF Si73401, the limit was increased to 100gb. In a future release IBM expects to allow larger file sizes.
CPF8361 RC95 and CPF432A RC9 - Commitment control errors
User did not have sufficient authority. They used QSECOFR, and the transfer completed. Try using QSECOFR and if the transfer completes, verify the other users have adequate authority.
ICC0114 Error reported by AWS S3 reason code 556
This was reported for customer using a proxy to get to the COS private endpoint. The nginx proxy config was incorrect, it was listening on port 80, but using https (instead of http) in the proxy_pass. See document Configuring Cloud Storage for i to use a proxy server for details.
ICC0114 Error reported by AWS S3 reason code 519
This may be an indication that part size is too small. The common mistake is to use a part size of 10,000,000 when the volume size set in BRMS is 100gb or larger. There are a max of 10,000 parts, multiply 10,000 * 10,000,000, which is 100,000,000,000 bytes or about 93gb. Part size is set using command CHGDTAARA QICC/QICCPRTSZ VALUE('100000000'). PTF SI76887 for product 5733ICC is required in order to set the part size.
ICC0114 Error reported by AWS S3 reason code 531
This may be due to attempting to copy a 0-byte file to cloud storage provider other than FTP. Transferring a 0-byte file to cloud storage other than FTP is currently a limitation. If the ICC flight recorders are enabled at the top you will see something like this:
---STARTING COPY TO CLOUD---
Local filename:
Path: /Directory/Filename.FileExtension
Cloud filename:
Path: Filename.FileExtension
validateLocalFile: size 0, ccsid xxxx
makeS3Client: Connect-to server: xxxx.yyyyy.com
Local filename:
Path: /Directory/Filename.FileExtension
Cloud filename:
Path: Filename.FileExtension
validateLocalFile: size 0, ccsid xxxx
makeS3Client: Connect-to server: xxxx.yyyyy.com
ICC0114 Error reported by AWS S3 reason code 531 & 560
This may be an indication that the proxy server does not have enough resources, and additional CPU may be required.
Transfers to IBM Spectrum Protect failing with ICC0114 Reason Code 560 and ICC0118 RC 301
IBM Spectrum Protect may be out of storage.
CPF6831 - Cannot place resource under commitment control reason code 95
Transfers to IBM Spectrum Protect failing with ICC0114 Reason Code 560 and ICC0118 RC 301
IBM Spectrum Protect may be out of storage.
CPF6831 - Cannot place resource under commitment control reason code 95
This is likely due to an incorrect or missing host table entry. To resolve:
1) EDTF '/tmp/test.txt'
add some text then press F3 to save and exit
2) CPYTOCLD RESOURCE(<Your resource>) ASYNC(*NO) LOCALFILE('/tmp/test.txt') CLOUDFILE(test.txt)
Presuming this fails, do the following:
3) STRDBG UPDPROD(*YES)
4) CPYTOCLD RESOURCE(<Your resource>) ASYNC(*NO) LOCALFILE('/tmp/test.txt') CLOUDFILE(test.txt)
5) in the joblog there should be a CPDBCC5 The name does not resolve for the supplied parameters message.
6) Do a WRKPRB, there should be a new problem log entry, do a 6 to print the details.
7) Look at the QSZPRTD spool file generated. At the bottom it lists the spool files associated with this problem log entry.
8) Do a WRKJOB filling in the information from the QPSRVDMP spool, take option 4 then option 5 to view QPSRVDMP.
9) In the eye catcher on the right side of the dump, look for the following or similar:
* Host name: xxxxx.YYYYY; *
* Socket rc: 6; *
* Error text: The name *
*does not resolve for the supplie*
*d parameters.; *
10) There needs to be a host table entry for the host name listed above. Add it using CFGTCP option 10
ICC0110 Directory or folder not found
This has been seen when the root directory specified in a FTP resource is not found on the FTP server. Make sure the directory exists, and that the whole path to the directory including the leading slash is used.
Failed CCSID conversion error found in flight recorder
Failed CCSID conversion error found in flight recorder
This is likely not an issue, look for previous errors in the joblog and flight recorder to determine why the transfer failed..
BRMS does not transfer the last (or only) volume in a cloud backup to the cloud
This can happen if the end of media option in the BRMS control group or backup policy is set to *LEAVE. When the end of media option is *LEAVE, it is an indication that more data is going to be saved to the volume, so it does not transfer it to the cloud. Check the end of media option using WRKCTLGBRM option 8, if it says *LEAVE, change it to *UNLOAD, if it says *BKUPCY, either change it to *UNLOAD here, or use WRKPCYBRM *BKU option 1 to change it to *UNLOAD. Note changing the backup policy will impact any control group that had end of media option *BKUPCY.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.2.0"}]
Was this topic helpful?
Document Information
Modified date:
05 March 2024
UID
ibm16416171