Summary
A multipart upload requires the use of OCI CLI.
Reference
Overview
An object, of size greater than 100MB, should be uploaded using multi-part upload. This will upload the data in parallel, and is faster and more efficient. Only failing parts need to be retried rather than the entire upload. Uploads can be paused and resumed later.
An object can be divided into a maximum of 10,000 parts each with a maximum size of 50GB. Maximum object size is 0.5PB.
Can create a lifecycle rule for multi-part uploads to automatically delete uncommitted multipart uploads after a specified number of days.
Using API
There are four (4) steps:
- Split the object into parts
- Initiate upload with Create Multipart Upload call and get ID
- With that ID, start upload of a part with an UploadPart call. One (1) call per part.
- When all parts have finished uploading, commit the upload with the Commit Multipart Upload call. Only then does Object Storage reconstruct the objects from the uploaded parts.
Lab
This topic is covered by Lab 11-1: Object Storage: Perform Multipart Upload Using CLI (Using Cloud Shell):
Overview
The Oracle Cloud Infrastructure (OCI) Object Storage supports multipart uploads for more efficient and resilent uploads, especially for large objects. With multipart uploads, the individual parts of an object can be uploaded in parallel to reduce the amount of time you spend uploading. In this lab, you will perform a multipart upload on the Command Line Interface (CLI) using Cloud Shell.
In this lab, you will:
- Access Cloud Shell via the Console
- Create a standard default storage tier bucket using CLI (cloud Shell)
- Upload a file (larger than 100 MiB) to Cloud Shell
- Perform a multipart upload using the CLI (Cloud Shell)
Create a Standard default storage tier bucket using CLI (Cloud Shell) and Upload a File
Perform a multipart upload using CLI (Cloud Shell)
Create a Standard Bucket
The OCI CLI commands used to create a standard default storage tier bucket are:
bucket_name='FRA-AA-LAB11-1-BKT-01'
name_space=$( \
oci os ns get \
--query 'data' \
--raw-output \
)
comp_id=$( \
oci iam compartment list \
--name 'Sandbox' \
--query 'data[0].id' \
--raw-output \
)
bucket_id=$( \
oci os bucket create \
--namespace ${name_space} \
--name ${bucket_name} \
--compartment-id ${comp_id} \
--query 'data.id' \
--raw-output \
)
To get the details of the created bucket, use the following command:
oci os bucket get --name 'FRA-AA-LAB11-1-BKT-01'
The sample output is:
{
"data": {
"approximate-count": null,
"approximate-size": null,
"auto-tiering": null,
"compartment-id": "ocid1.compartment.oc1..aaaaaaaamoo6uz2qmix2adls2cgoqxxhdt4wuam3wbcrw6co6z4osweos6da",
"created-by": "ocid1.user.oc1..aaaaaaaab6dmoec6utwsmvueyko32h6vvhmptr3yeaunitxj6733jpo6hsca",
"defined-tags": {
"Oracle-Tags": {
"CreatedBy": "default/tenancy_admin",
"CreatedOn": "2024-08-03T17:36:07.615Z"
}
},
"etag": "b02f41d0-644b-4c0f-8329-8c8b0f5c230d",
"freeform-tags": {},
"id": "ocid1.bucket.oc1.ap-sydney-1.aaaaaaaaz4b7hgfkmiqk6ovgpbhvmr746j4xdw3nw2wyexn4rsyorc4m5pdq",
"is-read-only": false,
"kms-key-id": null,
"metadata": {},
"name": "FRA-AA-LAB11-1-BKT-01",
"namespace": "sdorfvwhnhvj",
"object-events-enabled": false,
"object-lifecycle-policy-etag": null,
"public-access-type": "NoPublicAccess",
"replication-enabled": false,
"storage-tier": "Standard",
"time-created": "2024-08-03T17:36:07.627000+00:00",
"versioning": "Disabled"
},
"etag": "b02f41d0-644b-4c0f-8329-8c8b0f5c230d"
}
Perform a Multipart Upload
Use the following commands to find a large file to upload:
upload_file=$( \
ls -S ~/Downloads | \
head -n 1 \
)
ls -lh ~/Downloads/${upload_file}
The sample output is:
-rw-rw-r-- 1 douglas douglas 5.7G May 3 17:08 /home/douglas/Downloads/Oracle_Database_23ai_Free_Developer.ova
Use the following OCI CLI commands to perform the multipart upload:
oci os object put \
-bn ${bucket_name} \
--namespace ${name_space} \
--file ~/Downloads/${upload_file} \
--name FRA-AA-LAB11-1-OBJ-01 \
--part-size 20 \
--parallel-upload-count 5
After upload fails, the sample output is:
Upload ID: c8eed26d-e525-59ca-b95a-467dc08d02c2
Split file into 291 parts for upload.
Uploading object [##################################--] 95% 00:01:35
ServiceError:
{
"client_version": "Oracle-PythonSDK/2.126.4, Oracle-PythonCLI/3.41.0",
"code": "NotAuthenticated",
"logging_tips": "Please run the OCI CLI command using --debug flag to find more debug information.",
"message": "The required information to complete authentication was not provided.",
"opc-request-id": "syd-1:Kf3ngu83TrRYAqpvMO_VkEPVO3FwZt53u1ZJPtZhgzhHX8UcrbOsZ65FOakshXzD",
"operation_name": "upload_part",
"request_endpoint": "PUT https://objectstorage.ap-sydney-1.oraclecloud.com/n/sdorfvwhnhvj/b/FRA-AA-LAB11-1-BKT-01/u/FRA-AA-LAB11-1-OBJ-01",
"status": 401,
"target_service": "object_storage",
"timestamp": "2024-08-03T18:59:25.436307+00:00",
"troubleshooting_tips": "See [https://docs.oracle.com/iaas/Content/API/References/apierrors.htm] for more information about resolving this error. If you are unable to resolve this issue, run this CLI command with --debug option and contact Oracle support and provide them the full error message."
}
The Cloud Console shows the bucket information:
Abort Upload
I used the following commands to abort the above failed multipart upload:
upload_id=$( \
oci os multipart list \
--bucket-name "${bucket_name}" \
--query 'data[0]."upload-id"' \
--raw-output \
)
oci os multipart abort \
-bn "${bucket_name}" \
-on "FRA-AA-LAB11-1-OBJ-01" \
--upload-id ${upload_id} \
--force
Second Attempt
The second attempt succeeded with the following output:
Upload ID: 053df79d-85d4-dfe5-5eee-93c99f2aacf8
Split file into 291 parts for upload.
Uploading object [####################################] 100%
{
"etag": "54e58d4f-4e67-4212-bef2-b7486633a836",
"last-modified": "Sun, 04 Aug 2024 17:28:01 GMT",
"opc-multipart-md5": "yaAdBJn9suPt2BTxJaRs0Q==-291"
}
This is verified by using the following command:
oci os object list \
-bn FRA-AA-LAB11-1-BKT-01 \
--all \
--query 'data[*].{"Object Name":name,"Size":size,"Storage Tier":"storage-tier"}' \
--output table
The output is:
+-----------------------+------------+--------------+ | Object Name | Size | Storage Tier | +-----------------------+------------+--------------+ | FRA-AA-LAB11-1-OBJ-01 | 6083107840 | Standard | +-----------------------+------------+--------------+