AWS CLI has become a life saver when you want to manage your AWS infrastructure efficiently.

This also provides lot of possibilities for Automation and reduce the number of times that you have to login to AWS Direction console.

If you lot are new to AWS CLI. Please consider reading this commodity on how to setup AWS CLI

With no further ado, Lets become into the objective of this mail.

aws s3 cp

In this post we are going to talk about a very specific command of AWS CLI which is AWS S3

While AWS S3 provides consummate tool set to manage your S3 bucket. we are going to come across one specific feature of S3 CLI today. which is copy

Before we go there. we need to understand few things about AWS S3 CLI

At that place are 2 AWS S3 CLI commands available

  • AWS S3
  • AWS S3API

Let united states of america run into what they both accept to offer.

What is AWS S3 and S3API

The AWS CLI provides 2 tiers of commands for accessing Amazon S3

The s3 tier consists of high-level commands that simplify performing mutual tasks, such as creating, manipulating, and deleting objects and buckets.

The s3api tier behaves identically to the aforementioned S3 tier but it enables you to carry out advanced operations that might not be possible with s3 tier.

In this article we are going to talk about only the s3 tier and very specifically s3 cp command which helps usa copying files from and to S3 buckets.

Before going any further I desire you to know few handy commands which help to listing the buckets.

  1. aws s3 help – To go a listing of all of the commands available in high-level commands.
  2. aws s3 ls – To get the list of all buckets.
  3. aws s3 ls s3://bucket-proper name – Will listing all the objects and folders I that bucket.
  4. aws s3 ls s3://bucket-name/path/ – This command will filter the output to a specific prefix.

Quick Caveats on AWS S3 CP command

  • Copying a file from S3 saucepan to local is considered or chosen equally download
  • Copying a file from Local organization to S3 bucket is considered or called as upload
  • Please be warned that failed uploads can't be resumed
  • If the multipart upload fails due to a timeout or is manually cancelled by pressing CTRL + C, the AWS CLI cleans up any files created and aborts the upload. This process can take several minutes.
  • If the procedure is interrupted past a kill control or organisation failure, the in-progress multipart upload remains in Amazon S3 and must exist cleaned upwardly manually in the AWS Management Panel or with the s3api abort-multipart-upload command.

AWS S3 CP Command examples

Here nosotros accept listed few examples on how to use AWS S3 CP command to copy files.

Copying a local file to S3

Uploading a file to S3, in other words copying a file from your local file system to S3, is washed with aws s3 cp command

Permit'south suppose that your file name is file.txt  and this is how you tin upload your file to S3

aws s3 cp file.txt s3://bucket-proper name

while executed the output of that command would like something like this.

aws s3 cp

Copying a local file to S3 with Storage Class

S3 Provides various types of Storage classes to optimize the cost and to manage the disk efficiency and IO performance during file read and write operations.

  • S3 Standard
  • S3 Intelligent-Tiering
  • S3 Standard-IA
  • S3 One Zone-IA
  • S3 Glacier
  • S3 Glacier Deep Archive

Y'all can read more than information about all of them here

aws s3 cp file.txt s3://bucket-name --storage-class course-name

aws s3 cp

Panel Output:

In the prceding snapshot you can see that the test2.txt file which nosotros accept uploaded merely now is showing the Standard-IA every bit the storage class

Copying an S3 object from one bucket to another

At times we would want to copy the content of one S3 saucepan to another S3 bucket and this is how it can be done with AWS S3 CLI.

aws s3 cp s3://source-bucket-name/file.txt s3://destination-bucket-name/

aws s3 cp

How to Recursively upload or download (copy) files with AWS S3 CP control

When passed with the parameter --recursive  the aws s3 cp command recursively copies all objects from source to destination.

It can be used to download and upload big set of files from and to S3.

Here is the AWS CLI S3 control to Download list of files recursively from S3.  hither the dot . at the destination end represents the current directory

aws s3 cp s3://bucket-proper noun . --recursive

aws s3 cp

the same command tin can be used to upload a big set of files to S3. by just changing the source and destination

aws s3 cp . s3://bucket-proper name  --recursive

Here nosotros have just changed source to electric current directory and destination to the bucket and now all the files on the current directory(local) would exist uploaded to the bucket.

It includes all the subdirectories and hidden files

Setting the Access Control List (ACL) while copying an S3 object

For security and compliance reasons, we might want to restrict the files that we are copying to S3 buckets with prepare of Access control. Like Read only or Read Write etc.

Here is the command that copies a file from one bucket to some other bucket with specific ACL practical on the destination bucket ( after copied )

aws s3 cp s3://source-bucket-proper name/file.txt s3://dest-saucepan-name/ --acl public-read-write

At that place are 5 types of ACL permissions available with S3 which are listed here on the following snapshot.

aws s3 cp
Image credits to AWS

Promise this helps.

How to copy files between EC2 and S3

We take so far discussed how to copy files from local to EC2 and vice versa.

When it comes to EC2 instances it is a footling unlike equally the authentication can exist taken intendance of by IAM roles assigned to the server.

S3 buckets can likewise have certain saucepan policies to let servers coming from a certain AWS account of VPC endpoint etc.

We have created an article that talks about how to easily re-create files between EC2 and S3 with a uncomplicated iv step configuration.

Re-create files from EC2 to S3 Saucepan in 4 steps

Automate S3 tasks with Ansible

We take an exclusive article covering various examples of Ansible S3 module usage and examples.

If you are looking for some automation with S3. I would recommend you to give it a try.

How to use ansible with S3 – Ansible aws_s3 examples | Devops Junction

If yous accept any feedback or all-time practices. please feel complimentary to annotate and permit us know.

Cheers
Hanumanth

gritfy