How to Install s3cmd to manage AWS S3 Bucket
s3cmd is a command line utility to manage s3 buckets on amazon web services.S3 is Simple Storage Service in AWS. It is used in many cases, for example saving logs to s3 via logrotate, creating s3 buckets,upload, download, and manage directory and file with s3. The s3cmd configuration file contains all s3cmd settings.It can be installed by simply executing following commands .
Uses of s3cmd
- Uploading of data to s3
- Retrieving of data from s3
- Creating s3 bucket
- Manage data on s3 bucket for example logrotate
Installation
Follow the below command to install s3cmd .
sudo apt-get install s3cmd
Configure s3cmd environment
s3cmd --configure
This command will result in several input to be given from terminal itself, which are as follows .
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: xxxxxxxxxx
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes
New settings:
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Encryption password: xxxxxxxxxx
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
After configuring s3cmd successfully. Now Let’s look on how to use s3cmd command to manage s3 buckets.
Using s3cmd command
1. Creating New Bucket
Follow the below command to create new bucket in AWS S3.
s3cmd mb s3://mybucket
Bucket 's3://mybucket/' created
2. Uploading file in Bucket
Follow the below command to upload file in bucket.
s3cmd put file.txt s3://mycbucket/
file.txt -> s3://mybucket/file.txt [1 of 1]
190216 of 190216 100% in 0s 1668.35 kB/s done
3. Uploading Directory in Bucket
If we need to upload the entire directory in s3 bucket .We can use use -r to upload it recursively as below.
s3cmd put -r backup s3://mybucket/
backup/file1.txt -> s3://mybucket/backup/file1.txt [1 of 2]
9984 of 9984 100% in 0s 18.78 kB/s done
backup/file2.txt -> s3://mybackup/backup/file2.txt [2 of 2]
0 of 0 0% in 0s 0.00 B/s done
4. List Data of S3 Bucket
List the objects of s3 bucket using ls switch with s3cmd.
s3cmd ls s3://mybucket/
DIR s3://mybucket/backup/
2013-09-03 10:58 190216 s3://mybucket/file.txt
5. Download files from bucket
If we want to download files from the s3 bucket, Use following commands to download it.
s3cmd get s3://mybucket/file.txt
s3://mybucket/file.txt -> ./file.txt [1 of 1]
4 of 4 100% in 0s 10.84 B/s done
6. Remove Data of S3 Bucket
To remove data of s3 bucket use following commands.
#Removing file from s3 bucket
s3cmd del s3://mybucket/file.txt
File s3://mybucket/file.txt deleted
#Removing directory from s3 bucket
s3cmd del s3://mybucket/backup
File s3://mybucket/backup deleted
7. Remove S3 Bucket
If we don’t want s3 bucket any more, we can delete it using following command. Before removing bucket ,check that it should be empty.To remove bucket first remove all objects inside bucket and then use command .
s3cmd rb s3://mybucket
Bucket 's3://mybucket/' removed
8. List All S3 Bucket
This command is to enlist all s3 bucket.
s3cmd ls
9. Sync log file to S3 Bucket
To sync log file to s3 bucket use the following commands.
s3cmd sync /var/log/nginx/*/gz s3://mybucket/