Use of service from the command line (s3cmd)

S3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data, so it can be used on OpenCayle s3 storage.

  • s3cmd Ubuntu Installation
sudo apt install s3cmd
  • s3cmd CentOS Installation
yum install s3cmd

sudo vi s3test.cfg

Copy information above on the file and change ACCESS_KEY with your user access key to OpenCayle service.

[default]
access_key =  ACCESS_KEY
access_token = 
add_encoding_exts = 
add_headers = 
bucket_location = 
ca_certs_file = 
cache_file = 
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = ss3.scayle.es:443
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date = 
expiry_days = 
expiry_prefix = 
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = 
guess_mime_type = True
host_base = ss3.scayle.es
host_bucket = ss3.scayle.es:443/%(bucket)
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key = 
limitrate = 0
list_md5 = False
log_target_prefix = 
long_listing = False
max_delete = -1
mime_type = 
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
secret_key = ACCESS_KEY 
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = ss3.scayle.es:443
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class = 
urlencoding_mode = normal
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://ss3.scayle.es:443/%(bucket)
website_error = 
website_index = index.html

s3cmd -c <configuration file> ls

Successfully execution:

Example:

s3cmd -c s3test.cfg ls

2020-05-22 11:29  s3://bucket.empty.cephuser
2020-05-21 12:01  s3://bucket.link.cephuser
2020-03-12 09:17  s3://new-bucket-104aa0ba
2020-03-10 13:54  s3://new-bucket-7a7009d8

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg ls
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

s3cmd -c <configuration file> ls  s3://<bucket name>

Successfully execution:

Example:

s3cmd -c s3test.cfg ls   s3://bucket.link.cephuser

    2020-05-27 11:10      4128   s3://bucket.link.cephuser/data2.txt

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg ls s3://bucket.link.cephuser
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg ls   s3://xxxx
    ERROR: Bucket 'xxxx' does not exist
    ERROR: S3 error: 404 (NoSuchBucket)

s3cmd -c <configuration file> mb<s3://<bucket name>/<object name>

Successfully connection:

Example:

s3cmd -c s3test.cfg mb s3://bucket.link.cephuser2
Bucket 's3://bucket.link.cephuser2' created

To check the bucket was created list user buckets:

Example:

s3cmd -c s3test.cfg ls

s3cmd -c <configuration file> put <fichero a añadir>  s3://<bucket name>

When the object is an object container (folder) use the "-r" command option:

s3cmd -c <configuration file> put -r <fichero a añadir> s3://<bucket name>

Successfully execution:

Example:

s3cmd -c s3test.cfg put s3fs.sh s3://bucket.link.cephuser

    upload: 's3fs.sh' -> 's3://bucket.link.cephuser/s3fs.sh'  [1 of 1]
    1039 of 1039   100% in    0s     7.42 kB/s
    1039 of 1039   100% in    0s     7.42 kB/s

To check it was ok, you can list the object of the bucket.

Example:

s3cmd -c s3test.cfg ls   s3://bucket.link.cephuser

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg put s3fs.sh s3://bucket.link.cephuser
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg put s3fs.sh s3://xxxx
    ERROR: upload 's3fs.sh' → 's3://xxxx.s3fs.sh' [1 of 1]
        1039    100%    in 0s   5.05 kB/s done
    ERROR: Configuration file not available.
    ERROR: S3  error: 404 (NoSuchBucket)

s3cmd -c <configuration file> get  s3://<bucket name>/<object name> <file name which will contain the object>

Successfully execution:

Example:

s3cmd -c s3test.cfg get s3://bucket.link.cephuser/s3fs.sh test.txt

    download: 's3://bucket.link.cephuser/s3fs.sh' -> 'test.txt'  [1 of 1]
1039 of 1039   100% in    0s     3.11 kB/s
1039 of 1039   100% in    0s     3.11 kB/s  done

Check the test.txt file exists and has the object information.
When file exists the command line warns you about.

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg get  s3://bucket.link.cephuser/s3fs.sh  test.txt
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist: (not specific error)

Example:

s3cmd -c s3test.cfg get  s3://xxxx/s3fs.sh  tesst.txt
    download 's3://xxxx/s3fs.sh' → 'tesst.txt' [1 of 1]
    ERROR: S3 error: 404 (Not Found)

Error, object contained on bucket does not exist:(not specific error):

Example:

s3cmd -c s3test.cfg get  s3://bucket.link.cephuser/s3ffs.sh  tesst.txt
    download 's3://bucket.link.cephuser/s3ffs.sh' → 'tesst.txt' [1 of 1]
    ERROR: S3 error: 404 (Not Found)

Error, object download file already exists:

Example:

s3cmd -c s3test.cfg get  s3://bucket.link.cephuser/s3fs.sh  test.txt
    ERROR: Parameter problem: File test.txt already exists. Use either of –force / – continue / --skip-existing or give it a new name.

s3cmd -c <configuration file> del   s3://<bucket name>/<object name>

Successfully execution:

Example:

s3cmd -c s3test.cfg del s3://bucket.link.cephuser
        delete: 's3://bucket.link.cephuser/s3fs.sh'

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg del s3://bucket.link.cephuser/s3fs.sh
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg del s3://xxxx/s3fs.sh
                          ERROR: S3  error: 404 (NoSuchBucket)

Pay atention: When the object to download does not exists, s3cmd does not warn about it.

Example:

s3cmd -c s3test.cfg del s3://bucket.link.cephuser/xxxx.sh
        delete: 's3://bucket.link.cephuser/xxxx.sh'

s3cmd -c <configuration file> rb s3://<bucket name>

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg del s3://bucket.link.cephuser/s3fs.sh
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg rb s3://xxxx
                          ERROR: S3  error: 404 (NoSuchBucket)

Error, bucket still contains objects:

Example:

s3cmd -c s3test.cfg rb s3://bucket.link.cephuser

ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty.

When the bucket contains objects , it is not possible to delete it. To do it:

  • List objects contained on user bucket.

    Example:

    s3cmd -c s3test.cfg ls   s3://bucket.link.cephuser

        2020-05-27 11:10      4128   s3://bucket.link.cephuser/data2.txt
  • Delete all objects contained on user bucket.

    Example:

    s3cmd -c s3test.cfg rb s3://bucket.link.cephuser
            delete: 's3://bucket.link.cephuser/s3fs.sh'
  • Delete the user bucket. Sucessfully execution.

    Example:

    s3cmd -c s3test.cfg rb s3://bucket.link.cephuser
            Bucket 's3://bucket.link.cephuser/' removed

  1. How to config object visibility.

    • Public object:
      s3cmd -c <configuration file> setacl  s3://< bucket name>/<object name> --acl-public.
    • Private object:
      s3cmd -c <configuration file> setacl  s3://<bucket namet>/<object name> --acl-private.
  2. How to obtain the object URL.

    s3cmd -c <configuration file> info s3://<bucket namet>/<object name>

    IMPORTANT: Due to OpenCayle is a secure service, it is necessary to modify http by https.

Example:

  • List of bucket objects:
s3cmd -c /home/ubuntu/test.cfg ls s3://bucket.link.cepthuser
2020-06-11 08:30       581   s3://bucket.link.cepthuser/rclone.conf
2020-06-11 08:30       877   s3://bucket.link.cepthuser/result.txt
2020-06-11 08:32         9   s3://bucket.link.cepthuser/test
  • How to assign public access to an object:
ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg setacl 
s3://bucket.link.cepthuser/result --acl-public
s3://bucket.link.cepthuser/result: ACL set to Public  [1 of 1]
  • How to obtain the object URL:
ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg info s3://bucket.link.cepthuser/result.txt 
s3://bucket.link.cepthuser/result (object):
   File size: 877
   Last mod:  Mon, 29 Jun 2020 11:51:00 GMT
   MIME type: application/octet-stream
   Storage:   STANDARD
   MD5 sum:   e072dc7fea78b2630adaabf880ceb1d7
   SSE:       none
   policy:    none
   cors:      none
   ACL:       *anon*: READ
   ACL:       <userid>: FULL_CONTROL
   URL:       http://ss3.scayle.es/bucket.link.cepthuser/result.txt
   x-amz-meta-sha256: 9d55dbf844ec33ea7f18bbe7282bd9b51495f8e91386ba3a1eec0ec7a5997af2
   x-amz-meta-s3b-last-modified: 20200609T132803Z
ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg setacl s3://bucket.link.cepthuser/rclone.conf --acl-private
  • How to obtain the object URL:
ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg info s3://bucket.link.cepthuser/rclone.conf
s3://bucket.link.cepthuser/rclone.conf (object):
   File size: 581
   Last mod:  Thu, 11 Jun 2020 08:30:22 GMT
   MIME type: application/octet-stream
   Storage:   STANDARD
   MD5 sum:   5d39536aeebc4e5cd9d1ce194876703b
   SSE:       none
   policy:    none
   cors:      none
   ACL:       <userid>: FULL_CONTROL
   x-amz-meta-sha256: 4175548f3d01c00a00a0f96104387b4818e9f043924575fd5fd5cd498149c8f2
   x-amz-meta-s3b-last-modified: 20200611T063936Z
ubuntu@ubuntu-VirtualBox:~$ 

When the object access is private its URL is not generated.