OpenCAYLE. User guides Service use General use Use of service from the command line (s3cmd)

Use of service from the command line (s3cmd)

S3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data, so it can be used on OpenCayle s3 storage.

s3cmd Ubuntu Installation

sudo apt install s3cmd

s3cmd CentOS Installation

yum install s3cmd

sudo vi s3test.cfg

Copy information above on the file and change ACCESS_KEY with your user access key to OpenCayle service.

[default]
access_key =  ACCESS_KEY
access_token = 
add_encoding_exts = 
add_headers = 
bucket_location = 
ca_certs_file = 
cache_file = 
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = ss3.scayle.es:443
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date = 
expiry_days = 
expiry_prefix = 
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = 
guess_mime_type = True
host_base = ss3.scayle.es
host_bucket = ss3.scayle.es:443/%(bucket)
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key = 
limitrate = 0
list_md5 = False
log_target_prefix = 
long_listing = False
max_delete = -1
mime_type = 
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
secret_key = ACCESS_KEY 
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = ss3.scayle.es:443
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class = 
urlencoding_mode = normal
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://ss3.scayle.es:443/%(bucket)
website_error = 
website_index = index.html

s3cmd -c <configuration file> ls

Successfully execution:

Example:

s3cmd -c s3test.cfg ls

2020-05-22 11:29  s3://bucket.empty.cephuser
2020-05-21 12:01  s3://bucket.link.cephuser
2020-03-12 09:17  s3://new-bucket-104aa0ba
2020-03-10 13:54  s3://new-bucket-7a7009d8

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg ls
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

s3cmd -c <configuration file> ls  s3://<bucket name>

Successfully execution:

Example:

s3cmd -c s3test.cfg ls   s3://bucket.link.cephuser

    2020-05-27 11:10      4128   s3://bucket.link.cephuser/data2.txt

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg ls s3://bucket.link.cephuser
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg ls   s3://xxxx
    ERROR: Bucket 'xxxx' does not exist
    ERROR: S3 error: 404 (NoSuchBucket)

s3cmd -c <configuration file> mb<s3://<bucket name>/<object name>

Successfully connection:

Example:

s3cmd -c s3test.cfg mb s3://bucket.link.cephuser2
Bucket 's3://bucket.link.cephuser2' created

To check the bucket was created list user buckets:

Example:

s3cmd -c s3test.cfg ls

s3cmd -c <configuration file> put <fichero a añadir>  s3://<bucket name>

When the object is an object container (folder) use the "-r" command option:

s3cmd -c <configuration file> put -r <fichero a añadir> s3://<bucket name>

Successfully execution:

Example:

s3cmd -c s3test.cfg put s3fs.sh s3://bucket.link.cephuser

    upload: 's3fs.sh' -> 's3://bucket.link.cephuser/s3fs.sh'  [1 of 1]
    1039 of 1039   100% in    0s     7.42 kB/s
    1039 of 1039   100% in    0s     7.42 kB/s

To check it was ok, you can list the object of the bucket.

Example:

s3cmd -c s3test.cfg ls   s3://bucket.link.cephuser

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg put s3fs.sh s3://bucket.link.cephuser
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg put s3fs.sh s3://xxxx
    ERROR: upload 's3fs.sh' → 's3://xxxx.s3fs.sh' [1 of 1]
        1039    100%    in 0s   5.05 kB/s done
    ERROR: Configuration file not available.
    ERROR: S3  error: 404 (NoSuchBucket)

s3cmd -c <configuration file> get  s3://<bucket name>/<object name> <file name which will contain the object>

Successfully execution:

Example:

s3cmd -c s3test.cfg get s3://bucket.link.cephuser/s3fs.sh test.txt

    download: 's3://bucket.link.cephuser/s3fs.sh' -> 'test.txt'  [1 of 1]
1039 of 1039   100% in    0s     3.11 kB/s
1039 of 1039   100% in    0s     3.11 kB/s  done

Check the test.txt file exists and has the object information.
When file exists the command line warns you about.

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg get  s3://bucket.link.cephuser/s3fs.sh  test.txt
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist: (not specific error)

Example:

s3cmd -c s3test.cfg get  s3://xxxx/s3fs.sh  tesst.txt
    download 's3://xxxx/s3fs.sh' → 'tesst.txt' [1 of 1]
    ERROR: S3 error: 404 (Not Found)

Error, object contained on bucket does not exist:(not specific error):

Example:

s3cmd -c s3test.cfg get  s3://bucket.link.cephuser/s3ffs.sh  tesst.txt
    download 's3://bucket.link.cephuser/s3ffs.sh' → 'tesst.txt' [1 of 1]
    ERROR: S3 error: 404 (Not Found)

Error, object download file already exists:

Example:

s3cmd -c s3test.cfg get  s3://bucket.link.cephuser/s3fs.sh  test.txt
    ERROR: Parameter problem: File test.txt already exists. Use either of –force / – continue / --skip-existing or give it a new name.

s3cmd -c <configuration file> del   s3://<bucket name>/<object name>

Successfully execution:

Example:

s3cmd -c s3test.cfg del s3://bucket.link.cephuser
        delete: 's3://bucket.link.cephuser/s3fs.sh'

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg del s3://bucket.link.cephuser/s3fs.sh
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg del s3://xxxx/s3fs.sh
                          ERROR: S3  error: 404 (NoSuchBucket)

Pay atention: When the object to download does not exists, s3cmd does not warn about it.

Example:

s3cmd -c s3test.cfg del s3://bucket.link.cephuser/xxxx.sh
        delete: 's3://bucket.link.cephuser/xxxx.sh'

s3cmd -c <configuration file> rb s3://<bucket name>

Error, configuration file does not exist:

Example:

s3cmd -c s3test2.cfg del s3://bucket.link.cephuser/s3fs.sh
    ERROR: s3test2.cfg: None
    ERROR: Configuration file not available.
    ERROR: Consider using –configure parameter to create one.

Error, bucket does not exist:

Example:

s3cmd -c s3test.cfg rb s3://xxxx
                          ERROR: S3  error: 404 (NoSuchBucket)

Error, bucket still contains objects:

Example:

s3cmd -c s3test.cfg rb s3://bucket.link.cephuser

ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty.

When the bucket contains objects , it is not possible to delete it. To do it:

List objects contained on user bucket.

Example:

    s3cmd -c s3test.cfg ls   s3://bucket.link.cephuser

        2020-05-27 11:10      4128   s3://bucket.link.cephuser/data2.txt

Delete all objects contained on user bucket.

Example:

    s3cmd -c s3test.cfg rb s3://bucket.link.cephuser
            delete: 's3://bucket.link.cephuser/s3fs.sh'

Delete the user bucket. Sucessfully execution.

Example:

    s3cmd -c s3test.cfg rb s3://bucket.link.cephuser
            Bucket 's3://bucket.link.cephuser/' removed

How to config object visibility.

Public object:

s3cmd -c <configuration file> setacl  s3://< bucket name>/<object name> --acl-public.

Private object:

s3cmd -c <configuration file> setacl  s3://<bucket namet>/<object name> --acl-private.

How to obtain the object URL.
```
s3cmd -c <configuration file> info s3://<bucket namet>/<object name>
```
IMPORTANT: Due to OpenCayle is a secure service, it is necessary to modify http by https.

Example:

List of bucket objects:

s3cmd -c /home/ubuntu/test.cfg ls s3://bucket.link.cepthuser
2020-06-11 08:30       581   s3://bucket.link.cepthuser/rclone.conf
2020-06-11 08:30       877   s3://bucket.link.cepthuser/result.txt
2020-06-11 08:32         9   s3://bucket.link.cepthuser/test

How to assign public access to an object:

ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg setacl 
s3://bucket.link.cepthuser/result --acl-public
s3://bucket.link.cepthuser/result: ACL set to Public  [1 of 1]

How to obtain the object URL:

ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg info s3://bucket.link.cepthuser/result.txt 
s3://bucket.link.cepthuser/result (object):
   File size: 877
   Last mod:  Mon, 29 Jun 2020 11:51:00 GMT
   MIME type: application/octet-stream
   Storage:   STANDARD
   MD5 sum:   e072dc7fea78b2630adaabf880ceb1d7
   SSE:       none
   policy:    none
   cors:      none
   ACL:       *anon*: READ
   ACL:       <userid>: FULL_CONTROL
   URL:       http://ss3.scayle.es/bucket.link.cepthuser/result.txt
   x-amz-meta-sha256: 9d55dbf844ec33ea7f18bbe7282bd9b51495f8e91386ba3a1eec0ec7a5997af2
   x-amz-meta-s3b-last-modified: 20200609T132803Z

We would access the object with the URL:
https://ss3.scayle.es/bucket.link.cepthuser/result.txt
How to assign private access to an object:

ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg setacl s3://bucket.link.cepthuser/rclone.conf --acl-private

How to obtain the object URL:

ubuntu@ubuntu-VirtualBox:~$ s3cmd -c /home/ubuntu/test.cfg info s3://bucket.link.cepthuser/rclone.conf
s3://bucket.link.cepthuser/rclone.conf (object):
   File size: 581
   Last mod:  Thu, 11 Jun 2020 08:30:22 GMT
   MIME type: application/octet-stream
   Storage:   STANDARD
   MD5 sum:   5d39536aeebc4e5cd9d1ce194876703b
   SSE:       none
   policy:    none
   cors:      none
   ACL:       <userid>: FULL_CONTROL
   x-amz-meta-sha256: 4175548f3d01c00a00a0f96104387b4818e9f043924575fd5fd5cd498149c8f2
   x-amz-meta-s3b-last-modified: 20200611T063936Z
ubuntu@ubuntu-VirtualBox:~$

When the object access is private its URL is not generated.