OpenCAYLE. User guides How to use OpenCayle on HPC environment

How to use OpenCayle on HPC environment

* OpenCayle use is restricted only to login nodes (frontends), due to whatever iteration must be done direcly from the own device.
* Users can not create or access to anything assembly point from calculation nodes.

Connection

Generate your access key as it is explained on: OpenCayle User Guide.
In Python User Guide, you can see how to check your conection throught the script s3CheckConnection.py.

NOTE: To download python programs (.py extension) , please, follow the instructions:
1) Select the file
2) Press the right button on your mouse
3) Select the "save as" option

* When connection is successfully a message similar to this was shown:

    ```
    [mi_usuario_1_1@frontend1 ~]$ python s3Scripts/s3CheckConnection.py <access_key>
    Your accessKey connects successfully to OpenCayle
    ```
    >>>>>
    Where mi_usuario_1_1 will be the user connected to HPC.

* When the connection is wrong, please, contact to Support Scayle Team.

Bucket creation
Create your first bucket, when the bucket exists, it is not necessary its creation.

There are different options to create buckets:

Generate a bucket on your personal device:
- See “Use of service with graphical interface s3Browser.
- See “Use of service from the command line s3cmd.

Generate a bucket from the frontends:

Create the configuration file: "s3test.cfg".
To list the contained objets on directory execute "ls" command and verify there is nothing created on OpenCayle yet.
```
[mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg ls
```

Create your bucket using this command:

[mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg mb s3://<bucket_name>

Example:

[mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg mb s3://su_primer_bucket
Bucket 's3://su_primer_bucket/' created

Execute “ls” again, to check the bucket appears now.

[mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg ls
2020-06-16 09:32  s3://su_primer_bucket

Assembly point creation

The assembly point is any directory from which the mounted file system will be displayed. If the assembly point contained files, these will disappear, in order to display the files on the mounted device.

Create an assembly point to transfer the data from Caléndula cluster to OpenCayle.
It is necessary create a directory where you mount you bucket.
```
[mi_usuario_1_1@frontend1 ~]$ mkdir -p <directory_name>
```
Example:
Directory was called the same as the bucket “su_primer_bucket”, although you can called as you prefer.
```
[mi_usuario_1_1@frontend1 ~]$ mkdir -p s3mnt/su_primer_bucket
where s3mnt/su_primer_bucket is the directory_name
```

Use the following command to mount the bucket on the directory created.

[mi_usuario_1_1@frontend1 ~]$ s3fs <bucket_name> <directory_name>-o passwd_file=~/.passwd-s3fs -o url=https://ss3.scayle.es:443 -o use_path_request_style -o dbglevel=info

On the previous example:

[mi_usuario_1_1@frontend1 ~]$ s3fs su_primer_bucket s3mnt/su_primer_bucket -o passwd_file=~/.passwd-s3fs -o url=https://ss3.scayle.es:443 -o use_path_request_style -o dbglevel=info

If you list the directory , you can observe it is empty.

[mi_usuario_1_1@frontend1 ~]$ ls s3mnt/su_primer_bucket
total 0

Object creation

To create an object contained on bucket:

[mi_usuario_1_1@frontend1 ~]$ touch s3mnt/<directory_name>/<object_name>

Example:

[mi_usuario_1_1@frontend1 ~]$ touch s3mnt/su_primer_bucket/su_primer_objeto
[mi_usuario_1_1@frontend1 ~]$ ll s3mnt/su_primer_bucket
total 1
-rw-r--r-- 1 mi_usuario_1_1 hpc 0 jun 16 11:36 su_primer_objeto

Check the sincronization is successfully
To verify the sincronization, use the s3Resources.py (To download select the file and click on the ritgh button on your mouse, after that select the save as option).

[mi_usuario_1_1@frontend1 ~]$ python s3Scripts/s3Resources.py <access_key>
PARAMETERS: Access Key <<access_key>>
-----------------------------------------------------------------------------------------
                       Open Scayle Resources associated to user.  
-----------------------------------------------------------------------------------------

- BUCKET su_primer_bucket 2020-06-16T09:32:59.440Z
   - KEY su_primer_objeto   0   2020-06-16T09:36:02.067Z
-----------------------------------------------------------------------------------------
                        END Open Scayle Resources  
-----------------------------------------------------------------------------------------

Unmount the assembly point

To unmount the assembly point and no sincronize anymore your OpenCayle data:

[mi_usuario_1_1@frontend1 ~]$ fusermount -u <directory_name>

On the example:

[mi_usuario_1_1@frontend1 ~]$ fusermount -u s3mnt/su_primer_bucket