How to use OpenCayle on HPC environment

* OpenCayle use is restricted only to login nodes (frontends), due to whatever iteration must be done direcly from the own device.
* Users can not create or access to anything assembly point from calculation nodes.

  1. Connection
  • Generate your access key as it is explained on: OpenCayle User Guide.
  • In Python User Guide, you can see how to check your conection throught the script s3CheckConnection.py (to download the file right click save as .py extension).

    • When connection is successfully a message similar to this was shown:

      [mi_usuario_1_1@frontend1 ~]$ python s3Scripts/s3CheckConnection.py <access_key>
      Your accessKey connects successfully to OpenCayle

      Where mi_usuario_1_1 will be the user connected to HPC.

    • When the connection is wrong, please, contact to Support Scayle Team.
  1. Bucket creation
    Create your first bucket, when the bucket exists, it is not necessary its creation.

    There are different options to create buckets:

  • Generate a bucket on your personal device:

    • See “Use of service with graphical interface s3Browser.
    • See “Use of service from the command line s3cmd.
  • Generate a bucket from the frontends:

    • Create the configuration file: "s3test.cfg".
    • To list the contained objets on directory execute "ls" command and verify there is nothing created on OpenCayle yet.

      [mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg ls
    • Create your bucket using this command:

      [mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg mb s3://<bucket_name>

      Example:

      [mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg mb s3://su_primer_bucket
      Bucket 's3://su_primer_bucket/' created
    • Execute “ls” again, to check the bucket appears now.
      [mi_usuario_1_1@frontend1 ~]$ s3cmd -c s3test.cfg ls
      2020-06-16 09:32  s3://su_primer_bucket
  1. Assembly point creation

    The assembly point is any directory from which the mounted file system will be displayed. If the assembly point contained files, these will disappear, in order to display the files on the mounted device.

  • Create an assembly point to transfer the data from Caléndula cluster to OpenCayle.

  • It is necessary create a directory where you mount you bucket.

    [mi_usuario_1_1@frontend1 ~]$ mkdir -p <directory_name>

    Example:
    Directory was called the same as the bucket “su_primer_bucket”, although you can called as you prefer.

    [mi_usuario_1_1@frontend1 ~]$ mkdir -p s3mnt/su_primer_bucket
    where s3mnt/su_primer_bucket is the directory_name
    
  • Use the following command to mount the bucket on the directory created.

    [mi_usuario_1_1@frontend1 ~]$ s3fs <bucket_name> <directory_name>-o passwd_file=~/.passwd-s3fs -o url=https://ss3.scayle.es:443 -o use_path_request_style -o dbglevel=info

    On the previous example:

    [mi_usuario_1_1@frontend1 ~]$ s3fs su_primer_bucket s3mnt/su_primer_bucket -o passwd_file=~/.passwd-s3fs -o url=https://ss3.scayle.es:443 -o use_path_request_style -o dbglevel=info

    If you list the directory , you can observe it is empty.

    [mi_usuario_1_1@frontend1 ~]$ ls s3mnt/su_primer_bucket
    total 0
  1. Object creation

    To create an object contained on bucket:

    [mi_usuario_1_1@frontend1 ~]$ touch s3mnt/<directory_name>/<object_name>

    Example:

    [mi_usuario_1_1@frontend1 ~]$ touch s3mnt/su_primer_bucket/su_primer_objeto
    [mi_usuario_1_1@frontend1 ~]$ ll s3mnt/su_primer_bucket
    total 1
    -rw-r--r-- 1 mi_usuario_1_1 hpc 0 jun 16 11:36 su_primer_objeto
  2. Check the sincronization is successfully
    To verify the sincronization, use the s3Resources.py (to download the file right click save as .py extension).

    [mi_usuario_1_1@frontend1 ~]$ python s3Scripts/s3Resources.py <access_key>
    PARAMETERS: Access Key <<access_key>>
    -----------------------------------------------------------------------------------------
                           Open Scayle Resources associated to user.  
    -----------------------------------------------------------------------------------------
    
    - BUCKET su_primer_bucket 2020-06-16T09:32:59.440Z
       - KEY su_primer_objeto   0   2020-06-16T09:36:02.067Z
    -----------------------------------------------------------------------------------------
                            END Open Scayle Resources  
    -----------------------------------------------------------------------------------------
  3. Unmount the assembly point

    To unmount the assembly point and no sincronize anymore your OpenCayle data:

    [mi_usuario_1_1@frontend1 ~]$ fusermount -u <directory_name>

    On the example:

    [mi_usuario_1_1@frontend1 ~]$ fusermount -u s3mnt/su_primer_bucket