Oracle Cloud object Store access with rlone.

 If you are using the Oracle Object Store as part of the Oracle Public Cloud, "rclone" is an open source tool you can use to make things easier.

One of the things I really like about RCLONE is that it provides a command line like interface that is easy to use.  If you have looked at the OCI cli tool, it requires a myriad of parameters.  Below is the command I was using with OCI to view my list of buckets (I obfuscated some of the values).

oci os bucket list --endpoint https://objectstorage.us-ashburn-1.oraclecloud.com  --namespace-name id20xxxxxofo --compartment-id ocid1.compartment.oc1..aaaaaaxxxxxxxxxxxxxxxxxcpqyvzzb4ykd3tyq --config-file ~/.oci/natdconfig 

In order to use the OCI tool, I had to constantly keep a text file open to copy and paste commands.

In comparison, this is the command to list the buckets in my object store using rlcone.

rclone ls oci_bucket:

1) Configure compatibility for an S3 interface in the Public cloud.

In your public cloud council, in the top right hand corner, click on the "silhouette" that controls you settings. in the pull down menu click on "user settings" to bring up the window to configure you resources.  Once there, click on "Customer Secret keys" and then "Generate Secret Key" bring up the window to add a secret key.

On this window give your secret key a name (like S3Key" in my case).  When you click the "Generate Secret key" button, it will give you secret associated with key. SAVE THIS.

Once complete, you will have 2 items associated with your account

NAME:            S3Key                                                        or whatever you named your key.

Access Key:    ddddddddddddeeeeeeeeeffffffffggggg      A uniquely identified key ID

Secret Key :   dd32234sdwercfwe                                     A system generated "secret"

2) Download rclone.

     This can easily be done from the  RCLONE.ORG site.

    Note: You chose the platform you want to execute rclone on, then download the .zip file.

              The .zip file contains the execute, and documentation.

              Copy the "rclone" executable to the location of your choice and make it executable.

2) Configure Rclone.

    You start by executing "rclone config". This will create a configuration file in ~/.config/rclone/rclone called rclone.conf.  This is an interactive interface that will set the correct configuration parameters to be used.

This is an example of what I entered to connect to my Object Store using the S3 interface.

--> rclone config

Give this entry a unique name to identify the S3 object store.

 

Name> oci_s3    <-- my entry name in the config file 

Type of storage to configure.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value

..

 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)

..

Storage> 4      <-- 4 identifies this as an S3 compatible object store

Choose your S3 provider.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value

..

13 / Any other S3 compatible provider

..

provider> 13      <-- 13 identifies this as "other" S3 compatible object store

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).

Only applies if access_key_id and secret_access_key is blank.

Enter a boolean value (true or false). Press Enter for the default ("false").

Choose a number from below, or type in your own value

 1 / Enter AWS credentials in the next step

   \ "false"

 env_auth> 1     <-- 1 to identify that we are using  "AWS compatible Key" for authentication

AWS Access Key ID.

Leave blank for anonymous access or runtime credentials.

Enter a string value. Press Enter for the default ("").

access_key_id>  ddddddddddddeeeeeeeeeffffffffggggg   <-- This is the Access key ID that was generated from my name in the public cloud

AWS Secret Access Key (password)

Leave blank for anonymous access or runtime credentials.

Enter a string value. Press Enter for the default ("").

secret_access_key> dd32234sdwercfwe  --> The system generated key associated with my access key

Region to connect to.

Leave blank if you are using an S3 clone and you don't have a region.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value

 1 / Use this if unsure. Will use v4 signatures and an empty region.

   \ ""

region>        --> Leave blank

Endpoint for S3 API.

Required when using an S3 clone.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value

endpoint> {namespace}.compat.objectstorage.{region}.oraclecloud.com   --> Note that you will need to fill in your namespace from your account, and ensure the region is correct for the URL.

Location constraint - must be set to match the Region.

Leave blank if not sure. Used when creating buckets only.

Enter a string value. Press Enter for the default ("").

location_constraint>        --> Leave blank

Canned ACL used when creating buckets and storing or copying objects.

This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.

For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl

Note that this ACL is applied when server side copying objects as S3

doesn't copy the ACL from the source but rather writes a fresh one.

Enter a string value. Press Enter for the default ("").

Choose a number from below, or type in your own value

 1 / Owner gets FULL_CONTROL. No one else has access rights (default).

   \ "private"

acl>          --> Leave blank

Edit advanced config? (y/n)

y) Yes

n) No (default)

 y/n>           --> Leave blank

Remote config

--------------------

[oci_s3]

type = s3

provider = Other

env_auth = false

access_key_id = S3_Key

secret_access_key = ddddddd...

endpoint = xxxxxxx.compat.objectstorage.us-ashburn-1.oraclecloud.com

--------------------

y) Yes this is OK (default)

e) Edit this remote

d) Delete this remote

y/e/d> y           --> y to save this entry

3) Validate rclone.

Now let's verify what got create.

> cat ~/.config/rclone/rclone.conf

[oci_s3]

type = s3

provider = other

env_auth = false

access_key_id =dd32234sdwercfwe

secret_access_key = dddddxxxxxx

endpoint = xxxxxxx.compat.objectstorage.us-ashburn-1.oraclecloud.com

acl = authenticated-read

That's It.  In my case 

• the entry is "oci_s3"
• The access key for S3 is dd32234sdwercfwe"
• The secret associated with my S3 key is "dddddxxxxxx"
• The end point I am connecting to is "xxxxxxx.compat.objectstorage.us-ashburn-1.oraclecloud.com"
• "xxxxxxx" is my namespace
• "us-ashburn-1" is my region

4) Using rclone.

Now with rclone I can execute commands against my object store that are more linux like.

rclone mkdir oci_s3:mybucket  --> will create a bucket named "mybucket"

rclone ls oci_s3:  --> will list all my buckets

rclone ls oci_s3:mybucket --> will list all the objects in my bucket.

I can also use it to copy to and from my bucket.

rlcone copy /home/oracle/myfile.txt oci_s3:mybucket   --> copies the file to the bucket.

Finally, a great command is sync to synchronize the contents of my on-prem to the cloud

 rlcone sync /home/oracle/mydir/ oci_s3:mybucket  --> this will sync the two locations

Now how fun with it !!

ZDLRA - Using Protection Policies to manage databases that have migrated or to be retired

 One the questions that keeps coming up with ZDLRA is how to manage the backups for a database that has either

• Been migrated to another ZDRA
• Been retired, but the backup needs to be kept for a period of time

The best way to deal with this by the use of Protection Policies.

How Protection Policies work:

If you remember right, Protection Policies are way of grouping databases together that have the same basic characteristics.

The most important of which are :

Name/Description             - Used to identify the Protection Policy

Recovery Window Goal    - How many days of recovery do you want to store at a minimum 

Max Retention Window    - (Optional) Maximum number of days of backups you want to keep

Unprotected Window        - (Optional) Used to set alerts for databases that are no longer receiving recovery data.

One of the common questions I get is.. What happens if I change the Protection Policy associated with my database ?

Answer :  By changing the Protection Policy a database is associated with, you are only changing the metadata.  Once the change is made, the database follows the Protection Policy rules it is now associated with, and no longer is associated with the old Protection Policy

How this plays out with a real example is... 

My Database (PRODDB) is a member of a Protection Policy (GOLD) which has a Recovery Window Goal of 20 days, and a Max Retention Window of 40 days (the default value being 2x the Recovery Window Goal).

My Database (PRODDB) currently has 30 days of backups, which is right in the middle. 

 What would normally happen for this database is (given enough space), backups will continue to be kept until PRODDB has 40 days of backups.  On day 41, a maintenance job (which runs daily) will execute, and find that my database, PRODDB, has exceeded it's Recovery Window Goal.  This job will remove all backups (in a batch process for efficiency) that are older than 20 days.

BUT ........................

Today, I moved my database, PRODDB, to a new protection policy (Silver) which only has a 10 day Recovery Window Goal, and a Max Recovery Window of 20 Days.

As I pointed out, the characteristics of the NEW Protection Policy will be used, and the next time the daily purge occurs, this database will be flagged, and all backups greater than the Recovery Window Goal will be purged.

Retiring databases: - 

One very common question how to handle the retiring of database.  As you might know, when you remove a database from the ZDLRA, ALL backups are removed from ZDLRA.

When a database is no longer sending backups to the ZDLRA,  the backups will continue to be purged until only a single level 0 backup remains.  This is to ensure that at least one backup is kept, regardless of the Max Recovery Window.

The best way to deal with Retiring database (and still keep the last Level 0 backup) through the use of Protection Policies.

In my example for my database PRODDB, I am going to retire the database instead of moving it to the Silver policy.  My companies standard is to  keep the final backup for my database available for 90 days, and on day 91 all backups can be removed.

These are requirements from the above information.

• At least 1 backup is kept for 90 days, even though my Max Recovery Window was 40 days.
• I want to know when my database has been retired for 90 days so I can remove it from the ZDLRA.

In order to accomplish both of these items, I am going to create a Protection Policy named RETIRED_DB with the following attributes

• Recovery Window Goal of 2 days
• Max Recovery Window of 3 Days
• Unprotected Data Window of 90 days
• New Alert in OEM to tell me when a database in this policy violates its Unprotected Data Window

If you look closely at the attributes, you will noticed that I decreased the Recovery Window Goal to allow backups to be removed after 3 days.  I also set the Unprotected Data Window to be 90 days.

What this looks like over  time is 

As you can see by moving it to the new policy, within a few days, all backups except for the most recent Full back is removed.  You can also see that on day 91 (when it's time to remove this database) I will be getting an alert.

Migrating Databases:

Migrating databases is very similar to retiring databases, except that I don't want remove the old backups until they naturally expire.  For my example of PRODB with a Recovery Window Goal of 20 days, as soon as I have a new Level 0 on the new ZDLRA, I will move this database to a new policy (GOLD_MIGRATED) with the following attributes.

• Recovery Window Goal of 20 days, since I still need to preserve old backups
• Max Recovery Window goal of 21 days. This will  remove the old backups as they age off.
• Unprotected Data Window of 21 days, which will alert me that it time to remove this database.

How this would look over time time is.

Conclusion:

When retiring or migrating databases, Protection Policies can be leveraged to both

• Ensure backups are removed as they age out until only a single L0 (Full) remains
• Alert you when it is time to remove the database from the ZDLRA.

ZFS as a swift object store

 This blog post goes through a feature of the ZFS Appliance that has been around for at least 3 years now. The Openstack Swift Object store.

When looking at the S3 API, and the OCI API, I forgot all about where it started.. With the Swift API.

I will go through the 3 APIs, and how they came about (from what I can find by reading through articles)..

It all started with the Swift API. Swift (V1) has simple authentication and a simple interface.

A URI to manage/access objects has the format of

HTTP://{object store server}/object/v1/{Account}/{bucket name}/{object name}.

In the case of ZFS, 

• Account - this is the share name.. "/export/swiftshare" for example
• Bucket name - The name of the bucket that was created
• Object name - name of the object.

Authentication with Swift while using curl is typically a 2 step process.  

First the authorization URI is called

HTTP://{object store server}/auth/v1.0/{Account}

The username and password is sent with the authentication URI.  The URI then returns an auth token which is used in the curl command line to manage buckets/objects.

Username/password authentication (v1.0) is one of the 3 choices.

1. Local username/password created on the ZDLRA.
2. LDAP user ZFS ties to
3. Keystone authentication server.

For all the testing I am doing on my ZFS simulator, I use a local user.

Before I go into how to configure and use the Swift interface on ZFS, I'll share what I was able to find out.

The Swift API has some limitations, and these limitations is what drove the move to S3.

As you probably noticed, the authentication and tracking of objects does not have enough details to support the segregation of users, and billing.

The S3 API takes the Swift API, and adds the ability to create separate tenants, set up billing, etc. All the things an enterprise needs to do.

With S3, you probably noticed that the authentication layer changed. It is based on secret name/secret rather than a username/password returning an auth token..

Well lets go through what it takes to configure the Swift interface.

First, most of the steps around configuring ZFS for an object store, I documented in my previous blog posts. 

If you look the posts below you see the steps on configuring a share,creating a local user on ZFS, and configuring the http service.

ZFS Appliance - Your on-premise cloud store

Managing authentication for a ZFS Object Store

For Swift, I will just go the steps specific to Swift.

All I need to do is enable swift. That's it !

Swift gets enabled just like enabling the S3 API, and the OCI API. Because I do not have a Keystone Authentication Server (which would be the OpenStack Identity Service), I didn't fill those values in.

NOTE: Authentication for swift is a little different from S3, or OCI. Both of the other APIs do not tie directly to the local user.  S3 uses "secrets", and OCI uses a PEM file, and a Fingerprint.

Accessing my Swift bucket.

First some links to documentation that will give you examples of these ways of connecting.

Swift Guide for ZFS OS 8.8 release  -- Current release as of writing.

Using ZFS as an object store  -- This is old, but has a lot of detail and great detail

API Guide OS 8.8 for Swift Docs -- Current documentation guide

Also, since the Swift implementation is OpenStack, there is a lot of examples and documentation (non-oracle) available across the web.

I was able to access my bucket one of 3 ways

The first 2 ways are very similar

Swift command line tool - python based tool to connect to swift and manage buckets

CURL - Command line took similar to swift.

In order to create a bucket, and upload an object ......

First I execute the  curl command to get the authentication token.

NOTE: my ZFS emulator is 10.0.0.110 and my share is /export/short

curl -i http://10.0.0.110/auth/v1.0/export/short -X GET -H "X-Auth-User: oracle" -H "X-Auth-Key: oracle123"

HTTP/1.1 200 OK
Date: Thu, 21 Jan 2021 18:54:38 GMT
Server: Apache
X-Content-Type-Options: nosniff
X-Storage-Url: http://10.0.0.110:80/object/v1/export/short
X-Auth-Token: ZFSSA_522d6355-9056-4a95-9060-c88648007993
X-Storage-Token: ZFSSA_522d6355-9056-4a95-9060-c88648007993
Content-Length: 0
X-Trans-Id: tx62e2f031f21640c29a2bf-006009cdee
Content-Type: text/html; charset=utf-8

Next I execute create a bucket .

From the output  above I can get the "Auth Token", and the Storage URL to manage the object store in curl. Note that the Auth Token will expire.

Create a container in curl

curl -i http://10.0.0.110:80/object/v1/export/short/bucketswift -X PUT -H "Content-Length: 0" -H "X-Auth-Token: ZFSSA_522d6355-9056-4a95-9060-c88648007993"

Create a container in swift

swift post container -A http://10.0.0.110:80/object/v1/export/short -U oracle -K oracle123

That's all there is to it with the Swift Object Store on ZFS.