Monday, December 20, 2021

Backing up your database to a bucket in OCI and restoring it in OCI

 This is the first of a multi-part blog series walking through how to copy your TDE encrypted on premise Oracle Database to an OCI VM in the oracle cloud using the Oracle Database Backup Cloud Service. 


I am going to start with a simple test case of a small database which doesn't contain any TDE encryption or wallet, and back it up to an OCI bucket.

As far as where to start, below are some documentation links that will help.


NOTE: You will be doing downloading, and installing the library files on both the source database and the destination database.

Install Database backup module

The first thing I am going to do is unzip the Cloud Backup Module (opc_installer.zip).  This can downloaded using the link above, but it can also be found within the $ORACLE_HOME/lib  directory.  As always, I would recommend downloading the current copy to be sure it is the latest release.   Once unzipped you will find the module contains a directory "opc_installer".  Within "opc_installer" there are 2 subdirectories with a ".jar" file to install the library, and a readme file.

    oci_installer/                                                  ---> OCI (Oracle Cloud Native) library install
                       oci_install.jar
                       oci_readme.txt
    opc_installer/                                                  ---> OPC (Oracle Cloud Gen 1/swift) library install
                       opc_install.jar
                       opc_readme.txt

I am going to use "oci_install.jar" file and access the bucket using the Oracle Cloud Native API.

If I look in the "readme" file, I can see that I install the library using the following parameters.


I am going to install my files within a new directory for my Database host.

/home/oracle/ocicloud/
                                    config/
                                    lib/
                                    wallet/

To install and configure my library I am going to execute

java -jar oci_install.jar
         -host https://objectstorage.us-ashburn-1.oraclecloud.com  
        -pvtkeyFile  /home/oracle/ocicloud/myprivatekey.ppk 
        -pubFingerPrint 6d:f9:57:d5:ff:b1:c0:98:81:90:1e:6e:08:0f:d0:69 
        -tOCID ocid1.tenancy.oc1..aaaxxxnoq 
        -uOCID ocid1.user.oc1..aaaaaaaae2mlwyke4gvxxsaouxq 
        -bucket migest_backups  
        -walletDir /home/oracle/ocicloud/wallet 
        -configFile /home/oracle/ocicloud/config/migtestbackup.ora 
         -cOCID ocid1.compartment.oc1..aaaaaaaanqbquh2bwju4igzd3tyq 
        -libDir /home/oracle/ocicloud/lib  

Oracle Database Cloud Backup Module Install Tool, build MAIN_2021-08-31
Oracle Database Cloud Backup Module credentials are valid.
Backups would be sent to bucket migest_backups.
Oracle Database Cloud Backup Module wallet created in directory /home/oracle/ocicloud/wallet.
Oracle Database Cloud Backup Module initialization file /home/oracle/ocicloud/config/migtestbackup.ora created.
Downloading Oracle Database Cloud Backup Module Software Library from Oracle Cloud Infrastructure.
Download complete.

Now that it is successfully installed we can go to configuring the module.

Configure Database backup module

Running the command below, lets see what is in my directory now.

find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"

.
 |-lib
 | |-bulkimport.pl
 | |-libopc.so
 | |-metadata.xml
 | |-odbsrmt.py
 | |-perl_readme.txt
 | |-python_readme.txt
 |-config
 | |-migtestbackup.ora
 |-wallet
 | |-cwallet.sso.lck
 | |-cwallet.sso
 |-oci_install.jar
 |-myprivatekey.ppk

Looking at the configuration file created you can see the information used to connect to the bucket in the OCI Object store.

OPC_HOST=https://objectstorage.us-ashburn-1.oraclecloud.com/n/id20avsofo
OPC_WALLET='LOCATION=file:/home/oracle/ocicloud/wallet CREDENTIAL_ALIAS=alias_oci'
OPC_CONTAINER=migest_backups
OPC_COMPARTMENT_ID=ocid1.compartment.oc1..aaaaaaaanqbquh2bwju4igabu5g7xuiijael2fwcpqyvzzb4ykd3tyq
OPC_AUTH_SCHEME=BMC


Now we can create the channel configuration to send backups to the oci bucket.
The create channel would be executed like the command below filling in the library and configuration file

CONFIGURE CHANNEL DEVICE TYPE 
        'SBT_TAPE' PARMS 
         'SBT_LIBRARY={library name and location},
                SBT_PARMS=(OPC_PFILE=/{configuration file})';

Below are the commands I am going to execute in RMAN to configure my channel and settings to backup my database.




## Default device type is tape
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE'; 

## Backup using the library and config file we just installed
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ocicloud/lib/libopc.so,SBT_PARMS=(OPC_PFILE=/home/oracle/ocicloud/config/migtestbackup.ora)';

## Backup with 4 channels to a compressed backupset
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;

## Use medium compression since this is included in the license for the module.
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM';

## Encrypt the backup being sent, this is mandatory for writing to the cloud.
CONFIGURE ENCRYPTION FOR DATABASE ON;



Backup Database to an OCI bucket

Set a password to encrypt the backup (it must be encrypted to send to a bucket) and perform a full backup.



set encryption identified by oracle only;
backup incremental level 0 database plus archivelog not backed up;




This will send the backup to the object store

Configure Database backup module in OCI.

I am going to go through the same series of steps to install the Oracle Database Cloud backup Module in my OCI instance. 

Oracle Database Cloud Backup Module Install Tool, build MAIN_2021-08-31
Oracle Database Cloud Backup Module credentials are valid.
Backups would be sent to bucket migest_backups.
Oracle Database Cloud Backup Module wallet created in directory /home/oracle/ocicloud/wallet.
Oracle Database Cloud Backup Module initialization file /home/oracle/ocicloud/config/migtestbackup.ora created.
Downloading Oracle Database Cloud Backup Module Software Library from Oracle Cloud Infrastructure.
Download complete.

Configure pfile for database in OCI.


I now need to configure my database pfile in OCI. I just need a few basic things 

audit_file_dest='/u01/app/oracle/admin/migtest/adump'
*.audit_trail='db'
*.compatible='19.0.0'
*.control_files='/u01/app/oracle/oradata/MIGTEST/controlfile/controlfile1.ctl','/u01/app/oracle/oradata/MIGTEST/controlfile/controlfile2.ctl'
*.db_block_size=8192
*.db_create_file_dest='/u01/app/oracle/oradata'
*.db_name='migtest'
*.processes=300
*.sga_target=4638m


Restore pfile and controlfile for database in OCI

There a few steps to get ready to restore the spile and controlfile
  • I add my database to the "/etc/oratab" to ensure I can connect to it, and ". oraenv" to set the environment.
  • I now start up the database nomount
  • I go back to the original database to retrieve the dbid.

Now I am ready to restore the spfile (note that I am setting the password to decrypt the backups).


In RMAN I restore the spfile
set decryption identified by oracle;

 run {
 allocate CHANNEL c1 TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ocicloud/ib/libopc.so,SBT_PARMS=(OPC_PFILE=/home/oracle/ocicloud/config/migtestbackup.or)';
 restore spfile from autobackup ;
 release channel c1;
 }
rman target /
RMAN> set decryption identified by oracle;

executing command: SET decryption
 run {
 allocate CHANNEL c1 TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ocicloud/ib/libopc.so,SBT_PARMS=(OPC_PFILE=/home/oracle/ocicloud/config/migtestbackup.or)';
 restore spfile from autobackup ;
 release channel c1;
 }


RMAN> 2> 3> 4> 5>
allocated channel: c1
channel c1: SID=20 device type=SBT_TAPE
channel c1: Oracle Database Backup Service Library VER=21.0.0.1

Starting restore at 20-DEC-21

channel c1: looking for AUTOBACKUP on day: 20211220
channel c1: AUTOBACKUP found: c-286701374-20211220-00
channel c1: restoring spfile from AUTOBACKUP c-286701374-20211220-00
channel c1: SPFILE restore from AUTOBACKUP complete
Finished restore at 20-DEC-21


Then I restore the controlfile.


set decryption identified by oracle;

 run {
 allocate CHANNEL c1 TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ocicloud/ib/libopc.so,SBT_PARMS=(OPC_PFILE=/home/oracle/ocicloud/config/migtestbackup.or)';
 restore controlfile from autobackup ;
 release channel c1;
 }

RMAN>
executing command: SET decryption
using target database control file instead of recovery catalog


RMAN> 2> 3> 4> 5>
allocated channel: c1
channel c1: SID=17 device type=SBT_TAPE
channel c1: Oracle Database Backup Service Library VER=21.0.0.1

Starting restore at 20-DEC-21

channel c1: looking for AUTOBACKUP on day: 20211220
channel c1: AUTOBACKUP found: c-286701374-20211220-00
channel c1: restoring control file from AUTOBACKUP c-286701374-20211220-00
channel c1: control file restore from AUTOBACKUP complete
output file name=/u01/app/oracle/oradata/MIGTEST/controlfile/controlfile1.ctl
output file name=/u01/app/oracle/oradata/MIGTEST/controlfile/controlfile2.ctl
Finished restore at 20-DEC-21

released channel: c1



Now I can mount the database


Restore the datafile for the database in OCI


Since the location in OCI is different.

My on-premise database  "/home/oracle/app/oracle/oradata/"
My OCI database  "/u01/app/oracle/oradata/"

I am going to create a script to set newname from my datafiles to restore to.




set linesize 160
set pagesize 0

SELECT REPLACE(file_name,'/home/oracle/app/oracle/oradata/','/u01/app/oracle/oradata/') "Changes"
     FROM (select
           'set newname for datafile ' || file# || ' to ' || '''' || name || '''' || ';' file_name
           from v$datafile
           )
;

Which will create the script that sets "new name for my datafiles"
I just need to execute in RMAN within a run block.

run {
set newname ....
  }

Now I configure the channels just like I did in the for my on premise (unless they are are already set from the controlfile).



## Default device type is tape
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE'; 

## Backup using the library and config file we just installed
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ocicloud/lib/libopc.so,SBT_PARMS=(OPC_PFILE=/home/oracle/ocicloud/config/migtestbackup.ora)';

## Backup with 4 channels to a compressed backupset
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;

## Use medium compression since this is included in the license for the module.
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM';

## Encrypt the backup being sent, this is mandatory for writing to the cloud.
CONFIGURE ENCRYPTION FOR DATABASE ON;


Now we can restore and recover the database and switch to the new copy of the datafiles.


run {
  restore database; 
  recovery database;
  switch datafile all;
    }


And finally (if we want to start it up) open it resetlogs.

RMAN> alter database open resetlogs;

Statement processed

RMAN>


That's all there is to it.

Tuesday, November 16, 2021

ZFSSA now offers immutable snapshots

 The latest ZFSSA software release (as of this post) is OS8.8.39

 This release contains the ability to make both scheduled snapshots and manual snapshots immutable, and I will go through how this works in this post.



New Authorizations

By default non-root users are not authorized to create scheduled locked snapshots, or manual locked snapshots and you will see the message below.




There are 3 new authorizations added to support Snapshot immutability.  The authorizations are

  • releaseSnapRetention - This allows the role to release a snapshot from it's retention hold
  • scheduleLockedSnap - This allows the role to schedule a locked snapshot
  • retainSnap                   - This allows the role to create a manual locked snapshot

In order to show how this works I created a new role "Security_Admin" and granted this role the new authorizations.

You can see that the "Security_Admin" role has releaseSnapRetention, scheduleLockedSnap and retainSnap authorizations which reside under the "Projects and shares" scope.





I then added the new role "Security_Admin" to my administration user.  This limits who has the authority to create and change the status on the immutable snapshots.


Create a Manual Locked Snapshot (BUI) 

First I am going to create a manual locked snapshot.  Below is the window that appears when I click on the "+" to create the snapshot.
Notice below the name there is a new field "Retention policy". This can be either
  • Off           - There is no retention on this snapshot (normal)
  • Unlocked - There is a locked retention on this snapshot 
I am going to create my manual snapshot with an "unlocked" retention policy



Change the retention setting of a Snapshot (BUI) 

Once I create the manual snapshot, I can see that it has an "unlocked" retention when I click on the edit button.  Here I can update the snapshot and turn the retention policy to "Off" to unlock the snapshot when I am ready to delete it. I can also change the status of a snapshot without a retention to have a retention policy.




Create a Manual Locked Snapshot (CLI) 

1) Navigate to the share or project you want to snapshot.

zfssim:shares NFSbackups> select NFS_immutable
zfssim:shares NFSbackups/NFS_immutable>

 
 2) Enter snapshots
zfssim:shares NFSbackups> snapshots
zfssim:shares NFSbackups/NFS_immutable snapshots>


3) Use the snapshot command followed by a "-r" to set the retention lock, and set the new snapshot name

zfssim:shares NFSbackups/NFS_immutable snapshots> snapshot -r Save_until_Jan_1_2022
zfssim:shares NFSbackups/NFS_immutable snapshots>

4) You can use the list command to see the snapshot, and then select the snapshot
zfssim:shares objectstore> select rmanbackups
zfssim:shares objectstore/rmanbackups>

5) The "show" command will display the settings for the snapshot, and you will see that has a retentionpolicy of "unlocked"

zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> show
Properties:
                      creation = Tue Nov 16 2021 20:35:25 GMT+0000 (UTC)
                     numclones = 0
                        isauto = false
               retentionpolicy = unlocked
                          pool = generalpool1
                canonical_name = generalpool1/local/NFSbackups/NFS_immutable@Save_until_Jan_1_2022
                    shadowsnap = false
                  space_unique = 0
                    space_data = 31K




Change the retention setting of a Snapshot (CLI) 

Continuing from the previous set of commands, with the "show" I can see the status of the retention lock.
Using the "set retentionpolicy={off | unlocked}" you can change the status of a snapshot.

Below is the example when I turned the retention policy to off for the snapshot I took in the prior example.


zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> get retentionpolicy
               retentionpolicy = unlocked
zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> set retentionpolicy=off
               retentionpolicy = off (uncommitted)
zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> get retentionpolicy
               retentionpolicy = off (uncommitted)
zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> commit
zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> show
Properties:
                      creation = Tue Nov 16 2021 20:35:25 GMT+0000 (UTC)
                     numclones = 0
                        isauto = false
               retentionpolicy = off
                          pool = generalpool1
                canonical_name = generalpool1/local/NFSbackups/NFS_immutable@Save_until_Jan_1_2022
                    shadowsnap = false
                  space_unique = 0
                    space_data = 31K

Children:
                          backups => Configure Cloud Backups
                          targets => List snapshot parents per target

zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022>



Deleting a Manual Locked Snapshot

BUI 

When you delete a manual snapshot that has a retention policy, you will receive an error screen once you click through the "are you sure" message. Below is the message that will appear if the snapshot still has a retention lock.



In order to allow the snapshot to be deleted, you need to edit the snapshot, and set the retention to "Off".  Once you remove the retention lock the snapshot can be deleted.

CLI

You will receive an error when trying to delete the snapshot. You must release the lock (or in the case of a schedule snapshot) wait for it to roll off.
zfssim:shares NFSbackups/NFS_immutable@Save_until_Jan_1_2022> destroy
This will destroy all data in "Save_until_Jan_1_2022"! Are you sure? (Y/N) y
error: The action could not be completed because the target 'NFSbackups/NFS_immutable@Save_until_Jan_1_2022' is in use. It cannot be modified while it, or its children, are actively changing. Make sure no other users are editing the
       share configuration and try again. If this problem persists, contact your service provider.




Enable Scheduled Locked Snapshots (BUI)

The next step is to enable scheduled locked snapshots. You will notice (highlighted below) that there is a new option to enable the retention policy for locked scheduled snapshots under the project and share.

shares --> share/project --> snapshots




Create Scheduled Locked Snapshots (BUI)

To create a schedule snapshot that is locked, you will noticed there are  addition fields on the scheduling popup.  You have the ability to schedule a snapshot with retention either "Off" or "Locked".  When schedule with "Locked" you must also decide on how many of the "kept" snapshots will be locked. Below I am scheduling snapshots every half hour.  5 snapshots will be kept, and the most recent 3 snapshots will be locked (since I chose locked).


Viewing retention status of scheduled snapshots (BUI)


Using the schedule from above (5 snapshots, 3 of which are locked), below is what I am seeing after it has been executing for awhile.  I chose one of the 3 most recent snapshots and I can see that it has a status of "locked" and I am unable to change that status.






Deleting locked scheduled snapshots (BUI) - Not allowed

Now I am going to try to delete the schedule that contains locked snapshots.  I click on the delete button and hit apply. I get a message saying the snaps will be converted to manual snapshots.


I click on "CONVERT" but it won't let me convert them to manual snapshots.


If I try to change the schedule to have the snapshots no longer be "Locked", I get the same message.



Enable Scheduled Locked Snapshots (CLI)

I navigated through the CLI and got to the share that I wanted to create a scheduled, locked snapshot for.  I first need to make sure the property "snap" is set. In my case it wasn't so I set the value and committed the change.


zfssim:shares NFSbackups/nfstest> get snapret_enabled
               snapret_enabled = false (inherited)
zfssim:shares NFSbackups/nfstest> set snapret_enabled=true
               snapret_enabled = true (uncommitted)
zfssim:shares NFSbackups/nfstest> commit
zfssim:shares NFSbackups/nfstest> get snapret_enabled
               snapret_enabled = true
zfssim:shares NFSbackups/nfstest>



Create Schedule Locked Snapshots (CLI)

Navigate to the share --> snapshots --> automatic

Once there create a new snapshot, and set the properties for the snapshot.
In order to make this a locked snapshot, you need to set the property "retentionpolicy" to "locked".

Below is the steps I followed to create a daily snapshot, kept for 35 days, and immutable for 30 days.



 zfssim:shares NFSbackups/nfstest>
zfssim:shares NFSbackups/nfstest> snapshots
zfssim:shares NFSbackups/nfstest snapshots> automatic
zfssim:shares NFSbackups/nfstest snapshots automatic> create
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> show
Properties:
                     frequency = (unset)
                           day = (unset)
                          hour = (unset)
                        minute = (unset)
                          keep = 0
                 retentionhold = 0
               retentionpolicy = off

zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> set frequency=day
                     frequency = day (uncommitted)
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> set hour=06
                          hour = 06 (uncommitted)
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> set minute=00
                        minute = 00 (uncommitted)
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> set keep=35
                          keep = 35 (uncommitted)
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> set retentionhold=30
                 retentionhold = 30 (uncommitted)
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> set retentionpolicy=locked
               retentionpolicy = locked (uncommitted)
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> show
Properties:
                     frequency = day (uncommitted)
                           day = (unset)
                          hour = 06 (uncommitted)
                        minute = 00 (uncommitted)
                          keep = 35 (uncommitted)
                 retentionhold = 30 (uncommitted)
               retentionpolicy = locked (uncommitted)

zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)>
zfssim:shares NFSbackups/nfstest snapshots automatic-000 (uncommitted)> commit
zfssim:shares NFSbackups/nfstest snapshots automatic> show
Properties:
                       convert = false

Automatics:

NAME           FREQUENCY      DAY             HH:MM KEEP
automatic-000  day            -               06:00   35


 


Viewing retention status of scheduled snapshots (CLI)


Below I listed out the snapshots that were automatically created. I can see that the snapshot chose has a "retentionpolicy" of "locked" and this lock will be removed according to the schedule.


zfssim:shares NFSbackups/zfsshare> snapshots
zfssim:shares NFSbackups/zfsshare snapshots> list
.auto-Bihourly_snapshots-20211116T193000UTC
.auto-Bihourly_snapshots-20211116T200000UTC
.auto-Bihourly_snapshots-20211116T203000UTC
.auto-Bihourly_snapshots-20211116T210000UTC
.auto-Bihourly_snapshots-20211116T213000UTC
zfssim:shares NFSbackups/zfsshare snapshots> select .auto-Bihourly_snapshots-20211116T203000UTC
zfssim:shares NFSbackups/zfsshare@.auto-Bihourly_snapshots-20211116T203000UTC> show
Properties:
                      creation = Tue Nov 16 2021 20:30:00 GMT+0000 (UTC)
                     numclones = 0
                        isauto = true
               retentionpolicy = locked
                          pool = generalpool1
                canonical_name = generalpool1/local/NFSbackups/zfsshare@.auto-Bihourly_snapshots-20211116T203000UTC
                    shadowsnap = false
                  space_unique = 0
                    space_data = 1.22G

Children:
                          backups => Configure Cloud Backups
                          targets => List snapshot parents per target

zfssim:shares NFSbackups/zfsshare@.auto-Bihourly_snapshots-20211116T203000UTC>




BONUS : 


In the audit logs you can see the changes occur, and who made them.. I highlighted where I changed status of one of the shares from Unlocked to OFF and from Off to Unlocked.





Sunday, October 17, 2021

ZFS now supports Object Store Pre-Authenticated Requests

ZFS now supports Pre-Authenticated requests which can be useful for loading data into your data warehouse.


Version OS8.8.36 of the ZFS software was recently released, and one of the most interesting features is the support for Pre-Authenticated Requests. 

Here is the note "On-premise Object Storage Best Practices and Recommended Use Cases". This Document outlines how to use the new Object Store features, some of which I will cover in future posts.

Here is my post on configuring your database to point to ZFS as an object store. It is necessary to  configure ZFS as an object store if you want to do the same testing.

Here is my post on configuring your database to access ZFS as an object store.  This document walks through how to configure DBMS_CLOUD in an 19c+ database.

By going through these notes you can reach the same point that I am with my sample database and ZFS.  Below is the environment that I will be using for this demo.

Environment:

  • Oracle Database 21c (though 19c would work just as well)
    • I have updated the DATABASE_PROPERTIES to point to my wallet for SSL certificates.
    • I added the unsigned SSL certificate for my ZFS simulator HTTPS server to the wallet.
    • I have updated the C##CLOUD$SERVICE.dbms_cloud_store table to point to my ZFS appliance using the ORACLE_BMC authentication method.
    • I installed the sales schema into my account.
  • I am running the ZFS simulator
    • I updated the release of the simulator to OS8.8.36 (2013.06.05.8.36 of the ZFS software.
    • I created a user "salesdata" as a local user
    • I created a share named "salesupload" that is owned by salesdata.
    • The share "salesupload" is configured read/write using the OCI API within HTTP
    • I added the user "salesdata" and it's public SSH keys for authentication with OCI Protocol within HTTP.
  • I have the OCI Client tool installed
    • I installed the OCI client tool
    • I configured an entry for my object store in the ~/.oci/config file called salesdata

Pre-Authenticated Requests for uploading files


The first part of this post will go through creating a request for uploading files to an object store.
In my example, I will be uploading the sales.dat file which comes with the sample sales history schema.

Step 1: Create the bucket to store the files.


The first step is to create a bucket that  I am going to use to store the files that are being uploaded to the object store.  The beauty of using an object store, is that I can have many buckets on the same share that are isolated from each other.

I am going to call my bucket "salesdrop".

Below is the OCI client call I am going to use to create my bucket "salesdrop".


[oracle@oracle-server]$ oci os bucket create --config-file ~/.oci/config --profile SALESDATA --compartment-id salesupload --namespace-name salesupload --endpoint http://10.0.0.231/oci   --name salesdrop
{
  "data": {
    "approximate-count": null,
    "approximate-size": null,
    "compartment-id": "salesupload",
    "created-by": "salesdata",
    "defined-tags": null,
    "etag": "b7ced3b97859a3cc22a23670fc59a535",
    "freeform-tags": null,
    "id": null,
    "is-read-only": null,
    "kms-key-id": null,
    "metadata": null,
    "name": "salesdrop",
    "namespace": "salesupload",
    "object-events-enabled": null,
    "object-lifecycle-policy-etag": null,
    "public-access-type": "NoPublicAccess",
    "replication-enabled": null,
    "storage-tier": "Standard",
    "time-created": "2021-10-17T19:06:47+00:00",
    "versioning": "Disabled"
  },
  "etag": "b7ced3b97859a3cc22a23670fc59a535"
}


Step 2: Create a Pre-Authenticated URL for this bucket


Below is my OCI client call, and the what the parameters mean.


oci os preauth-request create --config-file ~/.oci/config --profile SALESDATA   --namespace-name salesupload --endpoint http://10.0.0.231/oci --bucket-name salesdrop --name upload_sales_data --access-type AnyObjectWrite --time-expires="2022-11-21T23:00:00+00:00"

To go through the parameter used they are
  • config-file: Location of the configuration file
  • profile: Entry to use within the configuration file (if not the default)
  • namespace-name: For ZFS this is the share name
  • endpoint: This is the URL for the ZFS http server + "/oci" to use the OCI API
  • bucket-name: Bucket to create the Pre-Authenticated Request for.
  • name: Identifying name given to this request
  • access-type: What type of Pre-Authenticated request to create
  • time-expires: When will this URL expire? This is mandatory.
Now to execute my request and create the URL.

[oracle@oracle-server ~]$ oci os preauth-request create --config-file ~/.oci/config --profile SALESDATA   --namespace-name salesupload --endpoint http://10.0.0.231/oci --bucket-name salesdrop  --name upload_sales_data --access-type AnyObjectWrite --time-expires="2022-11-21T23:00:00+00:00"
{
  "data": {
    "access-type": "AnyObjectWrite",
    "access-uri": "/oci/p/CQmkSnXYrLcVgUnmRhuOmMXGTzDEJrf/n/salesupload/b/salesdrop/o/",
    "id": "11c01b5c-92d8-4c2d-8cba-d9ec4e2649c5",
    "name": "upload_sales_data",
    "object-name": null,
    "time-created": "2021-10-17T19:15:32+00:00",
    "time-expires": "2022-11-21T22:59:59+00:00"
  }
}


My request was successful, and I can see the URL that was created.  I just need to add the access-uri to the end of  HTTP host URL.

http://10.0.0.231/oci/p/CQmkSnXYrLcVgUnmRhuOmMXGTzDEJrf/n/salesupload/b/salesdrop/o/


Step 3: Upload my file

Now I am going to upload the file from my Windows PC using curl.
The file "sh_sales.dat" is on my d: drive.

d:\> curl -X PUT --data-binary '@d:\sh_sales.dat' http://10.0.0.231/oci/p/CQmkSnXYrLcVgUnmRhuOmMXGTzDEJrf/n/salesupload/b/salesdrop/o/loadfiles/sales_history_05012021.dat
d:\> 

No errors. Let's check with and make sure the file got uploaded using the OCI Client tool

[oracle@oracle-server ~]$ oci os object list --endpoint http://10.0.0.231/oci --namespace-name salesupload   --config-file ~/.oci/config --profile SALESDATA     --bucket-name salesdrop --fields name,size,timeCreated
{
  "data": [
    {
      "etag": null,
      "md5": null,
      "name": "loadfiles/sales_history_05012021.dat",
      "size": 55180902,
      "time-created": "2021-10-17T19:35:34+00:00",
      "time-modified": null
    }
  ],
  "prefixes": []
}

I can see the file is there, and the size is 55MB.

Now where can you go with this ? Below is a diagram of how the Oracle IOT cloud can be used as a hub for datafiles from IOT. You can do the same thing by having all your IOT devices "drop" their data onto a central object store (hosted on ZFS), then filtered and loaded into a database.


Pre-Authenticated Requests for loading files

The part of this post is going to show you how to use Pre-Authenticated Requests to load data into your database.

First I wanted to do shout out to @thatjeffsmith. Jeff Smith is the product manager for SQL Developer, and he has a blog http://www.thatjeffsmith.com where he constantly blogs about SQL Developer and all the great work his team is doing.
I saw one of his posts on Pre-Authenticated Requests  to load data (which you can find here), and I realized that you can do almost the same things on any version of 19c+ with the object store on ZFS.

I am going to go through most of the same steps Jeff did in his post.

Step 1:Create the Pre-Authenticated Request to read the object.

Jeff does this in the Console, but I am going to do it with the OCI Client tool.

The command is similar to the command I used to create the "upload" request.
I am going to use a different access-type. I am going to use "ObjectRead" and create a request that points to the object that was uploaded.

[oracle@oracle-server]$ oci os preauth-request create --config-file ~/.oci/config --profile SALESDATA   --namespace-name salesupload --endpoint http://10.0.0.231/oci --bucket-name salesdrop  --name upload_sales_data --access-type ObjectRead --time-expires="2022-11-21T23:00:00+00:00" --object-name loadfiles/sales_history_05012021.dat
{
  "data": {
    "access-type": "ObjectRead",
    "access-uri": "/oci/p/apVWoQmeWWtireCzUqEjGBTRWQwotro/n/salesupload/b/salesdrop/o/loadfiles/sales_history_05012021.dat",
    "id": "547227b4-73b0-4980-bb94-ab5ee87d4c81",
    "name": "upload_sales_data",
    "object-name": "loadfiles/sales_history_05012021.dat",
    "time-created": "2021-10-17T19:56:45+00:00",
    "time-expires": "2022-11-21T22:59:59+00:00"
  }
}


Now I have my request URL.

http://10.0.0.231//oci/p/apVWoQmeWWtireCzUqEjGBTRWQwotro/n/salesupload/b/salesdrop/o/loadfiles/sales_history_05012021.dat


Step 2: Load the data

Now back to Jeff's example, I am going to log onto my database and load the data.

First I am going to count the rows in my table, then check again after.

SQL> select count(1) from sales;

  COUNT(1)
----------
         0

 SQL> BEGIN
  DBMS_CLOUD.COPY_DATA(
     table_name =>'SALES',
     file_uri_list =>'https://10.0.0.231/oci/p/apVWoQmeWWtireCzUqEjGBTRWQwotro/n/salesupload/b/salesdrop/o/loadfiles/sales_history_05012021.dat',
     format => json_object('delimiter' VALUE '|') );
END;


PL/SQL procedure successfully completed.


SQL> select count(1) from sales;

  COUNT(1)
----------
   1016271

SQL>


I can see that 1 Million rows were successfully loaded into the table.


Step 3: Verify through USER_LOAD_OPERATIONS

Now, like Jeff did with his example, I am going to look at the view USER_LOAD_OPERATIONS to see the information about my load job.


col id format 999
col type format a8
col status format a10
col start_time format a15
col owner_name format a10
col table_name format a20
col file_uri_list format a70
set linesize 160


select
        id,
        type,
        to_char(update_time,'mm/dd/yy hh24:mi:ss') update_time,
        status,
        owner_name,
        table_name,
        substr(file_uri_list,60,160) File_uri_list,
        rows_loaded
from
        user_load_operations
where status='COMPLETED';

SQL> 

  ID TYPE     UPDATE_TIME       STATUS     OWNER_NAME TABLE_NAME           FILE_URI_LIST                                                          ROWS_LOADED
---- -------- ----------------- ---------- ---------- -------------------- ---------------------------------------------------------------------- -----------
   3 COPY     10/17/21 16:13:21 COMPLETED  BGRENN     SALES                salesupload/b/salesdrop/o/loadfiles/sales_history_05012021.dat             1016271


Other Pre-Authenticated Requests 


There are 2 other "access-types" for Pre-Authenticated requests.

  • ObjectReadWrite: This will give both read and write to a specific object. 
  • ObjectWrite: This give write only access to a specific object (rather than having full access to the bucket).
If you try to use the Pre-Authenticated Requested to anything other than object granted you get an error message.

{"code": "NotAuthenticated", "message": "The required information to complete authentication was not provided"}

List Pre-Authenticated Requests 

You can list all of your Pre-Authenticated Requests to keep a handle on them.


[oracle@oracle-server ~]$ oci os preauth-request list --config-file ~/.oci/config --profile SALESDATA   --namespace-name salesupload --endpoint http://10.0.0.231/oci --bucket-name salesdrop
{
  "data": [
    {
      "access-type": "AnyObjectWrite",
      "id": "11c01b5c-92d8-4c2d-8cba-d9ec4e2649c5",
      "name": "upload_sales_data",
      "object-name": null,
      "time-created": "2021-10-17T19:15:32+00:00",
      "time-expires": "2022-11-21T22:59:59+00:00"
    },
    {
      "access-type": "ObjectRead",
      "id": "547227b4-73b0-4980-bb94-ab5ee87d4c81",
      "name": "load_sales_data",
      "object-name": "loadfiles/sales_history_05012021.dat",
      "time-created": "2021-10-17T19:56:45+00:00",
      "time-expires": "2022-11-21T22:59:59+00:00"
    },
    {
      "access-type": "ObjectReadWrite",
      "id": "87b4fe97-3e2e-4b22-96aa-a7e3b566dc59",
      "name": "read_write_sales_data",
      "object-name": "loadfiles/sales_history_06012021.dat",
      "time-created": "2021-10-17T20:37:23+00:00",
      "time-expires": "2022-11-21T22:59:59+00:00"
    },
    {
      "access-type": "ObjectWrite",
      "id": "828a0651-60f7-4d2a-998c-b3518e1bfa92",
      "name": "write_sales_data",
      "object-name": "loadfiles/sales_history_07012021.dat",
      "time-created": "2021-10-17T20:40:08+00:00",
      "time-expires": "2022-11-21T22:59:59+00:00"
    }
  ]
}


Get Detail on a Pre-Authenticated Requests 


If you want the detail of a specific Pre-Authenticated Request you can use the "get" option and include the --par-id (which is the ID from the list request command).

[oracle@oracle-server ~]$ oci os preauth-request get --config-file ~/.oci/config --profile SALESDATA   --namespace-name salesupload --endpoint http://10.0.0.231/oci --bucket-name salesdrop   --par-id 828a0651-60f7-4d2a-998c-b3518e1bfa92
{
  "data": {
    "access-type": "ObjectWrite",
    "id": "828a0651-60f7-4d2a-998c-b3518e1bfa92",
    "name": "upload_sales_data",
    "object-name": "loadfiles/sales_history_07012021.dat",
    "time-created": "2021-10-17T20:40:08+00:00",
    "time-expires": "2022-11-21T22:59:59+00:00"
  }
}


NOTE: this does not give you the URL.

Delete a Pre-Authenticated Requests 


Finally you can delete a Pre-Authenticated Request if it is no longer needed using the par-id of the request.


[oracle@oracle-server ~]$ oci os preauth-request delete --config-file ~/.oci/config --profile SALESDATA   --namespace-name salesupload --endpoint http://10.0.0.231/oci --bucket-name salesdrop   --par-id 828a0651-60f7-4d2a-998c-b3518e1bfa92
Are you sure you want to delete this resource? [y/N]: y





Hopefully this gives you an idea of all the things you can do with Pre-Authenticated URLs.