Tuesday, February 8, 2022

Managing your ZDLRA replication queue remotely

 With the rise of Cyber Crime, more and more companies are looking at an architecture with a second backup copy that is protected with an airgap.   Below is the common architecture that I am seeing.


In this post I will walk through an example of how to implement a simple Java program that performs the tasks necessary to manage the airgap for a ZDLRA that is implemented in a cyber vault (DC1 Vault in the picture).  Feel free to use this as a starting point to automate the process.

Commands

There are 3 commands that I need to be able execute remotely

  • PAUSE      -This will pause the replication server that I configured
  • RESUME - This will resume the replication server that I configured
  • QUERY    - This will query the queue on the upstream to determine how much is left in the queue.
First however I need to configure the parameters to execute the calls.

Config file (airgap.config).

I create config file to customize the script for my environment. Below are the parameters that I needed to connect to the ZDLRA and execute the commands.
  • HOST                    - This is name of the scan listener on upstream ZDLRA.
  • PORT                     - This is the Sqlnet port being used to connect to the upstream ZDLRA
  • SERVICE_NAME - Service name of the database on the upstream ZDLRA
  • USERNAME         - The username to connect to the upstream database
  • PASSWORD          - Password for the user. Feel free to encrypt this in java.
  • REPLICATION_SERVER - Replication server to manage

Below is what my config file looks like.

airgap.host=oracle-19c-test-tde
airgap.port=1521
airgap.service_name=ocipdb
airgap.username=bgrenn
airgap.password=oracle
airgap.replication_server=replairgap


Java code (airgap.java).

Java snippet start

The start of the Java Code will import the functions necessary and set up my class


import java.sql.*;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.FileInputStream;
import java.util.Date;
import java.util.Properties;

// Create a arigap class
public class airgap {

   private Properties prop = new Properties();


Java snippet get properties

The first method will get the airgap properties from the property files so that I can use them in the rest of the methods.

// Create a get_airgap_properties method
  public void get_airgap_properties()
        {
                String fileName = "airgap.config";
                try (FileInputStream fis = new FileInputStream(fileName)) {
                    prop.load(fis);
                } catch (FileNotFoundException ex) {
                    System.out.println("cannot find config file airgap.config");
                } catch (IOException ex) {
                    System.out.println("unknown issue finding config file airgap.config");
                }
        }



Java snippet pause replication server

The code below will connect to the database and execute DBMS_RA.PAUSE_REPLICATION_SERVER


// Create a pause_replication  method
  public void pause_replication()
        {
                try     {
                        //Loading driver
                        Class.forName("oracle.jdbc.driver.OracleDriver");

                        //creating connection
                        Connection con = DriverManager.getConnection
                                        ("jdbc:oracle:thin:@//"+
                                         prop.getProperty("airgap.host")+":"+
                                         prop.getProperty("airgap.port")+"/"+
                                         prop.getProperty("airgap.service_name"),
                                         prop.getProperty("airgap.username"),
                                         prop.getProperty("airgap.password"));

                        CallableStatement cs=con.prepareCall("{call dbms_ra.pause_replication_server(?)}");

                        //Set IN Parameters
                        String in1 = prop.getProperty("airgap.replication_server");
                        cs.setString(1,in1);

                        ResultSet rs = cs.executeQuery();   //executing statement


                        con.close();    //closing connection
                        System.out.println("replication server '"+ prop.getProperty("airgap.replication_server")+"' paused");
                        }
                catch(Exception e)      {
                        e.printStackTrace();
                                        }
        }



Java snippet resume replication server

The code below will connect to the database and execute DBMS_RA.RESUME_REPLICATION_SERVER


// Create a pause_replication  method
  public void resume_replication()
        {
                try     {
                        //Loading driver
                        Class.forName("oracle.jdbc.driver.OracleDriver");

                        //creating connection
                        Connection con = DriverManager.getConnection
                                        ("jdbc:oracle:thin:@//"+
                                         prop.getProperty("airgap.host")+":"+
                                         prop.getProperty("airgap.port")+"/"+
                                         prop.getProperty("airgap.service_name"),
                                         prop.getProperty("airgap.username"),
                                         prop.getProperty("airgap.password"));

                        CallableStatement cs=con.prepareCall("{call dbms_ra.resume_replication_server(?)}");

                        //Set IN Parameters
                        String in1 = prop.getProperty("airgap.replication_server");
                        cs.setString(1,in1);

                        ResultSet rs = cs.executeQuery();   //executing statement


                        con.close();    //closing connection
                        System.out.println("replication server '"+ prop.getProperty("airgap.replication_server")+"' resumed");
                        }
                catch(Exception e)      {
                        e.printStackTrace();
                                        }
        }


Java snippet query replication server

The java code below will query the replication queue in the upstream ZDLRA and return 4 columns
  • REPLICATION SERVER - name of the replication server
  • TASKS QUEUED - Number of tasks in the queue to be replicated
  • TOTAL GB QUEUED - Amount of data in the queue
  • MINUTES IN QUEUE - The number of minutes the oldest replication piece has been in the queue.
The last piece of information can be very useful to tell you how current the replication is. With real-time redo, the queue may never be empty.

// Create a queue_select method
  public void queue_select()
        {
                try     {
                        //Loading driver
                        Class.forName("oracle.jdbc.driver.OracleDriver");

                        //creating connection
                        Connection con = DriverManager.getConnection
                                        ("jdbc:oracle:thin:@//"+
                                         prop.getProperty("airgap.host")+":"+
                                         prop.getProperty("airgap.port")+"/"+
                                         prop.getProperty("airgap.service_name"),
                                         prop.getProperty("airgap.username"),
                                         prop.getProperty("airgap.password"));

                        Statement s=con.createStatement();      //creating statement

                        ResultSet rs=s.executeQuery("select replication_server_name,"+
                                                    "       count(*)  tasks_queued,"+
                                                    "       trunc(sum(total)/1024/1024/1024,0) AS TOTAL_GB_QUEUED,"+
                                                    "       round("+
                                                    "         (cast(current_timestamp as date) - cast(min(start_time) as date))"+
                                                    "             * 24 * 60"+
                                                    "         ) as queue_minutes "+
                                                    "from RA_SBT_TASK "+
                                                    "    join ra_replication_config on (lib_name = SBT_library_name) "+
                                                    "          where archived = 'N'"+
                                                    "group by replication_server_name");   //executing statement

                        System.out.println("Replication Server,Tasks Queued,Total GB Queued,Minutes in Queue");

                        while(rs.next()){
                                System.out.println(rs.getString(1)+","+
                                                   rs.getInt(2)+","+
                                                   rs.getInt(3)+","+
                                                   rs.getString(4));
                                        }

                        con.close();    //closing connection
                        }
                catch(Exception e)      {
                        e.printStackTrace();
                                        }
        }



Java snippet main section

Below is the main section, and as you can see you can pass one of the 3 parameters mentioned earlier.





  public static void main(String[] args)
        {

//      import java.sql.*;
         airgap airgap = new airgap();   // Create a airgap object


         airgap.get_airgap_properties();      // Call the queue_select() method
         switch(args[0]) {

                case "resume":
                        airgap.resume_replication();      // Call the resume_replication() method
                        break;
                case "pause":
                        airgap.pause_replication();      // Call the pause_replication() method
                        break;
                case "query":
                        airgap.queue_select();      // Call the queue_select() method
                        break;
                default:
                         System.out.println("parameter must be one of 'resume','pause' or 'query'");
                        }
        }
}


Executing the Java code (airgap.class).

Now if you take the snipets above and put them in a file airgap.java you can compile them into a class file.

javac airgap.java
This creates a class file airgap.class

In order to connect to my oracle database, I downloaded the jdbc driver.

"ojdbc8.jar"

Now I can execute it with the 3 parameters 

$ java -Djava.security.egd=file:/dev/../dev/urandom -cp ojdbc8.jar:. airgap pause
replication server 'replairgap' paused

$ java -Djava.security.egd=file:/dev/../dev/urandom -cp ojdbc8.jar:. airgap resume
replication server 'replairgap' resumed

$ java -Djava.security.egd=file:/dev/../dev/urandom -cp ojdbc8.jar:. airgap query
Replication Server,Tasks Queued,Total GB Queued,Minutes in Queue
ra_replication_config,4,95,58


It's that easy to create a simple java program that can manage your replication server from within an Airgap.


Thursday, December 23, 2021

Cataloging backups and recovering an Oracle Database from the OCI object store

   This is the fourth and final post of a multi-part blog series walking through how to copy your TDE encrypted on premise Oracle Database to an OCI instance in the oracle cloud. This blog post will focus on how to restore your database from the object store, when the backup pieces are not available from your controlfile. 





There a few reasons why this might be the case.

  • The backups were written to the ZDLRA directly.
  • You are using an RMAN catalog, and they have aged off the controlfile.
  • They are "keep" backups which will be stored in the RMAN catalog.
  • You had to rebuild the controlfile, and lost history of backups.
Whatever the reason, there is way to find out what backups are in the Object for your database, and you will be able to catalog them.

NOTE: You can use this same script to delete old backups directly if you've lost your catalog entries.

When you download the Oracle Cloud Backup installation zip file, and execute the "oci_install.jar" command to download the library you will find 5 extra files in the /lib directory with the "libopc.so" file that is used by the RMAN channel. The 2 we are going to use are 
  • odbsrmt.py             --> python script to manage the contents of the object store bucket
  • python_readme.txt --> Documentation for how to use the above python script.

Step #1 Execute odbsrmt.py to get a listing of your backup pieces.

NOTE: The python script uses python 2.x and will not work with python 3.x.  Python 3.x is typically the default version in your path, and you might have to find the 2.x version on your system. For my system this means executing "python2" rather than "python"

If I execute the script without any parameters, I can see what parameters are expected.



[oracle@oracle-19c-test-tde lib]$ python2 odbsrmt.py
usage: odbsrmt.py [-h] --mode
                  {report,rman-listfile,garbage-collection,delete,recall}
                  [--ocitype {classic,swift,bmc,archive}]
                  [--credential CREDENTIAL] [--token TOKEN] --host HOST
                  [--base BASE] [--forcename FORCENAME]
                  [--format {text,xml,json}] [--dbid DBID]
                  [--container CONTAINER] [--dir DIR] [--prefix PREFIX]
                  [--untildate UNTILDATE] [--exclude_deferred]
                  [--thread THREAD] [--proxyhost PROXYHOST]
                  [--proxyport PROXYPORT] [--tocid TOCID] [--uocid UOCID]
                  [--pubfingerprint PUBFINGERPRINT] [--pvtkeyfile PVTKEYFILE]
                  [--skip_check_status] [--debug]
odbsrmt.py: error: argument --mode is required

Now let's go through the most common parameters I am going to use to report on my backups




And now to execute the command to see some of the report.


python2  odbsrmt.py --mode report --ocitype bmc  --host https://objectstorage.us-ashburn-1.oraclecloud.com --dir /home/oracle/ocicloud/report --base mydbreport --pvtkeyfile  /home/oracle/ocicloud/myprivatekey.ppk --pubfingerprint 6d:f9:57:d5:ff:b1:c0:98:81:90:1e:6e:08:0f:d0:69 --tocid ocid1.tenancy.oc1..aaaaaaaanz4trskw6jm57cz2fztoasatto3i6z4h33gzfb3pmei5vvnoq --uocid ocid1.user.oc1..aaaaaaaae2mlwyke4gvd7kzxv5zxgg3k2dlcwvubv7vjy6jvbgsaouxq --container migest_backups  --dbid 301925655


And this will give me the following output in my report file.

FileName
Container                Dbname         Dbid        FileSize          LastModified                BackupType                  Incremental  Compressed   Encrypted
220h9q5f_66_1_1
migest_backups           OCITEST        301925655   72876032          2021-12-21 19:37:33         ArchivedLog                 false        true         true
230h9q5g_67_1_1
migest_backups           OCITEST        301925655   75759616          2021-12-21 19:37:32         ArchivedLog                 false        true         true
240h9q5g_68_1_1
migest_backups           OCITEST        301925655   54263808          2021-12-21 19:37:12         ArchivedLog                 false        true         true
250h9q5g_69_1_1
migest_backups           OCITEST        301925655   48496640          2021-12-21 19:36:58         ArchivedLog                 false        true         true
260h9q9n_70_1_1
migest_backups           OCITEST        301925655   159645696         2021-12-21 19:42:46         Datafile                    true         true         true
270h9q9n_71_1_1
migest_backups           OCITEST        301925655   408682496         2021-12-21 19:47:04         Datafile                    true         true         true
280h9q9n_72_1_1
migest_backups           OCITEST        301925655   524288            2021-12-21 19:37:46         Datafile                    true         true         true
290h9q9n_73_1_1
migest_backups           OCITEST        301925655   56885248          2021-12-21 19:39:37         Datafile                    true         true         true
2a0h9q9v_74_1_1
migest_backups           OCITEST        301925655   235667456         2021-12-21 19:45:05         Datafile                    true         true         true
2b0h9qdi_75_1_1
migest_backups           OCITEST        301925655   233832448         2021-12-21 19:46:18         Datafile                    true         true         true
2c0h9qjb_76_1_1
migest_backups           OCITEST        301925655   52166656          2021-12-21 19:44:31         Datafile                    true         true         true
2d0h9qmk_77_1_1
migest_backups           OCITEST        301925655   1572864           2021-12-21 19:44:43         Datafile                    true         true         true
2e0h9qn3_78_1_1
migest_backups           OCITEST        301925655   34865152          2021-12-21 19:45:41         Datafile                    true         true         true
2f0h9qns_79_1_1
migest_backups           OCITEST        301925655   524288            2021-12-21 19:45:20         Datafile                    true         true         true
2g0h9qrg_80_1_1
migest_backups           OCITEST        301925655   262144            2021-12-21 19:47:14         ArchivedLog                 false        true         true
c-301925655-20211221-00
migest_backups           OCITEST        301925655   524288            2021-12-21 19:47:22         ControlFile SPFILE          false        true         true
Total Storage: 1.34 GB


You can see that this report contains  the backup pieces I need. 

I am going to use the script (below) and pass it the report name to create the commands to catalog the backup pieces.



And when I execute the above script passing my report file, it produces my commands to catalog the backup pieces.

report file used for catalog scripts   : mydbreport4701.lst


catalog device type 'sbt_tape' backuppiece '220h9q5f_66_1_1';
catalog device type 'sbt_tape' backuppiece '230h9q5g_67_1_1';
catalog device type 'sbt_tape' backuppiece '240h9q5g_68_1_1';
catalog device type 'sbt_tape' backuppiece '250h9q5g_69_1_1';
catalog device type 'sbt_tape' backuppiece '260h9q9n_70_1_1';
catalog device type 'sbt_tape' backuppiece '270h9q9n_71_1_1';
catalog device type 'sbt_tape' backuppiece '280h9q9n_72_1_1';
catalog device type 'sbt_tape' backuppiece '290h9q9n_73_1_1';
catalog device type 'sbt_tape' backuppiece '2a0h9q9v_74_1_1';
catalog device type 'sbt_tape' backuppiece '2b0h9qdi_75_1_1';
catalog device type 'sbt_tape' backuppiece '2c0h9qjb_76_1_1';
catalog device type 'sbt_tape' backuppiece '2d0h9qmk_77_1_1';
catalog device type 'sbt_tape' backuppiece '2e0h9qn3_78_1_1';
catalog device type 'sbt_tape' backuppiece '2f0h9qns_79_1_1';
catalog device type 'sbt_tape' backuppiece '2g0h9qrg_80_1_1';
catalog device type 'sbt_tape' backuppiece 'c-301925655-20211221-00';


Now in RMAN I can execute these commands to catalog the backup pieces from the OCI bucket.

Note : By using "untildate" you  can control the dates that will be reported on.






Wednesday, December 22, 2021

Restoring OKV in the Oracle Cloud to manage your encrypted databases

  This is the third of a multi-part blog series walking through how to copy your TDE encrypted on premise Oracle Database to an OCI instance in the oracle cloud. This blog post will focus on how to restore OKV (Oracle Key Vault) into an instance in OCI to manage your encryption keys, and support restoring an encrypted database.



The first part of this series went through how to migrate a database from on premise to an instance in the cloud using the Oracle Database Cloud Backup Module. You can find it here.

The second part of this series went through how to backup OKV to an immutable OCI bucket leveraging ZFSSA. You can find it here.

I will add to this by restoring from my OKV backup into the Oracle Cloud (OCI), and then restoring my database.


I am going to skip over how to migrate  my database to using OKV. If you are starting at the beginning (small database no encryption), the steps to get to this next blog post are.

  1. Create a new database for testing.
  2. Implement Advanced Security (TDE) which is covered in my post here.
  3. Migrating from a local wallet to OKV which is covered in my post here.
  4. Backup your database to an OCI bucket encrypted, and compressed.
At this point my database (ocitest),  is using my OKV environment on premise, and I have a backup of both my database, and OKV in Object Storage in the Oracle Cloud.


Create a ZFS Image in OCI to restore OKV from Object Store.


Log into OCI (you can do this with the 30 day trial), and create a new instance using the ZFS image. Below you can see that you can find this image under "Oracle images".


Select this image, upload your public key, and create the new instance.

There are a couple of great step-by-step guides to help you get started with the ZFS image in OCI.
I am not going to go through the process, as those 2 documents are extremely thorough, and will give you the detail needed to configure ZFS with attached storage within OCI.

Create an OKV Image in OCI to restore OKV from Object Store.


The next step to restore OKV is to create an OKV image in OCI.  At this point it is CRITICAL to create an image that is the same version of the source OKV backup.  As of writing this post, I am on 21.2, and I will create a 21.2 instance in OCI.


Again there is great documentation on how to do go through this process.  You need to create a "SYSADMIN" user. Since the users within OKV will get replaced during the install, this user will only be used temporarily.  Below are the links to start with.
NOTE:
  • Always deploy the same version in OCI as the backup you are restoring from.
  • The command when first logging into the image to configure it may be different from the video, but the login screen will give you clear instructions.

Configure ZFS as a backup location for OKV


At this point if you follow my last blog post found here, you go through the same series of steps in OCI to configure OKV to use ZFS as a backup location that had been done to configure the original backups.
  • Create the user on the ZFS image to own the backups
  • Log into OKV and save the "public key" for Authentication.
  • Configure SFTP on the ZFS image, and add the "Public Key" for the new user.
  • Configure the OCI Object Store on the ZFS image as a "cloud target" pointing to the same bucket you had written to.
  • Create a new project on the ZFS image with the OKV backup owner as the owner of the project.
  • Configure protocols on the new project to ensure that "SFTP" is read/write.
The steps left NOT completed are
  • Creating a share within the project
  • Creating a backup location within OKV.

Restore the share to the ZFS image in OCI


Now we are ready to restore the backup from the OCI bucket to a share on the ZFS image.
On the ZFS, navigate to "SERVICES" => "Cloud", and within "Cloud" click on the "Backups" tab. Within that tab you will see the ZFS backups that have been sent to the target.
Find the backup that you want, and click on the circular arrow to restore that backup.


This will bring up a popup window where you will choose where to restore the backup to.  Chose the project that you previously created (with the OKV backup user, and "SFTP" protocol enabled"). Give the share a name, and click on "APPLY".


Then once you click on "APPLY" you will see a status popup telling you when it is completed.


When it completes the restore, take note of the share name, and you can configure OKV to restore from this share.

Restore the OKV backup in OCI


Now return to the OKV image in OCI, and navigate to "System" => "Backup and Restore" and create a new backup location, like we had done to create the original backup.
This time enter information for the ZFS image in OCI, and include the destination as "/export/{restored share name}".

Once this is configured click on the "Restore" button, and it will bring up a list of backups that are available to restore from the ZFS share.

Choose the backup you want to use (the backup time will help narrow it down). Click on "Restore" and it will bring up a popup window to enter the "Recovery Passphrase". Enter the passphrase set when OKV was originally installed in your data center, and click on "Restore".

NOTE: The backup is encrypted using the "Recovery Passphrase", and it is critical that you have the original passphrase available to complete this step.


When the restore starts, you will see a message, and OKV will not be available until the restore process completes.


Re-enroll your database  in OCI

Once OKV is restored, the users you created within OKV will be restored. The only items that will be saved are
  • root
  • support
  • "recovery passphrase"
Within OCI where you are restoring your database, you will configure the database environment to start the restore process.  I started by creating a pfile, and some of the directories needed.

audit_file_dest='/u01/app/oracle/admin/ocitest/adump'
audit_trail='db'
compatible='19.0.0'
control_files='/u01/app/oracle/oradata/OCITEST/controlfile/o1_mf_jo6q53rf_.ctl''
db_block_size=8192
db_create_file_dest='/u01/app/oracle/oradata'
db_name='ocitest'
db_recovery_file_dest='/u01/app/oracle/fast_recovery_area'
db_recovery_file_dest_size=32212254720
diagnostic_dest='/u01/app/oracle'
enable_pluggable_database=true
pga_aggregate_target=1547m
processes=300
sga_target=4638m
tde_configuration='KEYSTORE_CONFIGURATION=OKV|FILE'
undo_tablespace='UNDOTBS1'
wallet_root='/u01/app/wallets/ocitest'

NOTE: Since you need OKV to decrypt the RMAN backup of the controlfile, you need to ensure the pfile contains the "WALLET_ROOT" and "TDE_COFIGURATION". 

Within OKV I re-enrolled the endpoint for my database, and I downloaded and installed the "okvclient.jar" in  the "WALLET_ROOT"/okv location.

Now to restore my database, I can use a script, like the script below to
  • Startup nomount
  • Open the wallet pointing to my keys in OKV
  • Set the DBID
  • Allocate the channel
  • Restore the controlfile
  • Mount the database.



sqlplus / as sysdba
SQL> startup nomount;
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "0KV2021!";
SQL> exit


rman target /
RMAN> set dbid=301925655;
RMAN> run {
RMAN> allocate CHANNEL c1 TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so,SBT_PARMS=(OPC_PFILE=/home/oracle/ociconfig/config/ocitestbackup.ora)';
RMAN> restore controlfile from autobackup ;
RMAN> release channel c1;
RMAN> }
RMAN> alter database mount;
Once mounted, I can follow the normal steps to restore my database, and my encryption keys are available.  The backup information for my OCI bucket is in my controlfile.