Configuring Oracle Key Vault with OCI/DB@ for Exadata Dedicated

This blog post will walk through how to configure OCI (or multicloud) to use an existing Oracle Key Vault (OKV) with Exadata Database Service on Dedicated Infrastructure.

This topic has come up a lot lately with customers who need  a central TDE key management system and I wanted to walk through the steps myself and document them for easy access.

 I am going to start at the point where OKV is already installed, the initial passwords have been set for ROOT and an ADMIN account, and I can log into the OKV management console.

NOTE: For my testing, I used the marketplace image of OKV installed in an instance in OCI.

The basic required steps to give the OCI database automation the information it needs to connect to OKV are :

• Configure OKV to enroll databases using the REST API functionality.
• Create a REST API user for OKV
• Create an OCI vault if one doesn't already exist
• Create an encryption key to encrypt a secret
• Store the OKV REST user password in that secret
• Create the Exadata Key Store.

There are a few similarly named objects involved, so I want to explain what each one does before going through the screens.

Object	What I am using it for
Oracle Key Vault (OKV)	The server(s) that will hold the database TDE master encryption keys when a database is configured to use OKV.  OKV is typically configured as a cluster for high availability.
OCI vault	The OCI vault used for the encryption key to encrypt a secret in this walkthrough.  It is not used for the TDE encryption keys.
OCI encryption key	The symmetric key used to encrypt the saved password.
OCI secret	The password for the OKV REST user.
Exadata Key Store	The OCI resource containing the OKV connection IP address(es), REST username, and reference to the password secret for the REST user.

NOTE: The key I create in OCI below protects the password secret. The database TDE keys will be managed in OKV.

Before starting

My screenshots show OKV 21.15. OCI menu names can change, so I have included links to Oracle's documentation throughout the post.

Make sure the Exadata client network can reach OKV. Oracle lists ports 443, 5695, and 5696 for outbound access from the client network. Check the routes, security lists or network security groups, and any firewall on the path. 

I also need OCI permissions to create or use the vault, key, secret, and Key Store, plus an IAM administrator to configure the policies below. For this walkthrough, I will work in the Exadata deployment's region and select the appropriate compartment at each step.

Oracle's main reference is Managing Encryption Keys on External Devices for Exadata Dedicated. Start with the section titled Setting Up Your Oracle Exadata Database Service on Dedicated Infrastructure to Work With Oracle Key Vault.

Step #1 Enable REST access in OKV

First I am going to check that REST access is enabled. Log into OKV with a System Administrator account and go to System → Settings → RESTful Services.

In the screen below, REST access can be disabled, allowed from specific IP addresses, or allowed from all addresses. For a restricted configuration, select IP address(es) and include the actual client IP addresses used by all of the database VM hosts that will access OKV. Include every RAC node, and account for any source address translation in your network.  Save the change.

The supplied screenshot shows All selected. Use an explicit client IP list when restricting REST access.

Oracle documents these settings in Oracle Key Vault General System Administration, under Configuring RESTful Services Utility. Use the section appropriate to your standalone or multi-master cluster deployment.

Step #2a Create the OKV REST user

Next I am going to create a dedicated account for the integration. In OKV, go to Users → Manage Users and create a user such as okv_rest_user. Set its password and retain it securely. This is the password that I will put into the OCI secret.

Create the OKV user that OCI will use for the integration.

Open the user details, select the Create Endpoint privilege, and save the change. That is the REST user privilege specified for Exadata Dedicated in Oracle's integration prerequisites. The screenshot below shows it selected.

Grant Create Endpoint to the integration user.

Step #2b Reset the password for the REST user

Once you create the REST user, you need to reset the password before the first login.

This can be accomplished by logging into the OKV console as the okv_rest_user and you will be prompted to reset the password.

Step #3 Create or select an OCI vault

Now I am ready to move to the OCI Console. Open Identity & Security → Key Management and locate the vaults. Some Console layouts label this entry simply Vault.

If you already have an appropriate vault, select it and continue to the next step. Otherwise:

1. Select the compartment where the vault will be created.
2. Click Create Vault.
3. Enter a descriptive name, such as okv-integration-vault.
4. Select the vault type required by your organization. The virtual private vault option provides a dedicated HSM partition; it is not a requirement just to store this password.
5. Create the vault and wait for it to become available.

See Creating a Vault for the options. Choose the vault type carefully because it cannot be changed after creation.

Step #4 Create the key that will encrypt the secret

Within the vault, open Master Encryption Keys and select Create Key. Below are the settings in my example.

Field	Example setting
Compartment	The compartment chosen for the key
Name	bsg-okv-secret
Protection Mode	Software in this example; select HSM if required by your organization
Key Shape Algorithm	AES
Key Shape Length	256 bits
Import External Key	Off, so OCI generates the key

This key will encrypt the OKV password secret.

Click Create Key and wait for it to become available. Use a symmetric AES key for this secret; an asymmetric key is not supported for secret creation.

Oracle explains the protection modes and creation options in Creating a Master Encryption Key. A key's protection mode cannot be changed after creation.

Step #5 Create the secret containing the OKV password

Next go to Identity & Security → Secret Management → Secrets. Select the compartment and vault, then click Create secret.

I am storing the password that already exists in OKV, so I will use Manual secret generation. Fill in the fields as follows.

Field	What to enter
Create in compartment	The compartment where the secret should reside
Name	okv-rest-password
Description	Password for the OKV REST integration user
Vault compartment and Vault	The vault selected or created in Step #3
Encryption key compartment and Encryption key	The location and name of the AES key from Step #4
Secret generation	Manual secret generation
Secret type template	Plain-Text
Secret contents	Only the exact password for okv_rest_user

The key must belong to the vault selected for the secret. The compartment selectors matter here: the secret, vault, and key can have different compartment locations.

NOTE: Do not include the username, a JSON document, surrounding quotes, or an extra newline in the password value. With Plain-Text selected, enter the password directly. The Console handles the required base64 encoding; base64 itself is not encryption. OCI encrypts the secret using the selected key.

Click Create secret. Once it is active, record its name, compartment, and OCID for reference. I will select this secret in the Key Store form rather than paste the password into that form.

See Creating a Secret. Oracle now documents secrets under the Secret Management Service, which explains why older Vault documentation may direct you to a different page.

Step #6 Configure IAM access

Before creating the Key Store, I need the OCI resources to have access to each other. This is separate from the permissions that let me log into the Console and create the resources.

The following follows the compartment-based dynamic-group approach in Oracle's Exadata Dedicated setup guide. Have your IAM administrator adapt it to your compartment structure and identity domain. Replace every value in angle brackets before using it.

Create the dynamic groups under Identity & Security → Domains → your domain → Dynamic groups.

For a group named OKV-KeyStores, use the OCID of the compartment where the Key Store will reside:

ALL {resource.compartment.id = '<KEYSTORE-COMPARTMENT-OCID>'}

For a group named OKV-Exadata, use the compartment identified for the Exadata infrastructure in Oracle's setup procedure:

ALL {resource.compartment.id = '<EXADATA-INFRASTRUCTURE-COMPARTMENT-OCID>'}

Create the policy in a parent compartment that covers the target compartments. These examples use dynamic groups in the Default identity domain; substitute your domain if different.

Allow dynamic-group 'Default'/'OKV-KeyStores' to use secret-family in compartment <SECRET-COMPARTMENT>
Allow dynamic-group 'Default'/'OKV-Exadata' to use keystores in compartment <KEYSTORE-COMPARTMENT>
Allow service database to read secret-family in compartment <SECRET-COMPARTMENT>

The first statement gives the Key Store group access to the secret. The second lets the Exadata group use the Key Store. The third lets the database service read the secret.

NOTE: These matching rules select resources by compartment; they are not filters for a particular resource type. Use appropriately scoped compartments and have your IAM administrator review the resulting access. If the Exadata infrastructure and VM clusters are in different compartments, verify the resource membership against your deployment before proceeding.

For the full procedure, use Steps 2 through 4 in Oracle's Exadata Dedicated OKV setup documentation.

Step #7 Create the Exadata Key Store

Now I have the OKV connection information and the password stored as an OCI secret. Open Oracle AI Database → Oracle Exadata Database Service on Dedicated Infrastructure, select the compartment for the new Key Store, and click Key Stores.

Open Key Stores from the Exadata Dedicated service page.

Click Create Key Store and enter the following.

Field	What to enter
Name Your Key Store	A descriptive name, such as bsg-okv-keystore
Connection IP Addresses	The OKV IP address(s) reachable from the Exadata client network. You can enter multiple addreess if this is an OKV cluster.
Administrator Username	okv_rest_user, or the dedicated OKV username you created
Vault compartment and Vault	The vault associated with the password secret
Secret compartment and Secret	The compartment containing okv-rest-password, then that secret

The IP address and compartment names shown here belong to my example. Replace them with your own values.

For an OKV cluster, Oracle allows multiple comma-separated node IP addresses from the same OKV cluster. Enter addresses reachable from the database VMs. If you use a localhost tunnel to browse the OKV console, that localhost address does not belong in this field.

Review the settings, click Create Key Store, and wait for the resource to become active. Open its details and confirm the connection addresses, username, and password-secret reference.

If the secret does not appear in the selector, first check the selected region, vault, and secret compartment, followed by IAM access. If a later OKV operation cannot connect, check connectivity from every database VM, the REST access list, the Create Endpoint privilege, and whether the secret contains the current password.

Step #8 Using the Key Store

When creating a new database, you can now chose "Oracle Key Vault" from security choices for you encryption keys.

For an existing database, the next operation is on the database details page under Encryption → Change → Oracle Key Vault, where the Key Store is selected. Oracle states that this migration restarts the database, so plan a maintenance window. Follow the Administer Transparent Data Encryption (TDE) Keys section of the Exadata Dedicated documentation for that separate procedure.

NOTE: If you are already using an external key management system for this VM (OCI Vault for example), you cannot combining external Key Management systems on the same VM.

What I see in OKV

Endpoint

Within OKV, I can see that the tooling enrolled both VM cluster nodes as endpoints

The endpoints are named with the following format.

EXA_<database-name>_<resource-ID>_<VM-hostname>_EP

``` :chatgpt-content-reference{index="0"}

| Component           | Meaning |

|---                  |---|

| `EXA`               | Exadata automation prefix |

| `OKV1BSG_PPX_PHX`   | Database name/unique-name portion |

| `E47CC2A7-4C1F-40B3-8861-36CD245FB741` | Automation’s database resource identifier   |

| `EXACS-PHX3-OSRPA1` | DB VM hostname |

| `EP` | Endpoint |

**An endpoint authenticates the database host to OKV; a virtual wallet holds keys and grants access to endpoints.** If you see this name under **Wallets**, check its endpoint association—it may be an endpoint-associated wallet.

Wallet and Keys

I can see that wallet was created to store the keys

NOTE: on the DB side the unique identifier is the "database resource ID" rather than the OCID.

Summary:

These steps and links should give you the information you need to configure your database with OKV and have the tooling be able to properly manage the database.

A third recovery site: hybrid Data Guard and Autonomous Recovery Service

Until recently, most organizations I work with have been comfortable running their databases and applications across two regions. Sometimes that is an Active/Standby configuration, but more often I see some form of Active/Active. The big change I have seen recently is the addition of a third site, often described as a "cyber vault" or an IRE (Isolated Recovery Environment).

To provide that third site, many customers are looking at hosting a standby database in the cloud, either in OCI or through Oracle Database@ services in a multicloud environment. A practical way to limit traffic from the on-premises environment is to only send redo to the cloud database with Data Guard, then back up that cloud database to Autonomous Recovery Service. The backup workload stays on the cloud side.

Creating a cloud standby from your on-premises database is easier than you might think. In this post, I go through the standby build and the next step: configuring backups to Autonomous Recovery Service.

One thing to keep in mind is that a standby database and backups are only part of the picture. To use this as a cyber vault or IRE, you also need to control who can access it and which systems can connect to it. You need to protect the encryption keys and test that you can restore the database to a point before an attack or data corruption occurred. Plan for all of this as you build the standby and configure the backups..

The third-site pattern: send redo to a cloud standby, then back up the standby to Autonomous Recovery Service. The worked example below uses Oracle Exadata Database Service in OCI.

I recently went through the Oracle documentation for creating a hybrid Data Guard configuration. There is a lot of information to work through, especially when you are trying to figure out what needs to be ready before you start.

I am breaking this into four parts:

1. Get the prerequisites in place.
2. Run dbcactl to prepare the primary database.
3. Run dbaascli to create the standby database in OCI.
4. Configure backups from the cloud standby to Autonomous Recovery Service.

Once the preparation is done, the standby build comes down to two commands. I will go through their parameters and what dbcactl and  dbaascli are doing, then show where Recovery Service fits into the third-site design.

For this walkthrough, I am showing an on-premises Oracle RAC primary with a standby on Oracle Exadata Database Service in OCI. I have changed the database names and hostnames to use a generic sales example. My example build used a 19c Oracle Home and DBAAS CLI 26.3.1.0.0.

NOTE: The cloud VM cluster and Oracle Home need to exist before you start. configureStandby creates the database within that cluster. Refer to the full Oracle guide for platform requirements and configurations with additional standbys.

1) What needs to be ready before I start

First, let's go through the pieces that need to be in place before running either command.

• Download dbcactl. My Oracle Support document 3099785.1, linked from the hybrid guide, has the download and installation instructions. I am using /home/oracle/dbcacl/dbcactl in the example. The folder is named dbcacl, and the executable is dbcactl.
• Have the cloud Oracle Home ready. Match the primary release, RU, and one-off patches where possible. If you need a different RU, check the combinations allowed in the guide.
• Check connectivity in both directions. The primary needs to reach the cloud, and the cloud needs to reach the primary. Check DNS, SSH, and Oracle Net for all RAC nodes and SCAN endpoints. The network also needs enough bandwidth for the initial copy and ongoing redo.
• Existing standby database. If you already have an existing standby database you need to ensure that ALL databases can reach each other with DG broker. The existing standby needs to be able to resolve the scan listener on the new standby and connect. The new standby needs to be able to resolve the scan listener for the existing standby and connect.
• Have a shared ACFS directory on the primary if possible. This is where the network files and wallet configuration can be shared between instances. For this example I am using /acfs01/sales on an existing mounted ACFS filesystem.  If you do not have a shared ACFS directory you need to validate the connectivity on all DB nodes after executing dbcactl.
• Prepare the primary for Data Guard. This includes standby redo logs, logging, Flashback Database, Oracle Net encryption, and enough recovery-area space. Follow the MAA recommendations in the guide.

Make sure the encryption keys are there

One thing I want to call out is the TDE wallet. Even if the on-premises primary is not encrypted, it still needs a wallet with master encryption keys for this hybrid configuration. The cloud database needs those keys.

For this example I am using a file-based TDE keystore. Before starting, I would check that the wallet is usable and that the keys exist in the CDB root and each PDB other than the seed. The TDE with Data Guard documentation goes through the key handling.

For 19c at RU 19.16 or later, these are the settings I would look at:

Parameter	Example or policy	Purpose
WALLET_ROOT	/acfs01/sales/wallet	Root directory for wallet storage; the file-based TDE wallet is under tde. Use your existing keystore location.
TDE_CONFIGURATION	KEYSTORE_CONFIGURATION=FILE	Selects the file-based keystore used in this example.
TABLESPACE_ENCRYPTION	Choose for each site	AUTO_ENABLE is the cloud policy. DECRYPT_ONLY  is an on-premises option when encrypted tablespaces must be avoided. Do not apply it to the cloud standby.

NOTE: Setting the parameters does not create the wallet or set the master keys. Also, WALLET_ROOT is static, so changing it requires a restart.

If you still need to set up the wallet, follow the TDE configuration procedure. The TABLESPACE_ENCRYPTION reference explains the encryption policies. For earlier RUs, use the release-specific settings in the hybrid guide.

Here is a quick check I can run on the primary, connected through SQL*Plus as oracle:

SELECT name, db_unique_name, database_role, log_mode,
       force_logging, flashback_on
FROM v$database;

SHOW PARAMETER wallet_root
SHOW PARAMETER tde_configuration
SHOW PARAMETER tablespace_encryption

SELECT con_id, status, wallet_type, wrl_parameter
FROM v$encryption_wallet;

I want the primary in ARCHIVELOG mode, with force logging enabled and Flashback Database prepared before moving on. If archive logging still needs to be enabled, plan for the maintenance window that change may require.

I would also check V$DATABASE_KEY_INFO and V$ENCRYPTION_KEYS in the root and relevant PDBs. An open wallet is only part of the check; I need the master keys as well. If the wallet shows OPEN_NO_MASTER_KEY, that needs to be addressed before the build.

Use your existing key-management procedure and wallet location. There is no reason to replace an existing wallet just to match the path in my example.

Finally, have the primary SYS password, the TDE wallet password, and the password you will supply for AWR administration ready. These have different purposes and do not need to be the same.

Get Recovery Service ready as well

For the backup step, I also need the Recovery Service IAM policies, network connectivity, and a registered Recovery Service subnet. For 19c, the documented minimum is RU 19.18. Broker is mandatory for this backup destination in a Data Guard configuration with dbaascli 25.3.1.0.0 and later. I would also check regional availability, service limits, and the retention policy I need. The Exadata backup guide covers these prerequisites.

The names I am using in the example

Item	Generic value
Database name on both sites	sales
Primary database unique name	sales_primary
Standby database unique name	sales_stby
Primary SCAN	onprem-scan.example.com
Standby SCAN	cloud-scan.oraclevcn.com
Primary database service	sales_primary.example.com
SCAN listener port on both sites	1521
Existing cloud Oracle Home	/u02/app/oracle/product/19.0.0.0/dbhome_1

You can see that I am using sales as the database name on both sides. The unique names are different: sales_primary and sales_stby.

The other distinction to keep in mind is the SCAN name versus the database service. The SCAN gets me to the cluster, and the service identifies the database connection. Use the actual service registered with your primary listener when replacing sales_primary.example.com.

2) Run dbcactl on the primary

Figure 1. Run the preparation command on the first primary node. The standby SCAN already belongs to the provisioned cloud cluster.

Now I can prepare the primary. The diagram above shows the database and its SCAN listener, along with the cloud SCAN I will pass to the command.

I run this as oracle on the first primary node, with ORACLE_HOME and the local ORACLE_SID set for the source database. The shared directory /acfs01/sales also needs to be writable by oracle.

Here is the command with the example names:

/home/oracle/dbcacl/dbcactl \
  -silent \
  -oui_internal \
  -configureDatabase \
  -prepareForStandby \
  -dgTNSNamesoraFilePath /acfs01/sales \
  -sourceDB sales_primary \
  -gdbName sales.example.com \
  -standbyDBUniqueName sales_stby \
  -standbyScanName cloud-scan.oraclevcn.com \
  -standbyScanPort 1521 \
  -primaryScanPort 1521 \
  -standbyDBDomain oraclevcn.com \
  -blobFileLocation /tmp

The command prompts for the primary SYS password. In my run, I could see it creating the Data Guard services, updating tnsnames.ora and the include-file entry, and then preparing the standby bundle.

If you do not have a shared location for the the new tnsnames.ora file, you need to make sure ALL DB nodes have access to the entries in the new tnsnames.ora. This may involve manually adding the entries to your database.

What the dbcactl parameters mean

Parameter	Value in this example	Meaning
-silent	Flag	Runs without the graphical assistant; password prompts still appear.
-oui_internal	Flag	Workflow flag that allows for the use of prepareForStandby  (which is a hidden parameter)
-configureDatabase	Flag	Selects database configuration.
-prepareForStandby	Flag	Selects primary preparation for a standby.
-dgTNSNamesoraFilePath	/acfs01/sales	Directory for the Data Guard network configuration.  An "ifile" entry is created in the default tnsnames.ora file that points to a tnsnames.ora that is created in this directory which contains entries for both the primary database and the new standby database.
-sourceDB	sales_primary	Identifies the primary by its unique name.
-gdbName	sales.example.com	Global database name passed to the workflow.
-standbyDBUniqueName	sales_stby	Unique name planned for the standby.
-standbyScanName	cloud-scan.oraclevcn.com	SCAN hostname for the existing standby cluster.
-standbyScanPort	1521	Standby SCAN listener port.
-primaryScanPort	1521	Primary SCAN listener port.
-standbyDBDomain	oraclevcn.com	Standby database in OCI domain passed to the workflow.
-blobFileLocation	/tmp	Output directory for the generated bundle.

These are the options from my successful command, with the environment names changed. If you are using a different utility version, check its supported options, especially -gdbName and -pdbServiceDomain.

At the end, the command tells me where it wrote the .tar file. The filename will look like this, with a generated timestamp and ID:

Successfully created blob file:
/tmp/sales_<generated_timestamp>_<generated_id>.tar

This file contains the preparation material, including network files and the TDE wallet. I need to copy it securely to the first cloud node before running the second command.

For example, I can use scp:

scp /tmp/sales_<generated_timestamp>_<generated_id>.tar opc@cloud-node1.example.com:/tmp/

Replace the timestamp and ID with the actual filename from your output. For the next command, I am using /tmp/sales_standby.tar as a simpler name for the transferred file. You can keep the generated name instead; just use the same path in --standbyBlobFromPrimary.

NOTE: The bundle contains wallet material. Keep access to it restricted during the transfer and on the destination.

Autonomous Recovery Service Prechecks

 If you are configuring backups to utilize the Autonomous Recovery Service, there are some prerequisites that you need to be aware of.  If your Oracle Database was originally created in OCI and has always been OCI, those prerequisites are already configured for your database.  But, if you migrated a database to an OCI service, you might not realize that these items are required.

Prerequisites for Autonomous Recovery Service

1) WALLET_ROOT must be configured in the SPFILE.

WALLET_ROOT is a new parameter that was added in 19c, and its purpose is to replace the SQLNET.ENCRYTPION_WALLET_LOCATION in the sqlnet.ora file. Configuring the encryption wallet location in the sqlnet.ora file is depreciated.

WALLET_ROOT points to the directory path on the DB node(s) where the encryption wallet is stored for this database, and possibly the OKV endpoint client if you are using OKV to manage your encryption keys.

WALLET_ROOT allows each database to have it's own configuration location specific to each database.

There is a second parameter that goes with WALLET_ROOT that tells the database what kind of wallet is used (file, HSM or OKV), and that parameter is tde_configuration.

Running the script below should return the WALLET_ROOT location, and the tde_configuration information.

Checking the WALLET_ROOT and tde_configuration

set linesize 100 column value heading "Value" format a60 column name heading "Parameter" format a20 select name, value from v$parameter where upper(name) in ('WALLET_ROOT','TDE_CONFIGURATION');

Below you can see that both of these parameters are configured and I am using a wallet file.

Parameter            Value
-------------------- ------------------------------------------------------------
wallet_root          /var/opt/oracle/dbaas_acfs/bgrenndb/wallet_root
tde_configuration    keystore_configuration=FILE

If you need to migrate to WALLET_ROOT use the tooling .

dbaascli tde enableWalletRoot - enable wallet_root spfile parameter for existing database.

           Usage: dbaascli tde enableWalletRoot --dbname <value> [--dbRestart <value>] [--executePrereqs] [--resume [--sessionID <value>]]
                     Where:
                          --dbname - Oracle database name.
                          [--dbRestart - database restart option. Valid values : full|rolling ]
                          [ --executePrereqs - run the prerequisite checks and report the results. ]
                          [--resume - to resume the previous operation]
                          [--sessionID - to resume a specific session id.]

2) Encryption keys must be configured and available

In order to leverage the Autonomous Recovery Service, you must have an encryption key set and available for the CDB and each PDB.  If you migrated a non-TDE database (or plugged in a nonTDE PDB) to OCI you might not have configured encryption for one ore more PDBs.  The next step is to ensure that you have an encryption key set, and the wallet is open.  The query below should return "OPEN" for each CDB/PDB showing that the encryption key is available.

set linesize 170; set pagesize 100 column wrl_parameter format a60 column status heading 'Status' format a15 column pdb_name heading 'PDB Name' format a10 break on inst_id skip 1 select inst_id,b.name pdb_name,wrl_type, wrl_parameter, status from gv$encryption_wallet a,v$containers b where a.con_id = b.con_id(+) order by inst_id,b.name,wrl_type;

Below is the output from the query showing that the wallet is open for the CDB and the PDBs. 

   INST_ID PDB Name   Type       WRL_PARAMETER                                                Status
---------- ---------- ---------- ------------------------------------------------------------ ---------------
         1 BGRENNPDB1 FILE                                                                    OPEN
           CDB$ROOT   FILE       /var/opt/oracle/dbaas_acfs/bgrenndb/wallet_root/tde/         OPEN
           PDB$SEED   FILE                                                                    OPEN

         2 BGRENNPDB1 FILE                                                                    OPEN
           CDB$ROOT   FILE       /var/opt/oracle/dbaas_acfs/bgrenndb/wallet_root/tde/         OPEN
           PDB$SEED   FILE                                                                    OPEN

3) All tablespaces are TDE encrypted

TDE encryption is mandatory in OCI, and the Autonomous Recovery Service cannot be used if all of your tablespaces are not encrypted.  Below is a query to run that will tell you if your tablespaces are all encrypted.

column tablespaces heading "Encrypted tablespace information" format a60 select tablespaces from ( select 'Number of encrypted tablespaces : ' || count(1) tablespaces,1 ordercolumn from cdb_tablespaces where encrypted='YES' Union select 'Number of unencrypted tablespaces : ' || count(1) tablespaces,2 ordercolumn from cdb_tablespaces where encrypted='NO' Union select ' ' || '----' tablespaces,3 ordercolumn from dual Union select 'Total Number of tablespaces : ' || count(1) tablespaces,4 ordercolumn from cdb_tablespaces order by ordercolumn );

In my case I can see that all of the tablespaces are encrypted

Encrypted tablespace information
------------------------------------------------------------
Number of encrypted tablespaces   :      12
Number of unencrypted tablespaces :      0
                                         ----
Total Number of tablespaces       :      12

To find any tablespaces that are not encrypted you can run the query below.

set linesize 170 set pagesize 50 column pdb_name heading "PDB Name" format a10 column tablespace_name heading "Tablespace Name" format a30 column encrypted heading "Enc." format a5 break on pdb_name skip 1 select pdb_name, tablespace_name,encrypted FROM (select q.tablespace_name tablespace_name, z.name pdb_name, encrypted from v$containers z, cdb_tablespaces q where q.con_id = z.con_id and encrypted = 'NO' ) order by pdb_name,encrypted,tablespace_name;

4) Turn Off Any Manual Operational Backups.

In some cases, OCI users perform manual operational backups. These backups are run outside the standard tooling and support point-in-time recovery (non-KEEP backups).

Performing incremental backups to multiple locations can cause integrity issues with recovery.

The original backups can be kept to support the original retention window, and ensure that you have operational backups for a point-in-time prior to onboarding to the Recovery Service.  

Choose an appropriate cutover time, and switch to the Recovery Service, and slowly remove older backups as they expire until they are all completely removed.

RMAN create standby database - Restore or Duplicate ?

RMAN create standby database - Are you like me and use "restore database" for large databases, or like most people (based on my Linkedin poll) and use "duplicate for standby"? 

The table below shows you the 3 main differences between the 2 methods.

This post started with a discussion within my team around which method you use. I, being of the "restore database" camp, didn't realize how commonly used "duplicate for standby" is. 

I have also dug through the documentation, and there is no common method that is mentioned. Even the 21c documentation for creating a standby database doesn't mention using the duplicate command.

I also was pointed to a MOS note that goes through creating a standby directly from the primary across multiple nodes, and with encryption.  Creating a Physical Standby database using RMAN restore database from service (Doc ID 2283978.1)

Well in this post, I will explain why "restore database" has been my preference. 

NOTE : If you are creating a standby database that is encrypted and the source database is not (creating a standby database in OCI for example) then I have instructions at the end of this post for how to use "Restore Database" to create a hybrid standby database.

Duplicate database for standby

From the poll I ran, this is the most common way to create a standby database.  It is probably the simplest way also because a lot of the configuration of the standby database is done automatically as part of the automated process.

Below is the simplified steps to perform this process.

PRE work

1. Create simple initfile on the standby host.  The real SPFILE will be brought over as part of the duplication process.  This may contain location parameters for datafiles and redo logs if different from the primary.
2. Create directories on the standby host.  This includes the audit directory, and possibly the database file directories if they are different from the host.
3. Startup nomount.

Duplicate 

The duplicate process automatically performs these major steps using the standby as an auxiliary instance.

1.  Create an SPFILE. The process creates an SPFILE for the standby and sets parameters for the standby.
2. Shutdown/Startup standby database. This will use the newly created SPFILE during the rest of the processing
3. Restore backup controlfile for standby database. The controlfile for the standby database is put in place, and the spfile is updated to it's location
4. Mount controlfile . Mount the controlfile that was restored
5. Restore database . Restore the datafiles files for the CDB and PDBs to their new location on the standby
6. Switch datafile . Uses the new location of the datafiles that were restored.
7. Create standby redo logs.
8. Set parameters for standby database. The parameters necessary to communicate with the primary database are set.
9. Put standby in recover mode . By this time, you should have set the primary database to communicate with the standby database.

NOTES

If you noticed above, I highlighted the second step which forces a shutdown/startup of the standby database. Because of this step, it is not possible to use this method and restore across nodes in a RAC database.  This can cause the duplicate operation to take much longer for larger databases.

Then in step #5 you can see that the "Restore Database" is automatic in the processing and it is not possible to perform a "restore as encrypted" if you are migrating to OCI from a non-TDE database.  The duplicate process does support "restore as encrypted", but only for creating a new Database, not a standby database.

Restore Database

This is the method that I've always used.  There is no automation, but it gives you much more control over the steps.  

PRE work

1. Restore copy of prod SPFILE to standby host.  For this process, it doesn't matter if it is an intifile or spfile.  In this file you set all the parameters that are needed for the standby database to communicate with the primary and store datafiles/logfiles in the correct location.
2. Create directories on the standby host.  This includes the audit directory, and possibly the database file directories if they are different from the host.
3. Startup nomount.
4. Create copy of primary controlfile for standby. This will be used for the standby database, and should contain the backup catalog  of the primary database, and the RMAN settings including the  channel definitions.
5. Copy standby controlfile to standby host. The controlfile is copied to the standby host, and may be put in ASM at this point. Ensure the spfile points to the controlfile (and/or srvctl).
6. Alter database mount.  Mount the controlfile. 
7. Start up ALL nodes in the RAC cluster in mount mode.  This will allow you to restore the database across ALL nodes in the RAC cluster, and include all the networking from these nodes.  For a large database hosted on multiple DB nodes this can make a HUGE difference when restoring the database.
8. Create (or copy) TDE wallet.  If the standby database is going to be TDE, then include the wallet if the primary is TDE, or create a new wallet and key if the standby database is going to be TDE.

Restore Database 

The restore process is a manual process

1.  RMAN Connect to database (and possibly RMAN catalog). Connect to the database and make sure you have access to the backups. For ZDLRA this may mean connecting to the RMAN catalog.
2. Restore Database (as encrypted). This will restore the database to the new location.  With Restore Database, the database can be encrypted during the restore operation.  With 19c it is supported to have the standby database be encrypted without the primary database being encrypted (Hybrid dataguard).
3. Switch datafile . Uses the new location of the datafiles that were restored.
4. Recover database. This will use the archive logs that are cataloged to bring the standby database forward
5. Create standby redo logs.
6. Set parameters for standby database. The parameters necessary to communicate with the primary database are set.
7. Put standby in recover mode . By this time, you should have set the primary database to communicate with the standby database.

NOTES

With the restore database, there are 2 sections I highlighted and these are the advantages that I love about using this method.

• RMAN is restoring across multiple nodes in a RAC cluster which can make the restore operation much faster.
• Restore as encrypted allows you take a database that may have TDE partially implemented, or not implemented and create a new standby database that is encrypted. With the duplicate method, TDE would have to be implemented separately.

If you are restoring a VERY large database (200 TB for example) that was not TDE from object storage to the Exadata Cloud Service, both of these advantages can make a HUGE difference when creating a standby database.

Comparison

The chart below compares the the differences between "Duplicate Database" and "Restore Database".

WARNING: When using a ZDLRA for backups, it is NOT recommended to use the "Restore Database" to clone a database as a new copy. Registering the restored copy can cause issues with the RMAN catalog because the "restore database" leaves entries in the RC_SITE table.

Data Guard Hybrid Cloud Configuration

The ability to create a hybrid cloud configuration was introduced in Version 19.16 and there is a great blog post from Glen Hawkins explaining this feature.

This feature allows you have your Primary database remain unencrypted (no ASO license), but still have the standby database in OCI be TDE encrypted.

In this section I want to talk about how you can use "Restore Database as Encrypted" to implement this configuration quickly.

If you want to implement this feature using "Duplicate for standby" you have to separately encrypt the datafiles once they are restored in OCI.  This can be done online, or offline, but it is still a time consuming task.

Prepare the primary and future standby databases

The first step is prepare the primary database and future standby database by creating a wallet file and setting encryption keys.  There is a great video put together by Peter Wahl (PM for TDE and OKV) that goes through a lot of the steps.

Below is a summary of the steps you need to perform.  You can follow along the steps in Peter's video and I will point out where in the video you will find each step.

• Create the directories on the primary (3:40) -  Directories are specified in the video and need to be created on all nodes in a RAC cluster.
• Create the directories on the standby database (4:18) -Directories are specified in the video and need to be created on all nodes in a RAC cluster.
• Set the wallet_root in the primary (4:25) - This is set in the SPFILE only
• Set tablespace_encryption to decrypt_only on primary (4:40) -  This is set in the SPFILE only
• Set the default algorithm to AES256 on primary (4:50) - This is set in the SPFILE only
• Set wallet_root on standby, tablespace_encryption to auto_enable, and default algorithm on standby --  This is set in the initfile that you create prior to performing the restore.  This step is different from the video because there is no standby at this point.
• Bounce the primary database (5:50) - This can be in a rolling manner.
• Create a password protected wallet on the primary (7:25) - This gets created in the default location specified from WALLET_ROOT
• Create an auto open wallet on the primary (7:30) - This will make it automatically open for the database.
• Set the encryption keys in the primary (7:40) - The keys will be created so that they can be used when restoring the datafiles on the standby database.
• Copy the wallets from the primary to the standby (7:55) - This provides the standby database with the keys to encrypt.

Oracle Database Backup Cloud Service Primer

 One topic that has been coming a lot as customers look at options for offsite protected backups, is the use of the Oracle Database Backup Cloud Service.  This service can be used either directly from the database itself leveraging an RMAN tape library, or by performing a copy-to-cloud from the ZDLRA.  In this post I will try to consolidate all the information I can find on this topic to get you started.

Overview

The best place to start is by downloading, and reading through this technical brief. 

This document walks you through what the service is and how to implement it. Before you go forward with the Backup Cloud Service I suggest you download the install package and go through how to install it.

The key points I saw in this document are

• RMAN encryption is mandatory - In this brief you will see that the backups being sent to OCI MUST be encrypted, and the brief explains how to create an encrypted backup.  Included in the Backup Cloud Service is the use of encryption and compression (beyond basic compression) without requiring the ASO, or ACO license.
• How to install the client files - The brief explains the parameters that are needed to install the client files, and what the client files are that get installed. I will go into more detail later on explaining additional features that have been added recently.
• Config file settings including host - The document explains the contents of the configuration file used by the Backup Cloud Service library. It also explains how to determine the name of the host (OCI endpoint) based on the region you are sending the backups to.
• Channel configuration example - There is an example channel configuration to show you how to connect to the service.
• Best practices - The document includes sample scripts and best practices to use when using the Backup Cloud Service.
• Lifecycle policies and storage tiers - This is an important feature of using the Backup Cloud Service, especially for long term archival backups.  You most likely want have backups automatically moved to low cost archival storage after uploading to OCI.

NOTE: When using lifecycle polies to manage the storage tiers it is best to set the "-enableArchiving" and "-archiveAfterBackup" parameters when installing the backup module for a new bucket.  There are small metadata files that MUST remain in standard storage, and the installation module creates a lifecycle rule with the bucket that properly archives backup pieces, leaving the metadata in standard storage.

Download

The version of the library on OTN (at the time I am writing this) is NOT the current release of the library, and that version does not support retention lock of objects.

Please download the library from this location.

Documentation on the newer features can be found here, using retention lock can be found here, and there is a oci_readme.txt file that contains all the parameters available.

Updates

There were a few updates since the tech brief was written, and I will summarize the important ones here.  I also spoke the PM who is working on an updated brief that will contain this new information.

• newRSAKeyPair - The installer is now able to generate the key pair for you making it much easier to generate new key pair. In order to have the installer ONLY create a new key pair pair, just pass the installer the "walletDir" parameter.  The installer will generate both a public and private key, and place them in the walletDir (see below).

 /u01/app/oracle/product/19c/dbhome_1/jdk/bin/java -jar oci_install.jar -newRSAKeyPair -walletDir /home/oracle/oci/wallet 
Oracle Database Cloud Backup Module Install Tool, build 19.18.0.0.0DBBKPCSBP_2023-09-21
OCI API signing keys are created:
  PRIVATE KEY --> /home/oracle/oci/wallet/oci_pvt
  PUBLIC  KEY --> /home/oracle/oci/wallet/oci_pub
Please upload the public key in the OCI console.

Once you generate the public/private key, you can upload the public key to the OCI console. This will show you the fingerprint, and you can execute the installer using the private key file.

• "immutable-bucket" and "temp-metadata-bucket" - The biggest addition to library is the ability to support the use of retention rules on buckets containing backups.  The uploading of backups is monitored by using a "heartbeat" file, and this file is deleted when the upload is successful.  Because all objects in a bucket are locked, the "heartbeat" object must be managed from a second bucket without retention rules.  This is the temp-metadata-bucket.  When using retention rules you MUST have both buckets set in the config file.

NOTE: 

I ran into 2 issues when executing this script.

1)  When trying to execute the jar file, I used the default java version in my OCI tenancy that is located in "/user/bin". The installer received a java error

"java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter"

In order to properly execute the installer, I used the java executable located in $ORACLE_HOME/jdk/bin

2) When executing the jar file with my own RSA key that I had been previously used with OCI object storage, I received a java error.

Exception in thread "main" java.lang.RuntimeException: Could not produce a private key
at oracle.backup.util.FileDownload.encode(FileDownload.java:823)
at oracle.backup.util.FileDownload.addBmcAuthHeader(FileDownload.java:647)
at oracle.backup.util.FileDownload.addHttpAuthHeader(FileDownload.java:169)
at oracle.backup.util.FileDownload.addHttpAuthHeader(FileDownload.java:151)
at oracle.backup.opc.install.BmcConfig.initBmcConnection(BmcConfig.java:437)
at oracle.backup.opc.install.BmcConfig.initBmcConnection(BmcConfig.java:428)
at oracle.backup.opc.install.BmcConfig.testConnection(BmcConfig.java:393)
at oracle.backup.opc.install.BmcConfig.doBmcConfig(BmcConfig.java:250)
at oracle.backup.opc.install.BmcConfig.main(BmcConfig.java:242)
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence

I found that this was caused by the PKCS format. I was using a PKCS1 key, and the java installer was looking for a PKCS8 key.  The header in my private key file contained "BEGIN RSA PRIVATE KEY".

In order to convert my private PKCS1 key "oci_api_key.pem" to a PKCS8 key "pkcs8.key" I ran.

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in oci_api_key.pem -out pkcs8.key

Executing the install

The next step is to execute the install. For my install I also wanted configure a lifecycle rule that would archive backups after 14 days.  In order to implement this, I had the script create a new bucket "bsgtest".  Below is parameters I used (note I used "..." to obfuscate the OCIDs).

$ORACLE_HOME/jdk/bin/java -jar oci_install.jar -pvtKeyFile /home/oracle/oci/wallet/pkcs8.key -pubFingerPrint .... -tOCID  ocid1.tenancy.oc1... -host https://objectstorage.us-ashburn-1.oraclecloud.com -uOCID ocid1.user.oc1.... -bucket bsgtest -cOCID ocid1.compartment.oc1... -walletDir /home/oracle/oci/wallet -libDir /home/oracle/oci/lib -configFile /home/oracle/oci/config/backupconfig.ora -enableArchiving TRUE -archiveAfterBackup "14 days"

This created a new bucket "bsgtest" containing a lifecycle rule.

I then added a 14 day retention rule to this bucket, and created a second bucket "bsgtest_meta" for the temporary metadata. If you want to make this rule permanent you enable retention rule lock which I highlighted on the screenshot below.

I then updated the config file to use the metadata bucket because I set a retention rule on the main bucket. Note that there is also a parameter that determines how long archival objects are cached in standard storage before they are returned to archival storage.

OPC_CONTAINER=bsgtest
OPC_TEMP_CONTAINER=bsgtest_meta
OPC_AUTH_SCHEME=BMC
retainAfterRestore=48 HOURS

Testing

Once you execute the installer you will be able to begin backing up to OCI object storage.  Don't forget that you need to:

• Change the default device type to SBT_TAPE
• Change the compression algorithm. I recommend "medium" compression.
• Configure encryption for database ON.
• Configure the device type SBT_TAPE to send COMPRESSED BACKUPSET to optimize throughput and storage in OCI.
• Create a default channel configuration for SBT_TAPE (or allocate channels manually) that use the library that was downloaded, and point to the configuration file for the database.
• If you do not use ACO and don't have a wallet , manually set an encryption password in your session.

I recommend sending a "small" backup piece first to ensure that everything is properly configured.  My favorite command is

RMAN>backup incremental level 0 datafile 1;

Datafile 1 is always the system tablespace.

Below is what my configuration looks like for RMAN specifically for what I changed to use the Backup Cloud Service.

CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS  'SBT_LIBRARY=/home/oracle/oci/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/oci/config/backupconfig.ora)';
CONFIGURE ENCRYPTION FOR DATABASE ON;
CONFIGURE ENCRYPTION ALGORITHM 'AES256'; # default
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;

Network Performance

One of the big areas that comes up with using the Backup Cloud Service, is understanding the network capabilities.

The best place to start is with this MOS note

How to measure network performance from RMAN for ZDLRA or Cloud Backups (Doc ID 2371860.1) It is a short note, but it shows you how to determine the throughput you can expect from a single stream/channel of data. Below is an example of how to use this based on my configuration.  I used it to test sending 1000M (1GB) of data.  From the results I can see that a single channel can send and receive 1GB in about 15 seconds.  This works out to about 240 GB/hr.

RMAN> run {
2> allocate channel foo device type sbt  PARMS  'SBT_LIBRARY=/home/oracle/oci/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/oci/config/backupconfig.ora)';
3>  send channel foo 'NETTEST 1000M';
4> }

allocated channel: foo
channel foo: SID=431 device type=SBT_TAPE
channel foo: Oracle Database Backup Service Library VER=19.0.0.1

released channel: foo
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of send command at 11/22/2023 14:12:04
ORA-19559: error sending device command: NETTEST 1000M
ORA-19557: device error, device type: SBT_TAPE, device name:
ORA-27194: skgfdvcmd: sbtcommand returned error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   KBHS-00402: NETTEST sucessfully completed
KBHS-00401: NETTEST RESTORE: 1048576000 bytes received in 15068283 microseconds
KBHS-00400: NETTEST BACKUP: 1048576000 bytes sent

Executing Backups

Now to put it all together I am going to execute a backup of datafile 1.  My database is encrypted, so I am going to set a password along with the encryption key.

 set encryption on identified by oracle;

executing command: SET encryption

RMAN>  backup incremental level 0 datafile 1;

Starting backup at 22-NOV-23
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=404 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=19.0.0.1
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: SID=494 device type=SBT_TAPE
channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=19.0.0.1
allocated channel: ORA_SBT_TAPE_3
channel ORA_SBT_TAPE_3: SID=599 device type=SBT_TAPE
channel ORA_SBT_TAPE_3: Oracle Database Backup Service Library VER=19.0.0.1
allocated channel: ORA_SBT_TAPE_4
channel ORA_SBT_TAPE_4: SID=691 device type=SBT_TAPE
channel ORA_SBT_TAPE_4: Oracle Database Backup Service Library VER=19.0.0.1
channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/u01/app/oracle/oradata/ACMEDBP/system01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 22-NOV-23
channel ORA_SBT_TAPE_1: finished piece 1 at 22-NOV-23
piece handle=8t2c4fmi_1309_1_1 tag=TAG20231122T150554 comment=API Version 2.0,MMS Version 19.0.0.1
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:35
Finished backup at 22-NOV-23

Starting Control File and SPFILE Autobackup at 22-NOV-23
piece handle=c-1654679317-20231122-01 comment=API Version 2.0,MMS Version 19.0.0.1
Finished Control File and SPFILE Autobackup at 22-NOV-23

Restoring

Restoring is very easy as long as you have the entries in your controlfile. If you don't then there is a 

 script included in the installation that can catalog the backup pieces and I go through that process here.

This also allows you to display what's in the bucket.

Buckets 1 vs many

If you look at what created when executing backup you will see that there is a set format for the backup pieces. Below are the 2 backup pieces that I created

• 8t2c4fmi_1209_1_1 - This is the backup of datafile 1 for my database ACMEDBP
• c-16546791317-20231122-01 - This is the controlfile backup for this database

Notice that the DB name is not in the name of the backup pieces, or in the visible nesting.

If you think about a medium sized database (let's say 100 datafiles), that has 2 weeks of backups (14 days), you would have 1,400 different backup pieces for the datafiles within the "sbt_catalog" directory.

My recommendation is to group small databases together in the same bucket (keeping the amount of backup pieces to a manageable level).

For large database (1,000+ datafiles), you can see where a 30 day retention could become 30,000+ backup pieces.

Having a large number of objects within a bucket increases the time to report the available backup pieces.  There is no way to determine which database the object is a member of without looking at the metadata.

Keep this in mind when considering how many buckets to create.