Showing posts with label data guard. Show all posts
Showing posts with label data guard. Show all posts

Friday, October 2, 2026

A third recovery site: hybrid Data Guard and Autonomous Recovery Service

Until recently, most organizations I work with have been comfortable running their databases and applications across two regions. Sometimes that is an Active/Standby configuration, but more often I see some form of Active/Active. The big change I have seen recently is the addition of a third site, often described as a "cyber vault" or an IRE (Isolated Recovery Environment).

To provide that third site, many customers are looking at hosting a standby database in the cloud, either in OCI or through Oracle Database@ services in a multicloud environment. A practical way to limit traffic from the on-premises environment is to only send redo to the cloud database with Data Guard, then back up that cloud database to Autonomous Recovery Service. The backup workload stays on the cloud side.

Creating a cloud standby from your on-premises database is easier than you might think. In this post, I go through the standby build and the next step: configuring backups to Autonomous Recovery Service.

One thing to keep in mind is that a standby database and backups are only part of the picture. To use this as a cyber vault or IRE, you also need to control who can access it and which systems can connect to it. You need to protect the encryption keys and test that you can restore the database to a point before an attack or data corruption occurred. Plan for all of this as you build the standby and configure the backups..

Hybrid Data Guard from an on-premises primary to a cloud standby and Autonomous Recovery Service inside a multicloud Cyber Vault or Isolated Recovery Environment

The third-site pattern: send redo to a cloud standby, then back up the standby to Autonomous Recovery Service. The worked example below uses Oracle Exadata Database Service in OCI.

I recently went through the Oracle documentation for creating a hybrid Data Guard configuration. There is a lot of information to work through, especially when you are trying to figure out what needs to be ready before you start.

I am breaking this into four parts:

  1. Get the prerequisites in place.
  2. Run dbcactl to prepare the primary database.
  3. Run dbaascli to create the standby database in OCI.
  4. Configure backups from the cloud standby to Autonomous Recovery Service.

Once the preparation is done, the standby build comes down to two commands. I will go through their parameters and what dbcactl and  dbaascli are doing, then show where Recovery Service fits into the third-site design.

For this walkthrough, I am showing an on-premises Oracle RAC primary with a standby on Oracle Exadata Database Service in OCI. I have changed the database names and hostnames to use a generic sales example. My example build used a 19c Oracle Home and DBAAS CLI 26.3.1.0.0.

NOTE: The cloud VM cluster and Oracle Home need to exist before you start. configureStandby creates the database within that cluster. Refer to the full Oracle guide for platform requirements and configurations with additional standbys.

1) What needs to be ready before I start

First, let's go through the pieces that need to be in place before running either command.

  • Download dbcactl. My Oracle Support document 3099785.1, linked from the hybrid guide, has the download and installation instructions. I am using /home/oracle/dbcacl/dbcactl in the example. The folder is named dbcacl, and the executable is dbcactl.
  • Have the cloud Oracle Home ready. Match the primary release, RU, and one-off patches where possible. If you need a different RU, check the combinations allowed in the guide.
  • Check connectivity in both directions. The primary needs to reach the cloud, and the cloud needs to reach the primary. Check DNS, SSH, and Oracle Net for all RAC nodes and SCAN endpoints. The network also needs enough bandwidth for the initial copy and ongoing redo.
  • Existing standby database. If you already have an existing standby database you need to ensure that ALL databases can reach each other with DG broker. The existing standby needs to be able to resolve the scan listener on the new standby and connect. The new standby needs to be able to resolve the scan listener for the existing standby and connect.
  • Have a shared ACFS directory on the primary if possible. This is where the network files and wallet configuration can be shared between instances. For this example I am using /acfs01/sales on an existing mounted ACFS filesystem.  If you do not have a shared ACFS directory you need to validate the connectivity on all DB nodes after executing dbcactl.
  • Prepare the primary for Data Guard. This includes standby redo logs, logging, Flashback Database, Oracle Net encryption, and enough recovery-area space. Follow the MAA recommendations in the guide.

Make sure the encryption keys are there

One thing I want to call out is the TDE wallet. Even if the on-premises primary is not encrypted, it still needs a wallet with master encryption keys for this hybrid configuration. The cloud database needs those keys.

For this example I am using a file-based TDE keystore. Before starting, I would check that the wallet is usable and that the keys exist in the CDB root and each PDB other than the seed. The TDE with Data Guard documentation goes through the key handling.

For 19c at RU 19.16 or later, these are the settings I would look at:

ParameterExample or policyPurpose
WALLET_ROOT/acfs01/sales/walletRoot directory for wallet storage; the file-based
TDE wallet is under tde. Use your existing keystore location.
TDE_CONFIGURATIONKEYSTORE_CONFIGURATION=FILESelects the file-based keystore used in this example.
TABLESPACE_ENCRYPTIONChoose for each siteAUTO_ENABLE is the cloud policy. DECRYPT_ONLY
 is an on-premises option when encrypted tablespaces must be avoided. Do not apply it to the cloud standby.

NOTE: Setting the parameters does not create the wallet or set the master keys. Also, WALLET_ROOT is static, so changing it requires a restart.

If you still need to set up the wallet, follow the TDE configuration procedure. The TABLESPACE_ENCRYPTION reference explains the encryption policies. For earlier RUs, use the release-specific settings in the hybrid guide.

Here is a quick check I can run on the primary, connected through SQL*Plus as oracle:

SELECT name, db_unique_name, database_role, log_mode,
       force_logging, flashback_on
FROM v$database;

SHOW PARAMETER wallet_root
SHOW PARAMETER tde_configuration
SHOW PARAMETER tablespace_encryption

SELECT con_id, status, wallet_type, wrl_parameter
FROM v$encryption_wallet;

I want the primary in ARCHIVELOG mode, with force logging enabled and Flashback Database prepared before moving on. If archive logging still needs to be enabled, plan for the maintenance window that change may require.

I would also check V$DATABASE_KEY_INFO and V$ENCRYPTION_KEYS in the root and relevant PDBs. An open wallet is only part of the check; I need the master keys as well. If the wallet shows OPEN_NO_MASTER_KEY, that needs to be addressed before the build.

Use your existing key-management procedure and wallet location. There is no reason to replace an existing wallet just to match the path in my example.

Finally, have the primary SYS password, the TDE wallet password, and the password you will supply for AWR administration ready. These have different purposes and do not need to be the same.

Get Recovery Service ready as well

For the backup step, I also need the Recovery Service IAM policies, network connectivity, and a registered Recovery Service subnet. For 19c, the documented minimum is RU 19.18. Broker is mandatory for this backup destination in a Data Guard configuration with dbaascli 25.3.1.0.0 and later. I would also check regional availability, service limits, and the retention policy I need. The Exadata backup guide covers these prerequisites.

The names I am using in the example

ItemGeneric value
Database name on both sitessales
Primary database unique namesales_primary
Standby database unique namesales_stby
Primary SCANonprem-scan.example.com
Standby SCANcloud-scan.oraclevcn.com
Primary database servicesales_primary.example.com
SCAN listener port on both sites1521
Existing cloud Oracle Home/u02/app/oracle/product/19.0.0.0/dbhome_1

You can see that I am using sales as the database name on both sides. The unique names are different: sales_primary and sales_stby.

The other distinction to keep in mind is the SCAN name versus the database service. The SCAN gets me to the cluster, and the service identifies the database connection. Use the actual service registered with your primary listener when replacing sales_primary.example.com.

2) Run dbcactl on the primary

Primary RAC database, SCAN listener, preparation command, and generated standby bundle

Figure 1. Run the preparation command on the first primary node. The standby SCAN already belongs to the provisioned cloud cluster.

Now I can prepare the primary. The diagram above shows the database and its SCAN listener, along with the cloud SCAN I will pass to the command.

I run this as oracle on the first primary node, with ORACLE_HOME and the local ORACLE_SID set for the source database. The shared directory /acfs01/sales also needs to be writable by oracle.

Here is the command with the example names:

/home/oracle/dbcacl/dbcactl \
  -silent \
  -oui_internal \
  -configureDatabase \
  -prepareForStandby \
  -dgTNSNamesoraFilePath /acfs01/sales \
  -sourceDB sales_primary \
  -gdbName sales.example.com \
  -standbyDBUniqueName sales_stby \
  -standbyScanName cloud-scan.oraclevcn.com \
  -standbyScanPort 1521 \
  -primaryScanPort 1521 \
  -standbyDBDomain oraclevcn.com \
  -blobFileLocation /tmp

The command prompts for the primary SYS password. In my run, I could see it creating the Data Guard services, updating tnsnames.ora and the include-file entry, and then preparing the standby bundle.

If you do not have a shared location for the the new tnsnames.ora file, you need to make sure ALL DB nodes have access to the entries in the new tnsnames.ora. This may involve manually adding the entries to your database.

What the dbcactl parameters mean

ParameterValue in this exampleMeaning
-silentFlagRuns without the graphical assistant; password prompts still appear.
-oui_internalFlagWorkflow flag that allows for the use of prepareForStandby
 (which is a hidden parameter)
-configureDatabaseFlagSelects database configuration.
-prepareForStandbyFlagSelects primary preparation for a standby.
-dgTNSNamesoraFilePath/acfs01/salesDirectory for the Data Guard network configuration.  An "ifile" entry is created in the default tnsnames.ora file that points to a tnsnames.ora that is created in this directory which contains entries for both the primary database and the new standby database.
-sourceDBsales_primaryIdentifies the primary by its unique name.
-gdbNamesales.example.comGlobal database name passed to the workflow.
-standbyDBUniqueNamesales_stbyUnique name planned for the standby.
-standbyScanNamecloud-scan.oraclevcn.comSCAN hostname for the existing standby cluster.
-standbyScanPort1521Standby SCAN listener port.
-primaryScanPort1521Primary SCAN listener port.
-standbyDBDomainoraclevcn.comStandby database in OCI domain passed to the workflow.
-blobFileLocation/tmpOutput directory for the generated bundle.

These are the options from my successful command, with the environment names changed. If you are using a different utility version, check its supported options, especially -gdbName and -pdbServiceDomain.

At the end, the command tells me where it wrote the .tar file. The filename will look like this, with a generated timestamp and ID:

Successfully created blob file:
/tmp/sales_<generated_timestamp>_<generated_id>.tar

This file contains the preparation material, including network files and the TDE wallet. I need to copy it securely to the first cloud node before running the second command.

For example, I can use scp:

scp /tmp/sales_<generated_timestamp>_<generated_id>.tar opc@cloud-node1.example.com:/tmp/

Replace the timestamp and ID with the actual filename from your output. For the next command, I am using /tmp/sales_standby.tar as a simpler name for the transferred file. You can keep the generated name instead; just use the same path in --standbyBlobFromPrimary.

NOTE: The bundle contains wallet material. Keep access to it restricted during the transfer and on the destination.

Tuesday, January 9, 2024

RMAN create standby database - Restore or Duplicate ?

RMAN create standby database - Are you like me and use "restore database" for large databases, or like most people (based on my Linkedin poll) and use "duplicate for standby"? 

The table below shows you the 3 main differences between the 2 methods.


This post started with a discussion within my team around which method you use. I, being of the "restore database" camp, didn't realize how commonly used "duplicate for standby" is. 
I have also dug through the documentation, and there is no common method that is mentioned. Even the 21c documentation for creating a standby database doesn't mention using the duplicate command.
I also was pointed to a MOS note that goes through creating a standby directly from the primary across multiple nodes, and with encryption.  Creating a Physical Standby database using RMAN restore database from service (Doc ID 2283978.1)

Well in this post, I will explain why "restore database" has been my preference. 

NOTE : If you are creating a standby database that is encrypted and the source database is not (creating a standby database in OCI for example) then I have instructions at the end of this post for how to use "Restore Database" to create a hybrid standby database.

Duplicate database for standby


From the poll I ran, this is the most common way to create a standby database.  It is probably the simplest way also because a lot of the configuration of the standby database is done automatically as part of the automated process.
Below is the simplified steps to perform this process.

PRE work

  1. Create simple initfile on the standby host.  The real SPFILE will be brought over as part of the duplication process.  This may contain location parameters for datafiles and redo logs if different from the primary.
  2. Create directories on the standby host.  This includes the audit directory, and possibly the database file directories if they are different from the host.
  3. Startup nomount.

Duplicate 

The duplicate process automatically performs these major steps using the standby as an auxiliary instance.

  1.  Create an SPFILE. The process creates an SPFILE for the standby and sets parameters for the standby.
  2. Shutdown/Startup standby database. This will use the newly created SPFILE during the rest of the processing
  3. Restore backup controlfile for standby database. The controlfile for the standby database is put in place, and the spfile is updated to it's location
  4. Mount controlfile . Mount the controlfile that was restored
  5. Restore database . Restore the datafiles files for the CDB and PDBs to their new location on the standby
  6. Switch datafile . Uses the new location of the datafiles that were restored.
  7. Create standby redo logs.
  8. Set parameters for standby database. The parameters necessary to communicate with the primary database are set.
  9. Put standby in recover mode . By this time, you should have set the primary database to communicate with the standby database.

NOTES

If you noticed above, I highlighted the second step which forces a shutdown/startup of the standby database. Because of this step, it is not possible to use this method and restore across nodes in a RAC database.  This can cause the duplicate operation to take much longer for larger databases.
Then in step #5 you can see that the "Restore Database" is automatic in the processing and it is not possible to perform a "restore as encrypted" if you are migrating to OCI from a non-TDE database.  The duplicate process does support "restore as encrypted", but only for creating a new Database, not a standby database.

Restore Database


This is the method that I've always used.  There is no automation, but it gives you much more control over the steps.  

PRE work

  1. Restore copy of prod SPFILE to standby host.  For this process, it doesn't matter if it is an intifile or spfile.  In this file you set all the parameters that are needed for the standby database to communicate with the primary and store datafiles/logfiles in the correct location.
  2. Create directories on the standby host.  This includes the audit directory, and possibly the database file directories if they are different from the host.
  3. Startup nomount.
  4. Create copy of primary controlfile for standby. This will be used for the standby database, and should contain the backup catalog  of the primary database, and the RMAN settings including the  channel definitions.
  5. Copy standby controlfile to standby host. The controlfile is copied to the standby host, and may be put in ASM at this point. Ensure the spfile points to the controlfile (and/or srvctl).
  6. Alter database mount.  Mount the controlfile. 
  7. Start up ALL nodes in the RAC cluster in mount mode.  This will allow you to restore the database across ALL nodes in the RAC cluster, and include all the networking from these nodes.  For a large database hosted on multiple DB nodes this can make a HUGE difference when restoring the database.
  8. Create (or copy) TDE wallet.  If the standby database is going to be TDE, then include the wallet if the primary is TDE, or create a new wallet and key if the standby database is going to be TDE.

Restore Database 

The restore process is a manual process

  1.  RMAN Connect to database (and possibly RMAN catalog). Connect to the database and make sure you have access to the backups. For ZDLRA this may mean connecting to the RMAN catalog.
  2. Restore Database (as encrypted). This will restore the database to the new location.  With Restore Database, the database can be encrypted during the restore operation.  With 19c it is supported to have the standby database be encrypted without the primary database being encrypted (Hybrid dataguard).
  3. Switch datafile . Uses the new location of the datafiles that were restored.
  4. Recover database. This will use the archive logs that are cataloged to bring the standby database forward
  5. Create standby redo logs.
  6. Set parameters for standby database. The parameters necessary to communicate with the primary database are set.
  7. Put standby in recover mode . By this time, you should have set the primary database to communicate with the standby database.


NOTES

With the restore database, there are 2 sections I highlighted and these are the advantages that I love about using this method.
  • RMAN is restoring across multiple nodes in a RAC cluster which can make the restore operation much faster.
  • Restore as encrypted allows you take a database that may have TDE partially implemented, or not implemented and create a new standby database that is encrypted. With the duplicate method, TDE would have to be implemented separately.
If you are restoring a VERY large database (200 TB for example) that was not TDE from object storage to the Exadata Cloud Service, both of these advantages can make a HUGE difference when creating a standby database.

Comparison

The chart below compares the the differences between "Duplicate Database" and "Restore Database".

WARNING: When using a ZDLRA for backups, it is NOT recommended to use the "Restore Database" to clone a database as a new copy. Registering the restored copy can cause issues with the RMAN catalog because the "restore database" leaves entries in the RC_SITE table.



Data Guard Hybrid Cloud Configuration

The ability to create a hybrid cloud configuration was introduced in Version 19.16 and there is a great blog post from Glen Hawkins explaining this feature.
This feature allows you have your Primary database remain unencrypted (no ASO license), but still have the standby database in OCI be TDE encrypted.

In this section I want to talk about how you can use "Restore Database as Encrypted" to implement this configuration quickly.

If you want to implement this feature using "Duplicate for standby" you have to separately encrypt the datafiles once they are restored in OCI.  This can be done online, or offline, but it is still a time consuming task.

Prepare the primary and future standby databases

The first step is prepare the primary database and future standby database by creating a wallet file and setting encryption keys.  There is a great video put together by Peter Wahl (PM for TDE and OKV) that goes through a lot of the steps.

Below is a summary of the steps you need to perform.  You can follow along the steps in Peter's video and I will point out where in the video you will find each step.

  • Create the directories on the primary (3:40) -  Directories are specified in the video and need to be created on all nodes in a RAC cluster.
  • Create the directories on the standby database (4:18) -Directories are specified in the video and need to be created on all nodes in a RAC cluster.
  • Set the wallet_root in the primary (4:25) - This is set in the SPFILE only
  • Set tablespace_encryption to decrypt_only on primary (4:40) -  This is set in the SPFILE only
  • Set the default algorithm to AES256 on primary (4:50) - This is set in the SPFILE only
  • Set wallet_root on standby, tablespace_encryption to auto_enable, and default algorithm on standby --  This is set in the initfile that you create prior to performing the restore.  This step is different from the video because there is no standby at this point.
  • Bounce the primary database (5:50) - This can be in a rolling manner.
  • Create a password protected wallet on the primary (7:25) - This gets created in the default location specified from WALLET_ROOT
  • Create an auto open wallet on the primary (7:30) - This will make it automatically open for the database.
  • Set the encryption keys in the primary (7:40) - The keys will be created so that they can be used when restoring the datafiles on the standby database.
  • Copy the wallets from the primary to the standby (7:55) - This provides the standby database with the keys to encrypt.




Friday, September 29, 2023

DG per PDB with 23c on DBCS (Oracle Base Database)

At Oracle Cloud World (a few weeks ago as of writing this) 23c was announced as being available with DBCS.
 I've been wanting to test out DGPDB, which is Data Guard per PDB. This new feature was introduced with 21c and is covered in detail in this blog here .


Since I cover a lot of topics around backup, recovery, and availability this feature intrigued me.

NOTE: DGPDB is NOT supported with the tooling on DBCS and by testing it in your environment as I am going to do here your databases will not be supported.

First I am going to start with the documentation which can be found here.

My Environment :

I created 2 database services in ExaCS, both of which are 23c databases that are TDE encrypted (since they are in DBCS), and I am using a locally managed key (local wallet file).


CDB Name DB Unique Name PDB name
cdb1 cdb1_c83_iad pdb1
cdb2 cdb2_cdx_iad cdb2_pdb2


Step # 1 - Prepare the 2 CDBs

The first step is to prepare the 2 CDBs and follow the steps in the documentation

On both databases execute
  • ALTER DATABASE FORCE LOGGING;
  • ALTER DATABASE FLASHBACK ON;

On the source database execute
  • alter system set dg_broker_start=true scope=both;
  • alter system set log_archive_dest_1="LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb1_c83_iad";
  • alter system set standby_file_management=auto scope=both; ;
On the destination database execute
  • alter system set dg_broker_start=true scope=both;
  • alter system set log_archive_dest_1="LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb2_cdx_iad";
  • alter system set standby_file_management=auto scope=both;

Step # 2 - Configure tnsname.oras and SEPS wallets

First I am going to ensure that both tnsnames.ora files contains entries for both CDBs.
  • CDB1_C83_IAD
  • CDB2_CDX_IAD
Next I create a SEPS wallet on both nodes, and add entries for both databases
mkdir -p $ORACLE_HOME/dbs/wallets/dgpdb
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createALO
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createCredential cdb1_c83_iad sys {password}
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createCredential cdb2_cdx_iad sys {password}
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -listCredential


Then I update the sqlnet.ora to use this wallet file.

WALLET_LOCATION =
    (SOURCE =
      (METHOD = FILE)
      (METHOD_DATA =
        (DIRECTORY = /u01/app/oracle/product/23.0.0.0/dbhome_1/dbs/wallets/dgpdb)
    )
)
SQLNET.WALLET_OVERRIDE = TRUE

And finally I test the connection
  • sqlplus /@cdb1_c83_iad as sysdba
  • sqlplus /@cdb2_cdx_iad as sysdba

Step # 3 - Create source and target configuration

Configure the source


dgmgrl /@cdb1_c83_iad <<EOF

CREATE CONFIGURATION 'cdb1_c83_iad' AS CONNECT IDENTIFIER IS cdb1_c83_iad; 
SHOW CONFIGURATION; 

EOF

Configure the target
dgmgrl /@cdb2_cdx_iad <<EOF

CREATE CONFIGURATION 'cdb2_cdx_iad' AS CONNECT IDENTIFIER IS cdb2_cdx_iad; 
SHOW CONFIGURATION; 

EOF

Step # 4 - Establish the connection between configurations

I ran the step in the documentation to configure the connection.


dgmgrl /@cdb1_c83_iad <<EOF

ADD CONFIGURATION 'cdb2_cdx_iad' CONNECT IDENTIFIER IS cdb2_cdx_iad; 
SHOW CONFIGURATION; 
ENABLE CONFIGURATION ALL; 
SHOW CONFIGURATION; 
EOF


And I validated that the output looked correct.

Configuration - cdb1_c83_iad

  Protection Mode: MaxPerformance
  Members:
  cdb1_c83_iad - Primary database
  cdb2_cdx_iad - Primary database in cdb2_cdx_iad configuration

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 1 second ago)

Step # 5 - Prepare the databases for DGPDB

The first step was to ensure that the PDBS are open in both database, which they are.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
SQL>

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 CDB2_PDB1                      READ WRITE NO
SQL>


Then prepare the databases for DGPDB

 dgmgrl /@cdb1_c83_iad
DGMGRL for Linux: Release 23.0.0.0.0 - Production on Thu Sep 28 17:40:38 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "cdb1_c83_iad"
Connected as SYSDBA.
DGMGRL> EDIT CONFIGURATION PREPARE DGPDB;
Enter password for DGPDB_INT account at cdb1_c83_iad:
Enter password for DGPDB_INT account at cdb2_cdx_iad:

Prepared Data Guard for Pluggable Database at cdb2_cdx_iad.

Prepared Data Guard for Pluggable Database at cdb1_c83_iad.
DGMGRL>


Step # 6 - Restart the listeners and databases with the wallet location


On both the source and destination, I performed an "su - grid" and restarted the listeners.
I also restarted both databases. Until I restarted the databases I kept getting a 

 "Error: ORA-12578: A wallet file was not found or failed to open."

error message.


Step # 7 - Add the pluggable database to the destination

The next step is to add the pluggable database to the destination in dgmgrl.

DGMGRL> add pluggable database pdb1_stby at cdb2_cdx_iad source is pdb1 at cdb1_c83_iad  pdbfilenameconvert is "'+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/','+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/'"  ;
ORA-46697: Keystore password required.
Help: https://docs.oracle.com/error-help/db/ora-46697/


At this point by following the documentation, adding the pluggable database failed with  "ORA-46697: Keystore password required."

After some digging I found that I need to pass the wallet password (which is my sys password), and that the phrase needs to be in single quotes.


DGMGRL> add pluggable database pdb1_stby at cdb2_cdx_iad source is pdb1 at cdb1_c83_iad  pdbfilenameconvert is "'+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/','+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/'"  'keystore  IDENTIFIED BY "W3lCom3#123#123"';

Pluggable Database "PDB1_STBY" added


Step # 8 - List datafiles in the source PDB


     FILE#  NAME
----------  ------------------------------------------------------------------------------------------------------------------------
         8  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/system.273.1148746431
         9  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/sysaux.270.1148746437
        10  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/undotbs1.271.1148746445
        12  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/users.274.1148746543

Step # 9 - Copy the datafiles to the destination


First I logged onto the primary database and closed it.

 alter pluggable database pdb1 close;

I decided to use DBMS_FILE_TRANSFER.GET_FILE

Step 9a - create directory on source pointing to datafiles


SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 11:33:11 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> create directory SOURCE_DUMP as '+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/';

Directory created.

SQL> grant read,write on directory SOURCE_DUMP to public;

Grant succeeded.

SQL>

Step 9b - Log onto ASM and precreate directory

ASMCMD> mkdir 
+data/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/datafile


Step 9c - create directory on target pointing to datafiles

[oracle@cdb1 admin]$ sqlplus /@cdb2_cdx_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 11:33:44 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> create directory TARGET_DUMP as '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/';

grant read,write on directory TARGET_DUMP to public;



Directory created.

SQL> SQL>
Grant succeeded.

Step 9d - create database link on target  pointing to source database


SQL> SQL> SQL> create public database link SOURCEDB connect to system identified by {password} using 'CDB1_C83_IAD';

Database link created.


SQL> select sysdate from dual@SOURCEDB;
select sysdate from dual@SOURCEDB
                         *
ERROR at line 1:
ORA-02085: database link SOURCEDB.ZFSADMIN.VCN.ORACLEVCN.COM connects to
CDB1.ZFSADMIN.VCN.ORACLEVCN.COM
Help: https://docs.oracle.com/error-help/db/ora-02085/


SQL> show parameter global

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
allow_global_dblinks                 boolean     FALSE
global_names                         boolean     TRUE
global_txn_processes                 integer     1
SQL> alter system set global_names=false;

System altered.

SQL> select sysdate from dual@SOURCEDB;

SYSDATE
---------
29-SEP-23

SQL>

Step 9e - Get the datafiles from the source


SQL> set timing on
BEGIN
dbms_file_transfer.get_file('SOURCE_DUMP',
'system.273.1148746431',
'SOURCEDB',
'TARGET_DUMP',
'SYSTEM.DBF');
END;
/SQL>   2    3    4    5    6    7    8

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.21

Step # 10 - Rename datafiles in the destination.


List the placeholder datafiles in  destination database
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 CDB2_PDB1                      READ WRITE NO
         4 PDB1_STBY                      MOUNTED
SQL> alter session set container=pdb1_stby;

Session altered.

Elapsed: 00:00:00.02
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/MUST_RENAME_THIS_DATAFILE_8.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_9.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_10.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_12.4294967295.4294967295

Elapsed: 00:00:00.02
SQL>

Match the file number in the destination files to the source datafiles and rename them.

SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_8.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/SYSTEM.dbf';

Database altered.

Elapsed: 00:00:00.04
SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_9.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/SYSAUX.dbf';

Database altered.

Elapsed: 00:00:00.02

SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_10.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/UNDOTBS1.dbf';

Database altered.

Elapsed: 00:00:00.05
SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_12.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/USERS.dbf';

Database altered.

Elapsed: 00:00:00.03
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/system.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/sysaux.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/undotbs1.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/users.dbf

Elapsed: 00:00:00.02
SQL>


Step # 11 - Bring over the TDE keys

Export the encryption keys from the source database

SQL> ADMINISTER KEY MANAGEMENT
  EXPORT ENCRYPTION KEYS WITH SECRET "export_secret"
  TO '/tmp/mykey'
  FORCE KEYSTORE
  IDENTIFIED BY "{password}";  2    3    4    5

keystore altered.



Import them into the target database
ADMINISTER KEY MANAGEMENT
  IMPORT KEYS WITH SECRET "export_secret"
  FROM '/tmp/mykeypdb1'
  force keystore
  IDENTIFIED BY "W3lCom3#123#123"
  WITH BACKUP;

SQL> SQL>   2    3    4    5    6
keystore altered.

SQL> SQL>

List the encryption keys on the  source database
PDB Name        Key ID                              Master Key ID
--------------- ----------------------------------- -------------------------
CDB$ROOT        35B424EFDB6D4F19BFF31B5D3DE0BFFA    ATW0JO/bbU8Zv/MbXT3gv/o=
PDB$SEED        00000000000000000000000000000000    AQAAAAAAAAAAAAAAAAAAAAA=
PDB1            1AAD19EB6F364F3BBF6FE226A25AF1E5    ARqtGetvNk87v2/iJqJa8eU=


List the encryption keys on the destination database
Master Key ID                                           Tag                  PDB Name        KEYSTORE_TYPE     Origin     Key Creation Time  Key Act. Time
------------------------------------------------------- -------------------- --------------- ----------------- ---------- ------------------ ------------------
AZ5CKuJLrU/Nv5M5pXQCFcEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB$ROOT        SOFTWARE KEYSTORE LOCAL      09/28/2023 09:09   09/28/2023 09:09
ATW0JO/bbU8Zv/MbXT3gv/oAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB$ROOT        SOFTWARE KEYSTORE LOCAL      09/28/2023 09:10   09/28/2023 09:10
AVAymCcev0/Pv94Tiqw/P50AAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB2_PDB1       SOFTWARE KEYSTORE LOCAL      09/28/2023 09:12   09/28/2023 09:12
ARqtGetvNk87v2/iJqJa8eUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB2_PDB1       SOFTWARE KEYSTORE LOCAL      09/28/2023 09:13   09/28/2023 09:13




Step # 12 - Add the standby redo logs to the PDB and validate

I created standby redo logs in the PDB on Target database

select 'ALTER DATABASE ADD STANDBY LOGFILE thread ' ||a.thread# || ' group 1' ||a.group# || ' (''' ||substr(b.member,1,35) || 'standby_pdb1_1' || a.group# ||  ''') size ' || a.bytes ||';'
  from v$log a, v$logfile b
  where a.group#=b.group# ;SQL>   2    3

'ALTERDATABASEADDSTANDBYLOGFILETHREAD'||A.THREAD#||'GROUP1'||A.GROUP#||'('''||SUBSTR(B.MEMBER,1,35)||'STANDBY_PDB1_1'||A.GROUP#||'
----------------------------------------------------------------------------------------------------------------------------------
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 13 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_13') size 1073741824;
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 12 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_12') size 1073741824;
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 11 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_11') size 1073741824;

SQL> alter session set container=PDB1_STBY;

Session altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 4 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_13') size 1073741824;

Database altered.


SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 5 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_12') size 1073741824;

Database altered.

SQL>  ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 6  ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_11') size 1073741824;

Database altered.

SQL>

And then I validated the configuration

DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Not Running
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad



Step # 13 - change the state of standby to APPLY-ON

I changed the state of the standby and checked the configuration.


DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Not Running
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad


DGMGRL>
EDIT PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad SET STATE='APPLY-ON';DGMGRL>
Succeeded.
DGMGRL> SHOW CONFIGURATION;

Configuration - cdb2_cdx_iad

  Protection Mode: MaxPerformance
  Members:
  cdb2_cdx_iad - Primary database
    Warning: ORA-16910: inconsistency detected for one or more pluggable databases

  cdb1_c83_iad - Primary database in cdb1_c83_iad configuration

Data Guard for PDB:   Enabled in TARGET role

Configuration Status:
SUCCESS   (status updated 37 seconds ago)

DGMGRL> SHOW PLUGGABLE DATABASE pdb1 AT cdb1_c83_iad;

Pluggable database - PDB1 at cdb1_c83_iad

  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       con_id 4 at cdb2_cdx_iad

Pluggable Database Status:
SUCCESS

DGMGRL> SHOW PLUGGABLE DATABASE pdb1_stby AT cdb2_cdx_iad ;

Pluggable database - PDB1_STBY at cdb2_cdx_iad

  Data Guard Role:     Physical Standby
  Con_ID:              4
  Source:              con_id 3 at cdb1_c83_iad
  Transport Lag:       20 hours 26 minutes 24 seconds (computed 9 seconds ago)
  Apply Lag:           (unknown)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      cdb2
  Average Apply Rate:  (unknown)
  Real Time Query:     OFF

Pluggable Database Status:
SUCCESS

DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Waiting for Redo Data
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad

DGMGRL>

Step # 14 - Perform a log switch at both databases



 sqlplus /@cdb1_c83_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 15:57:52 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> exit
Disconnected from Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09
[oracle@cdb2 admin]$ sqlplus /@cdb2_cdx_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 15:58:11 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL>


Step # 15 - Validate standby is applying redo.


DGMGRL> SHOW PLUGGABLE DATABASE pdb1_stby AT cdb2_cdx_iad ;

Pluggable database - PDB1_STBY at cdb2_cdx_iad

  Data Guard Role:     Physical Standby
  Con_ID:              4
  Source:              con_id 3 at cdb1_c83_iad
  Transport Lag:       0 seconds (computed 1 second ago)
  Apply Lag:           4 seconds (computed 1 second ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      cdb2
  Average Apply Rate:  489 KByte/s
  Real Time Query:     OFF

Pluggable Database Status:
SUCCESS




SUMMARY :

This is an exciting new feature, and I am looking forward to see it expand.

Since a target database, can only have one source, I see this being useful to "pair" CDBs.
Instead of having 2 Primary CDBs in DC1 (for example) and 2 standby CDBs in DC2 you would have 2 Primary CDBs in in each Data Center, and you would be able to switch over any PDBs individually.


Since this is a paradigm change for RMAN, I am going to be following how this affects RMAN, the RMAN catalogs, and the ZDLRA/Recovery Service in it's support.


Finally :

It is all working, and for fun I am going to look at the destination settings on the source Database.

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_1                   string      LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb1_c83_iad
log_archive_dest_2                   string      service="cdb2_cdx_iad", ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 reopen=300 db_unique_name="cdb2_cdx_iad" net_timeout=30, valid_for=(online_logs)


Today I did have time to perform a switchover and a second switchover to bring it back.
Because my database is encrypted, I did import the keys from CDB2 into CDB1.  I am thinking when you implement this with TDE, you do need to keep the keys from both CDBs syncronized.
In OKV, I'm thinking you would want to possibly even use a single wallet. At the very least both primary databases would need access to both wallets.

OBSERVATIONS:


I will continue to update this post as I find useful observations.

Observation 1 :

I looked at the archive log using v$archived_logs view in the target (CDB2).
In it I only 6 archive logs were created.
SQL> select thread#,sequence#,dest_id,blocks,status from v$ARCHIVED_LOG;

   THREAD#  SEQUENCE#    DEST_ID     BLOCKS S
---------- ---------- ---------- ---------- -
         1          2          1     230208 A
         1          3          1     896391 A
         1          4          1          7 A
         1          5          1      35474 A
         1          6          1        387 A

I added a couple more PDBS and made changes to a second PDB (that didn't have a standby) and the original PDB (that had a standby).
I then created a workload in both PDBS and looked at the archive logs on the source.

   THREAD#  SEQUENCE#    DEST_ID     BLOCKS S
---------- ---------- ---------- ---------- -
         1          3          2       4894 A
         1          4          2     567681 A
         1          5          2         14 A
         1          6          2     871136 A
         1          7          2    1756544 A
         1          8          2    1847012 A
         1          9          2    1931501 A
         1         10          2    1901263 A
         1         11          2    1931491 A
         1         12          2    1931837 A
         1         13          2    1465002 A
         1         14          2    1955017 A
         1         15          2    1936946 A
         1         16          2    1685507 A
         1         17          2    1931499 A
         1         18          2    1808038 A
         1         19          2    1874455 A
         1         20          2    1910501 A
         1         21          2    2038138 A
         1         22          2    1812968 A
         1         23          2    1935941 A
         1         24          2    1939459 A
         1         25          2    2030413 A
         1         26          2    1934927 A

24 rows selected.

Then I looked at ASM on the target and I could ALL the archive logs on the target.


ASMCMD> ls
thread_1_seq_10.282.1148837541
thread_1_seq_11.283.1148837541
thread_1_seq_12.284.1148837557
thread_1_seq_13.285.1148837667
thread_1_seq_14.287.1148837943
thread_1_seq_15.288.1148837963
thread_1_seq_16.289.1148837979
thread_1_seq_17.290.1148837979
thread_1_seq_18.291.1148838023
thread_1_seq_19.292.1148838023
thread_1_seq_20.293.1148838101
thread_1_seq_21.294.1148838119
thread_1_seq_22.295.1148838137
thread_1_seq_23.296.1148838137
thread_1_seq_24.297.1148838183
thread_1_seq_25.298.1148838213
thread_1_seq_26.299.1148838237
thread_1_seq_3.275.1148831895
thread_1_seq_4.276.1148831899
thread_1_seq_5.274.1148831883
thread_1_seq_5.278.1148837115
thread_1_seq_6.277.1148836711
thread_1_seq_6.286.1148837681
thread_1_seq_7.279.1148837481
thread_1_seq_8.280.1148837499
thread_1_seq_9.281.1148837517


Observation 2 :

RMAN does not support this feature at this point.

 What I have discovered is that

  • Within the CDB, V$ARCHIVED_LOG does not recognize archive logs from the second primary needed to recovery the standby
  • Within the CDB, V$DATAFILE does not recognize the datafiles for the “standby database”
  • Within the standby PDB, V$DATAFILE recognizes the datafiles for that standby database, and only the standby datafiles
  • Within the standby PDB V$ARCHIVED_LOG only sees this archive logs for the standby database, and not the local archive logs produced
  • When executing RMAN for the CDB, only the locally produced archive logs are backed up
  • When executing RMAN for the CDB, only the primary datafiles are backed up.
  • When executing RMAN for the standby PDB, no datafiles are backed up.  “pluggable database PDB1_STBY does not have any data files”
  • When executing RMAN for the standby PDB, only the primary database’s locally produced archive logs are backed up.

 

SUMMARY – with DGPDB

 Within the PDB, the PDB can see the datafiles, and archive logs needed to recover with V$DATAFILE and V$ARCHIVED_LOG

 RMAN does not currently see the standby datafiles, or it’s archive logs to back them up.