Why DBCS (Oracle Base Database Service) in OCI can make a DBA's life much easier (even with BYOL)

DBCS (now named Oracle Base Database service, but I will call it DBCS throughout this post) in OCI  can help make a DBA's life easier.  When I was testing the new Autonomous Recovery Service for Oracle Database in OCI, I created a LOT of different DBCS systems to test backup and recovery.  Along the way I learned a lot about the workings of DBCS, and I came to appreciate how it makes sense, even if you are a BYOL (bring your own license) customer.

I'm more of a an "old school" DBA, preferring command line, and scripting processes myself.  I am typically not a fan of automation.  When using DBCS I was surprised by all the things it would do for me that I would have to do manually.

Install oracle software and create a database

Having installed oracle software hundreds of times, and having created test databases, I didn't think I would care much about automation that did this for me.

Central Software image management

What I found in OCI, is that you can create your own software images that can be used to ensure each new database environment is consistent.  OCI gives you ability to create your own set of release images (which can include patches).  This ensures each time I create a new DBCS environment, and choose my custom image, it's running the same version in all environments. No more installing base release, then patches, and then then any possible one-off patches.  This makes the installation of the database software much, much easier, and ensures consistency.

Easy Database creation

Recently I've gotten familiar with performing a silent database creation, as using dbca isn't always easy to configure.  The tooling provided by DBCS will not only create a database for you, but will also configure TDE encryption (with a local wallet, or using OCI vault).  It can even create a RAC database across 2 nodes.  And don't forget, it can create the standby for me also.

Configure ASM storage

Now this is the most interesting piece I found when using DBCS.  Not only does the DBCS service create a disk group, but it automatically stripes multiple block volumes together maximizing performance.  This is a HUGE help in ensuring I am getting the best performance.

When I was going through what the configuration did, I tried to build tables showing how the different storage sizes translate to the storage configurations.

There were 2 configurations and DB data storage sizes, one for Flex, and one for Standard shapes.

Flex

First I looked at flex, and regardless of the performance level these were the sizes.

Then within Flex, I looked at the "Balanced performance" configuration.

Balanced Performance configuration

You can see that as the DB storage available goes up, the number of disks goes up also allowing for a higher  possible IOPS than you would get from a single Block Storage device.

Below is the chart for "High Performance"

High Performance configuration

You can see that the IOPS is even higher, and it is using even more disks to get that performance.

Standard

Next looked at standard shapes, and regardless of the performance level these were the sizes. Note that with Standard shapes, there were many more options for configurations.

Balanced Performance configuration

High Performance configuration

Benefits of DBCS

I also went through what some of the other benefits of DBCS are, and below is the list I came up with.

• When using the DBCS service,  the storage cost is based on the Block Storage cost. This is the same cost as you would pay in an IaaS service.  Having the storage striped and configured for maximum IOPS makes this a huge plus.

• DBCS allows you purchase licenses if you don't have enough licenses to use the BYOL option.

• The DBCS service price is based on OCPU and is the same regardless of the shape. Memory is included in the OCPU cost.

• DBCS automatically configures RAC if you choose it.

• DBCS provides tooling that automatically configures backups, can apply patches, and rotate encryption keys.

• DBCS allows you to automate the cloning of your database, and automate any restores.

• DBCS includes TDE, and relieves you of having to own the ASO license.  

Conclusion:

DBCS offers a lot more than you realize. Take a deep dive into what it can do for you to save time as DBA and you also might realize that sometimes tooling along with automation has it's benefits.

Oracle Database recovery using Incremental merge, snapshots, OMF and "switch to copy"

I work with backup and recovery of the Oracle Database, and sometimes this means looking at the Incremental Merge backup strategy.  I know this isn't the best backup/recovery strategy, and below are few posts giving you more detail on the topic.

• Chris Craft's blog post - "Using incremental merge"
• Tim Chien's blog post - "Why CDB(snapshots) are NOT backups"

They have some great points, and I typically don't recommend using incremental merge backups.  The incremental merge backup strategy is almost always paired with snapshots to increase the recovery window.

Below is an image of how these are typically paired with snapshots.

One of the biggest draws of using the incremental merge strategy with snapshots, is the ability to perform a "switch to copy" as a recovery strategy.

NOTE: When you perform "switch to copy" the database is now accessing datafiles using the backup copy.  This is not supported on Exadata for any storage other than Oracle ZFS.

If you review the MOS note "Using External Storage with Exadata (Doc ID 2663308.1)" you will find that "Use of non-Oracle storage for database files is not supported."

Given all of that, I got the question "I am using the incremental merge strategy on Oracle an ZFS appliance using snapshots. If I perform a switch-to-copy recovery of one or more datafiles, how do I avoid forcing a new full backup on the next incremental merge backup?".

I thought this was a great question, and I created a test database, and started googlin'.  Below are some of the posts I looked at.

• Oracle 12c and RMAN switch datafile to copy, is it really so easy?
• Oracle docs for "alter database move datafile"

I started  by using the first post and walked through a testing scenario using a DBCS database in OCI.

My database was a 19.8 database using local storage (to make things easier to see the datafiles), and it was using OMF by default.  

The piece that was missing from the first post was the "alter database move datafile 'xxx' to 'xxx' KEEP;

What I found is that it wasn't so easy with my database using OMF for 2 reasons.

1. Using OMF, you don't specify the "to 'xxx'" since OMF will automatically name the destination datafile.
2. Using "KEEP" is ignored when the source file is OMF.  This meant that the original image copy being used by the database is removed when move process completes.  I couldn't catalog the image copy.

Since it took a bit of research to find the best strategy I wanted to share the process that I would recommend when dealing with OMF and non-OMF image copy backups with snapshots.

NON-OMF image copy backups

1. Snap the backup storage just to preserve the starting point. --> optional but recommended
2. Take the tablespaces offline
3. Perform a "switch to copy" of the datafiles --> This will use the incremental merge backup.
4. Recover the datafiles
5. Bring the tablespaces online ---> Application is running using the external image copy
6. Perform an "alter database move datafile 'xxx' to 'xxx' KEEP; --> Using keep will preserve the original copy, but will only work if the image copy is NOT OMF. If the destination is an OMF file, you will not use the "to"
7. Catalog the image copy that was preserved with the "KEEP" ensuring you use the same tag used for the incremental merge. "catalog datafilecopy '+fra/ocm121/datafile/MONSTER.346.919594959' level 0 TAG 'incr_update';"
8. The next incremental merge will pick up with the updated image copy.

OMF image copy backups

1. Snap the backup storage to create a copy for the switch to copy.
2. Unmount the "current" image copy 
3. Mount the snap copy using the same mount point as the "current" image copy.
4. Take the tablespaces offline
5. Perform a "switch to copy" of the datafiles --> This will use the snap copy of the incremental merge backup on external storage.
6. Recover the datafiles
7. Bring the tablespaces online ---> Application is running using external copies.
8. Perform an "alter database move datafile 'xxx' ; --> Since the source is an OMF file you cannot use "KEEP" to preserve the original copy. The original copy will be removed.
9. Once all moves are complete, unmount the snapped copy.
10. Mount the "current" copy. This is as of when you started this process.
11. Catalog the image copy for all datafiles that performed the "switch to copy" ensuring you use the same tag used for the incremental merge "catalog datafilecopy '+fra/ocm121/datafile/MONSTER.346.919594959' level 0 TAG 'incr_update';"
12. You can now destroy the snap that was created to perform the switch to copy.
13. The next incremental merge will pick up with the current image datafile copies where it left off.

As you can see, using OMF greatly complicates preserving the incremental merge backup, and forces you to start at the last backup.

ZDLRA - Quick Start Guide

 This post is intended to be a Quick Start Guide for those who are new to ZDLRA (RA for short).  I spend part of time working with customers who are new the RA and often the same topics/questions come up.  I wanted to put together a "Quick Start" guide that they can use to learn more about these common topics.

The steps I would follow for anyone new to the RA are.

1. Read through the section on configuring users and security settings for the RA. Decide which compliance settings make sense for the RA and come with a plan to implement them.
2. Identify the users, both OS users (if you are disabling direct root access), and users within the databases that will mange and/or monitor the RA. OS users can be added with "racli add admin_user". Database users can be added with "racli add db_user"
3. Create protection policies that contain the recovery window(s) that you want to set for the databases. You will also set compliance windows when creating policies. This can be done manually using the package DBMS_RA.CREATE_PROTECTION_POLICY.
4. Identify the VPC user(s) needed to manage the database. Is it a single DBA team, or different teams requiring multiple VPC users? Create the VPC user using "racli add vpc_user"
5. Add databases to be backed up to the RA, associate the database with both a protection policy and a VPC user who will be managing the database. NOTE that you should look at the Reserved Space, and adjust it as needed.  Databases can be added manually by using two PL/SQL calls. DBMS_RA.ADD_DB will add the database to the RA. DBMS_RA.GRANT_DB_ACCESS will allow the VPC user to manage the database.
6. Configure the database to be backed up to the RA either by using OEM, or manually. The manual steps would be

• Create a wallet on the DB client that contains the VPC credentials to connect to the RA.
• Update the sqlnet.ora file to point to this wallet
• Connect to the RMAN catalog on the RA from the DB client
• Register the database to the RA
• Configure the channel configuration to point to the RA
• Configure Block change tracking (if it is not configured).
• Configure the redo destination to point to the RA if you want to configure real-time redo.
• Change the RMAN retention to be "applied to all standby" if using real-time redo, or "backed up 1 time" if not.
• Update OEM to have the database point to the RMAN catalog on the ZDLRA.

Documentation

The documentation can be found here. Within the documentation there are several sections that will help you manage the RA.

Get Started 

The get started section contains some subtopics to delve into

Install and configure the Recovery Appliance

The links in this section cover all the details about the installation and configuration of the RA.  I won't be talking about those sections in the post, but be aware this is where to look for general maintenance/patching/expanding information.

Learn about the Recovery Appliance.

This section covers an overview of the RA, and is mostly marketing material. If you are not familiar with the RA, or want an overview this is the place to turn.

Administer the Recovery Appliance

These sections are going to be a lot more helpful to get you started. This section of the documentation covers 

Managing Protect Policies - Protection policies is the place to start when configuring an RA. Protection policies group databases together and it is critical to make sure you have the correct protection policies in place before adding databases to be backed up.

Copying Backups to Tape - This section is useful if you plan on creating backups (either point in time or archival) that will be sent externally from the RA. This can be either to physical/virtual tape, or to an external media manager.

Archiving Backups to the Cloud - This section covers how to configure the RA to send backups to an OCI compatible object storage.  This can either be OCI, or it can be an on-premises ZFS that has a project configured as OCI object storage.

Accessing Recovery Appliance Reports - This section covers how to access all the reports available to you.  You will find these reports are priceless to manage the RA over time. Some examples of the areas these reports cover are.

• Storage Capacity Planning reports with future usage projections
• Recovery Window Summary reports to validate backups are available
• Active incident reports to manage any alerts
• API History Report to audit any changes to the RA

NOTE : If you are using the RA in a charge backup model to your internal business units, there is specific reporting that can be used for this. Talk your Oracle team find out more.

Monitoring the Recovery Appliance - This section covers how to monitor the RA and set up alerts. This will allow you identify any issues that would affect the recovery of the backups including space issues, and missing backups.

Administer the Recovery Appliance

Configure Protected Databases - This section goes through how to configure databases to be backed up to the recovery appliance and includes instructions for both using OEM, and adding databases using the command line.

Backup Protected Databases - This section covers how to backup a database from either OEM, or from the traditional RMAN command line. I would also recommend looking at the MOS note to ensure that you are using the current best practices for backups. "RMAN best practice recommendations for backing up to the Recovery Appliance (Doc ID 2176686.1)".

Recover Databases - This section covers how to recover databases from the RA. This section also covers information about cloning databases. Cloning copies of production is a common use case for the RA, and this section is very useful to help you with this process.

Books

This section contains the documentation you look at regularly to manage the RA and answer questions that you may have on managing it.  I am only going to point the sections that you find most useful.

Deployment

The one important section under deployment is the Zero Data Loss Recovery Appliance Owners Guide.

Zero Data Loss Recovery Appliance Owners Guide - This guide contains information on configuring users on the RA, and the most critical sections to look at are

•  "Part III Security and Maintenance of Recovery Appliance".   If you are using the RA to manage immutable backups, it is important to go through this section to understand how users will be managed for maximum protection of your backups.
• Part IV Command Reference - This section covers the CLI commands you will use the manage the RA.

Administration

This is probably the most important guide in the documentation. It covers many of the areas of you will be managing as you configure databases to be backed up.  The most critical sections are

Part I Managing Recovery appliance - This section covers

• Implementing Immutable Backups
• Securing the Recovery Appliance operations
• Managing Protection Policies
• Configuring replication and replication concepts
• Additional High Availability strategies

Part III Recovery Appliance Reference - This section covers

• DBMS_RA packages to manage the RA through commands
• Recovery Appliance View Reference to see what views are available

MOS Notes

There are number of useful MOS notes that you will want to bookmark

• Zero Data Loss Recovery Appliance (ZDLRA) Information Center (Doc ID 2673011.2)
• How to Backup and Recover the Zero Data Loss Recovery Appliance (Doc ID 2048074.1)
• Zero Data Loss Recovery Appliance Supported Versions (Doc ID 1927416.1)
• Zero Data Loss Recovery Appliance Software Updates Guide (Doc ID 2028931.1)
• Cross Platform Database Migration using ZDLRA (Doc ID 2460552.1)
• How to Move RMAN Catalog To A Different Database (Doc ID 351918.1)

Helpful Blogs

Fernando Simon

Fernando has a number of helpful blog entries. Be aware he has been blogging for a long time on the RA, and some of the management processes have changed. One example is RACLI is now used to create VPC users. Some of the Blogs to note are

• How to use ZDLRA and enroll a database
• Duplicate, Restore and ZDLRA

• ZDLRA, Creating the Replication Config

Bryan Grenn

I have a number of blog posts on features of the ZDLRA.