How many IP addresses do I need for the Autonomous Recovery Service

 One of the most common questions that comes up is "How many IP addresses do I need to set aside for the Autonomous Recovery Service" or "How big does the CIDR block need to be for my Recovery Service subnet"?

In this blog post, I will explain how IPs are used by the service, but how many IP address you will need is hard to put an exact number on.

First below is a diagram showing how this works.

Recovery Service Subnet(s)

The first piece to understand is how the Autonomous Recovery Service uses the Subnet(s) that are registered.

First you might be wondering why I have the "(s)" on the end.  When you register a recovery service subnet there are two levels.

You register a name for "Recovery Service Subnet" and this is actually a group of subnets. You can register multiple subnets as eligible to be used for a "Recovery Service Subnets".

 The screenshot above is what you will see in OCI. 

When you register a Recovery Service subnet,

• You give it a name for the "Recovery Service subnet group" 
• You identify the VCN that this subnet is registered for. Each VCN will have it's own registered subnet group.
• You add one or more subnet within that VCN that can be used for endpoint IP address.

Any of these registered subnets can be used for Autonomous Recovery Service IP addresses.

Also subnets can be added, and removed within the group.

How many IP addresses for a Database backup?

I am going to start with a single database before I explain what happens when you have multiple databases using the service.  In order to support Oracle Database backups, the Autonomous Recovery Service uses endpoint IP addresses that map to a pair of ZDLRAs that store the backups as a service.  The pair of ZDLRAs provide an always available service.

For a single database below is what you would see for the endpoints that get created. In my example, you can see that there are 3 IP address per RA in the "Recovery Service Group".

Above, this shows the 6 private endpoint IP addresses that are created for the database backups being sent to two ZDLRAs (RA-018 and RA-020).  There are also FQDN names that are created for each each of the endpoints and you can see that the names map to the specific ZDLRAs that are storing the backups

NOTE: There are are also some 4 node ZDLRAs in some regions. In that case there will be 4 endpoint IP address for each ZDLRA in the pair, and a total of 8 IP addresses will be utilized.

How many IP addresses do I need for multiple databases?

This is where the answer is "It depends".  The simple example above shows you what happens for a single database. When you add another database it might not end up on the same "Recovery Service group". It is possible the new database backups could end up on another "Recovery Service group" needing additional IP addresses.

There are number of factors that affect how many "Recovery Service groups" are used when backing up multiple databases.

• Number of databases - If you have a large number of databases, this increases the chances that more backup locations will be used to spread out the backups across multiple groups.
• Size of the Database backups - if your backups are very large, the Recovery Service tries to balance larger database backups across more groups. 
• Number of groups in the region - Some regions contain more "recovery Service groups" than other regions.  If you are backing up in a larger region there is a higher chance that more groups will be utilized to support many databases.

The diagram I started with below shows you what happens with 3 databases that are storing their backups across two different Recovery Service groups.

The first database is sending it's backup to a Recovery Service Group containing two X 2 DB node  ZDLRAs and it is utilizing 6 IP addresses.

The second and third databases are using the same Recovery Recovery Group which consists of Two X 3 DB node ZDLRAs and they are using the same 8 IP addresses.

How to interpret this?

The recommendation for Recovery Service Subnets is to create a separate subnet that is a /24 CIDR block which will provide the ability to have 254 private endpoint IP addresses. This will allow for at least 31 different Recovery Service groups.

If you only have a few databases, then this may be too big for what you need, and you may be able to have a smaller CIDR block, or have multiple subnets with smaller CIDR blocks.

The recommendation of /24 CIDR blocks ensures you will not have any issues with enough IP address.

As you decrease the number of available IP addresses you increase the chances that you will not enough IP address to add another database to be backed up to the Autonomous Recovery Service.

What happens if I don't register enough free IPs?

Once a database is added is configured for backups, it will not affect the need for additional free IP addresses.  The only time you will have an issue with free IP addresses for the recovery service is when you add a new database to be backed up. If the onboarded process decides that the backups need to reside a new Recovery Service Group of ZDLRAs, and there are not enough free IP address you will receive an error when configuring backups. At that point you can add more subnets to the Recovery Service subnet group registered with the VCN.

Do I have to worry about space since databases are assigned to Recovery Service groups?

No.  The recovery service will automatically manage the underlying storage for the database backups and move backups from one group to another group if needed in order ensure there is enough space for backups. Because of this, you may find that the names of the ZDLRAs where the backups reside could change over time. This is one of the reasons why the service dynamically creates the TNSNAMES entry as needed. The FQDN used for backups of a database will change if the database is moved because of space constraints.

Summary

There is no set number of number of IP addresses that need to be registered with the recovery service and freely available to be assigned for backups.  It is dependent on the size of your environment, and number of IP addresses utilized could grow as your environment adds more databases to be backed up.

If you have a start with a smaller number of IP addresses, keep an eye on the number of available IP address in subnets registered with the recovery service to ensure you have room to grow.

Recovery Service failure checks

 When using the Autonomous Recovery Service there are some prerequisites that need be met. I have a checklist that goes through these requirements, and you can find that checklist here.

This blog post will help you perform some basic debugging and demonstrate what errors you will see if you miss some of the steps.

I want to point out that Billy Zou created a create post that will help you work through issues with Cross-region restore. Billy's post has some great information to use for debugging and you can find it here.

This post is broken into two possible places where you will have issues.

1. Unable to Submit request. This can be caused by
• Policy issues
• Limits issue
2. You submitted backup, but it failed to configure the Recovery Service. This can be caused by
• DNS issues with resolving FQDN used by Recovery Service
• Routing/port issues accessing the Recovery Service or Object Storage

Unable to submit Autonomous Recovery Service as a backup location

Policies for the tenancy

The first step is to ensure that you have configured policies for the recovery service.  The easiest way to do this is by utilizing Policy Builder.

NOTE: There is a policy that grants access to the "ADMIN" group. If your administrator group is a different group, you would 

Visible Issue

 If policies are not configured properly, you find that "Recovery Service" is greyed out as an option.

Limits for the Recovery Service

By default if you are not in a multi-cloud environment your paid tenancy will have a limit of

• 10 Database
• 10 TB of backups storage

If you are using Multi-cloud, and your database is in partner cloud, there is no default limits (defaulting to 0!), which means you have to apply for a limit increase!

This is the most common issue I see with multicloud.  You need to set the limit specifically for the multi-cloud subscription.

Visible Issue

 If limits  are not configured properly, you find that "Recovery Service" is greyed out as an option.

Below the choice for "Recovery service", you will see that there is a warning, telling you that you have exceeded your limits.

Sudden ORA-12578: TNS:wallet open failed when logging in as SYS

ORA-12578: TNS:wallet open failed when logging in as SYS

This blog post covers a possible cause of a "ORA-12578: TNS:wallet open failed" error when trying to log into your database using 

>sqlplus / as sysdba

I have seen this issue a few times with DB 19.x.  I noticed the behavior changed with AI DB 26 and is much less likely to happen.

What causes this ?

The most likely reason why you would suddenly see this error code when trying to log in using "sys as sysdba" is a change to the sqlnet.ora file.

When logging in using "sys as sysdba", the sqlnet.ora file used by your environment will be parsed as part of the authentication process.  If the sqlnet.ora in your environment has any issues during the parsing, this will cause your login using "sys as sysdba" to fail with the above error.

Fortunately, this does not happen in AI DB 26.

How to test for the sqlnet.ora being the cause

The easiest way to test is by using the TNS_ADMIN environmental variable setting.

The steps I would follow are.

1. cd to your $ORACLE_HOME/network/admin directory on the server
2. Execute mkdir to create a new directory named "test"
3. cd to that new directory "test"
4. set TNS_ADMIN with "export TNS_ADMIN=$ORACLE_HOME/network/admin/test"
5. Try logging in using "/ as sysdba"

Since there is no sqlnet.ora file in this new directory, if you can successfully login we have proven that the issue is the sqlnet.ora file.

Now that we have proven it is the sqlnet.ora (or ruled it out sorry I couldn't help), we can look at the causes.

Finding the issue

Now that you have a new directory, $ORACLE_HOME/network//admin/test, we can start working through the possible causes.

Step 1- copy the sqlnet.ora from the default location to this new directory so that we can update it and find the issue without affecting other users. "cp ../*.ora ."

Below is my sqlnet.ora that I am showing different issues with.

# sqlnet3189722425551944721.ora Network Configuration File: /tmp/sqlnet3189722425551944721.ora
# Generated by Oracle configuration tools.

SQLNET.WALLET_OVERRIDE = true

NAMES.DIRECTORY_PATH= (TNSNAMES, ONAMES, HOSTNAME)

WALLET_LOCATION =
 (SOURCE =
 (METHOD = FILE)
 (METHOD_DATA =
 (DIRECTORY = /opt/oracle/admin/ORCLCDB/wallet1/server_seps)
 )
)

Cause 1 - Wallet file

The first possible cause is that the location of the wallet location is not correct. The same issue will most likely occur if you are setting the encryption_wallet_location in the sqlnet.ora file.

Both of these must be true when looking at the wallet file.

1. The directory in the sqlnet.ora file MUST exist. If the directory location is incorrect, you will have an issue opening the wallet
2. There must be a wallet file in that directory. Not only must the directory exist, but there must also be a wallet file within the directory to read.

Cause 2 - Syntax in sqlnet.ora file

When the database parses the sqlnet.ora file, it can be sensitive to any issues within the sqlnet.ora file.  Simple issues can cause parsing failures, and cause your login to fail.

Some of the most common issues are:

1. Hidden characters in the file. This can happen when copying across platforms (windows to Linux for example). If there are any characters in the file that cause parsing to fail, your login will fail.
2. Missing "(" or ")". This can cause parsing to fail, and your login will also fail.
3. Starting "(" in the first column.  Unfortunately this causes a parsing failure. This can be the most annoying, and difficult to find cause. 

Below is an example of a sqlnet.ora file that fails, and you can compare to my sqlnet.ora at the beginning of this blog.

# sqlnet3189722425551944721.ora Network Configuration File: /tmp/sqlnet3189722425551944721.ora
# Generated by Oracle configuration tools.

SQLNET.WALLET_OVERRIDE = true

NAMES.DIRECTORY_PATH= (TNSNAMES, ONAMES, HOSTNAME)

WALLET_LOCATION =
 (SOURCE =
 (METHOD = FILE)
 (METHOD_DATA =
(DIRECTORY = /opt/oracle/admin/ORCLCDB/wallet/server_seps)
 )
)

Can you spot the difference ? 

...

...

It's the "(" before the word "DIRECTORY". When it is in the first column, parsing fails.

Prevention-

What I tell customers to avoid any issues like this is the following:

• NEVER change the default sqlnet.ora for all databases. This is the copy that is stored in the $ORACLE_HOME/network/admin directory.
• ALWAYS set the WALLET_ROOT parameter in the database. This is interpreted first by the database, and replaces the encryptioin_wallet_location in the sqlnet.ora file.
• ALWAYS put the SEPS wallet for Real-time redo with the ZDLRA in the {WALLET_ROOT}/server_seps directory. Even if it is a symbolic link.
• ALWAYS use TNS_ADMIN when it is necessary to customize the sqlnet.ora.  When backing up using the ZDLRA I recommend customers create a customized sqlnet.ora file and use TNS_ADMIN when executing backup/restore scripts.

This will prevent any issues with the shared sqlnet.ora that may cause unexpected issues with logging in as sys.

SUMMARY:

If you suddenly receive a ORA-12578: TNS:wallet open failed when logging in as SYS the first place to look would be your sqlnet.ora file for any parsing errors.