Taking backups during development, testing, or upgrades

The process of upgrading an Oracle E-Business Suites (EBS) system is one requiring dozens or hundreds of steps. Think of it like rock climbing. Each handhold or foothold is analogous to one of the steps in the upgrade.

What happens if you miss a handhold or foothold?

If you are close to the ground, then the fall isn’t dangerous. You dust yourself off, take a deep breath, and then you just start climbing again.

But what happens if you’re 25 or 35 feet off the ground? Or higher?

A fall from that height can result in serious injury, perhaps death, unless you are strapped to a climbing rope to catch your fall.  And hopefully that climbing rope is attached to piton or chock not too far from your current position.  The higher you climbed from your most recent anchor, the further you fall, and more painful.  If you set the last chock 10 feet below, that means you’re going to fall 20 feet, which is going to HURT.  It makes more sense to set chocks every 5 feet, so your longest drop will only be 10 feet at worst.

And that is the role that backups play in any lengthy process with many discrete steps, such as an EBS upgrade. Performing a backup of the systems you’re upgrading every half-dozen or dozen steps allows you to restore to the most recent backup and resume work while having only lost a relatively small amount of work.

But think of the costs of taking a full backup of the systems you’re upgrading?

Let’s assume that the EBS database is 1 TB in size, the dbTier is 4 GB, and the appsTier is another 4 GB. All told each backup takes up over a TB of backup media and takes 2 hours to perform, start to finish. So, for an upgrade process of 100 steps, and performing a backup every dozen steps, means performing 8-9 backups, at 2 hours apiece, totaling 16-18 hours alone! So not only do the backups represent a significant portion of the total number of steps to perform, but those steps also consume a significant portion of the total elapsed time of the entire upgrade process.

So, in the event of a mistake or mishap, it will probably take about 2 hours to restore to the previous backup, and then the intervening steps have to be replayed, which might take another couple of painstaking hours. Assuming there’s nothing wrong with the backup. Assuming that tapes haven’t been sent offsite to Iron Mountain. Assuming… yeah, assuming…

So just as with climbing, taking periodic backups of your work protects you against a fall.  But if each backup takes 2 hours, are you going to do that after every 15 mins?  Or after every hour?  After every 2 hours?  After every 4 hours?  Doing those time-consuming backups allows more productive work, but think of how far you’re going to fall if you slip?

Because of this high cost of failure, believe it that each step will be double-checked and triple-checked, perhaps by a second or third person, before it is committed or the ENTER key is pressed.

No wonder upgrading EBS is so exhausting and expensive!

The fastest operation is the one you never perform

What if you never had to perform all those backups, but you could still recover your entire EBS systems stack to any point-in-time as if you had?

This is what it is like to use data virtualization technology from Delphix. You start by linking a set of “source” EBS components into Delphix, and these sources are stored in compressed format within the Delphix appliance. Following that, these sources can produce virtual copies taking up very little storage in a matter of minutes. A Delphix administrator can provision virtual copies of the entire EBS systems stack into a single Delphix JetStream container for use by an individual or a team. The components of the container, which include the EBS database, the EBS dbTier, and the EBS appsTier, become the environment upon which the upgrade team is working.

Delphix captures all of the changes made to the EBS database, the EBS dbTier, and the EBS appsTier, which is called a timeflow. Additionally, Delphix JetStream containers allow users to create bookmarks, or named points-in-time, within the timeflow to speed up the location of a significant point-in-time, and easily restore to it.

So each developer or team can have their own private copy of the entire EBS systems stack. As the developer is stepping through each step of the upgrade, they can create a bookmark, which takes only seconds, after every couple steps. If they make a mistake or a mishap occurs, then they can rewind the entire JetStream container, consisting of the entire EBS systems stack, back to the desired bookmark in minutes, and then resume from there.

Let’s step through this, using the following example EBS system stack already linked and ready to use within a Delphix appliance…

Understanding concepts: timeflows, snapshots, and bookmarks

A dSource is the data loaded into Delphix from a source database or filesystem. Initially a dSource is loaded (or linked) into Delphix using a full backup, but after linking, all changes to the source database or filesystem are captured and stored in the Delphix dSource.

A timeflow is an incarnation of a dSource. The full lifecycle of a dSource may include several incarnations, roughly corresponding to a RESETLOGS event in an Oracle database. Delphix retains not just the current point-in-time of a dSource (as like replication packages like Golden Gate or Data Guard), but every action ever performed from all incarnations, from the time the dSource was initially linked into Delphix, up to the current point-in-time.

Even though Delphix compresses, de-duplicates, and performs other clever things to make the data loaded much smaller (i.e. usually about 30% of the original size), you still cannot retain everything forever. It’s just impossible. To deal with this, data is purged from timeflows that are older than the retention policy configured by the Delphix administrator.

Another useful concept to know is a snapshot, which is a small data structure that represents a usable full virtual image of a timeflow at a specific point-in-time.   Snapshots in Delphix can portray a full Oracle database as a virtual image of a point-in-time, and snapshots are subject to purge during the retention policy configured with the timeflow.

A bookmark is a named snapshot that is not subject to the configured retention policy on the timeflow, but instead can be configured with its own private retention policy, which defaults to “forever”.  Bookmarks are intended to represent points-in-time that have meaning to the end user.

An example EBS system stack in Delphix

So here we see a full E-Business Suites R12.1 system stack linked into Delphix. In the screenshot below, you can see three dSource objects in the left-hand navigation bar, under the Sources database group or folder. These three dSource objects are named…

1. EBS R12.1 appsTier
2. EBS R12.1 dbTechStack
3. EBSDB

The first two items are vFiles, containing the file system sub-directories representing the E-Business Suites R12.1 applications tier and database tier, respectively. The last item is a virtual database or VDB, containing the Oracle database for the E-Business Suites R12.1 system. Each are indicated as dSources by the circular black dSource icon with the letters “dS” within (indicated below by the red arrows)…

Registering EBS dSources as JetStream templates

The three dSources have been encapsulated within the Delphix JetStream user-interface within a templates, as shown by the JetStream icons showing a tiny hand holding out a tiny cylinder representing a datasource (indicated below by the red arrows) just to the right of the dSource icons…

Within the JetStream user-interface itself, we see that all three dSources are encapsulated within one JetStream template, with the three dSources comprising the three datasources for the template…

Encapsulating EBS VDBs as JetStream containers

Once dSources are encapsulated as a JetStream template, then we can encapsulate VDBs and vFiles created from those dSources as JetStream containers. Here, we see the database group or folder called Targets circled in the left-hand navigation bar, and the blue icons for VDBs and vFiles with the letter “V” within are indicated by red arrow, while the darker icons for JetStream Containers are indicated by the green arrow…

Working with JetStream containers for EBS

With an entire EBS system stack under the control of Delphix JetStream, we can begin the lengthy process of upgrading EBS. Before we begin, we should create a JetStream bookmark called Upgrade Step 0 to indicate the starting point of the upgrade…

Then, after performing the first five steps of the upgrade in the EBS system stack, come back to the JetStream user-interface and create a JetStream bookmark named Upgrade Step 5, as shown below…

After every couple steps of the upgrade, or prior to any lengthy step, come back to the Delphix JetStream user-interface and create a bookmark named appropriate for the step in the upgrade process, as shown below…

Performing a rewind in a JetStream container for EBS

So we’re cruising along, performing productive steps in the EBS upgrade process, and not worrying about making backups.

WHOOPS! At upgrade step 59, we made a serious mistake!

If we had been making backups every 12 steps or so, then the most recent backup may have been Upgrade Step 48, which was 11 steps ago. Besides the multiple hours it may take to restore the backup in the database, the dbTier, and the appsTier, we are going to have to perform 11 steps over again.

In contrast, using Delphix JetStream, we have better choices. From the Delphix JetStream user-interface, we can restore to any point-in-time on the timeflow, whether it is one of the bookmarks we previously created at a known step in the upgrade process, or to any point-in-time location on the timeflow line, which may or may be located in the middle of one the upgrade steps, and thus not a viable point from which to restart.

So, because we want to restart the upgrade from a known and viable step in the process, we’re going to restore the entire EBS systems stack consistently back to the JetStream bookmark Upgrade Step 55…

So, first let’s click on the JetStream bookmark named Upgrade Step 55 and then click on the icon for the Restore operation to start the rewind…

After being prompted Are You Sure and then confirming Yes I Am, then the Restore process starts to create a brand new timeflow for all three datasources in the container…

Please notice that the old timeflow is still visible and accessible to the left in the JetStream user-interface, even with the new timeflow being restored to the right.

Finally, the restore operation completes, and now all three parts of the EBS systems stack (i.e. appsTier, dbTier, and database) have been restored to Upgrade Step 55 in about 15 minutes elapsed time, a fraction of the 2 hours it would have taken to restore from backups.

Now we can resume the EBS upgrade process with barely any time lost!

Summary of benefits

So a few things to bear in mind from this example when using Delphix JetStream…

1. Once a JetStream container has been provisioned by the Delphix administrators (usually a role occupied by database administrators or DBAs), all the operations shown above are performed by JetStream data users, who are usually the developers performing the upgrade themselves
   • No need to coordinate backups or restores between teams == less time wasted
2. Using Delphix removed the need to perform backups, because all changes across the timeflow of all three components of the EBS systems stack (i.e. appsTier, dbTechStack, and database) are recorded automatically within Delphix
   • No backups == less storage consumed == less time wasted
3. Using Delphix removed the need to restore from backups, because the timeflow can be restored to any previous point-in-time in a fraction of the time it takes to perform a restore from backup
   • No restores == less time wasted

It is difficult to imagine not performing EBS upgrades using the methods we’ve used for decades, but the that time is here. Data virtualization, like server virtualization, is fast becoming the new norm, and it is changing everything for the better.

Free webinar

Please view the joint webinar on these topics, featuring Mike Swing of TruTek and Kyle Hailey of Delphix and myself, on the topic of “Avoiding The Pitfalls Of An Oracle E-Business Suites Upgrade Project” about how data virtualization can reduce the risk and accelerate the completion of EBS upgrades, recorded on 07-Apr 2016.

You can also download the slidedeck from the webinar here.