Tag Archives: recovery

What makes a DBA?

I wrote this article as a foreword for the 2007 Apress book “RMAN Recipes for Oracle Database 11g: A Problem-Solution Approach” by Darl Kuhn, Sam Alapati, and Arup Nanda (ISBN 1590598512), and I’m pleased to learn it will be included in the exciting new Apress update “RMAN Recipes for Oracle Database 12c: A Problem-Solution Approach” (ISBN 143024836X), scheduled for 14-Aug 2013 publication, assuming that Oracle Database 12c^H^H^HNextGeneration is released prior to then…

RMAN Recipes 12c cover
What skills set the database administrator (DBA) apart from other technologists? Of the many responsibilities laid upon a DBA, which cannot be performed by someone else?
Adding database accounts? Creating tables and indexes? Installing and configuring databases? Optimizing the database and the applications that access and manipulate it?
All of these things are regularly performed by people who do not consider themselves database administrators. They consider themselves to be programmer/analysts, to be application developers, to be managers and directors, and they do all these things just to be able to move forward with their own job. Most application developers know how to run the Oracle Universal Installer – it’s just another graphical application, and accepting all the default choices is a perfectly valid way to get the job done, these days. Adding database accounts? That’s easy! Granting database privileges? Just give ‘em “DBA” or “SYSDBA” and no more problems! Creating tables and indexes? C’mon, that’s more of a developer’s job than the DBA’s job, isn’t it? Tuning Oracle databases is mostly about crafting efficient SQL statements, and while this job often falls to DBAs, it is best handled by the developers and programmers who write the SQL in the first place.
While many of these duties are correctly assigned to a DBA, they are not a hallmark of the job.
Think about the people flying airliners. With the degree of automation in aircraft cockpits now, it can be argued (with a lot of merit) that the planes can fly themselves, from take-off, through navigated flight, to touch-down. So, what are the pilots for?
If something goes wrong with the plane, you want the best pilots at the controls of that plane. Because when things go wrong, they go wrong in a hurry, and it takes somebody who knows exactly what all that PlayStation gadgetry is really controlling in that cockpit, and it takes somebody who can intelligently take control and land the thing safely when dozens of lights are flashing and dozens of alarms are buzzing. It’s not too hard to justify the presence of pilots on airplanes, in the end.
Likewise, fifty years ago, at the dawn of the American space program, a debate was underway then, as there is now – should space flights be manned or unmanned? There were good arguments in favor of the latter. The first astronauts weren’t human – they were dogs and chimps. When humans were finally included, the spacecraft engineers assured them that they were redundant, just along for the ride, superfluous, and that they were just “spam in a can”, went the gallows humor.
But it didn’t take long to prove those people wrong. The presence of a well-trained and comprehensively knowledgeable pilot in the spacecraft has proven its worth, time and time again. A classic example is the final 2 minutes of the historic Apollo 11 moon landing, when Neil Armstrong looked out the window of the Eagle lunar module and realized that their automated descent, controlled from Houston via computer, was dropping them into a boulder field. Only a few hundred meters from the lunar surface, Armstrong flipped the controls to manual and pushed the lunar module higher, seeking a more viable landing site. While Houston nervously and repeatedly queried for status, Armstrong calmly replied, “Hold, Houston” until, with only 30 seconds of fuel remaining, he set the lunar module down and declared that the Eagle had landed.
That why we have human astronauts. This is what sets “spam in a can” apart from a pilot. This is why airliners, while heavily automated, have highly-trained pilots at the controls.
Which brings us back to database administrators … I hope!
What sets a DBA apart from an ambitious programmer or a developer doing what needs to be done to move forward?
It is the ability to prepare for trouble and recover from it. Database recovery in the event of failure or mishap is the most vital skill in a DBA’s toolkit.
The Oracle RDBMS has been around now for about 30 years. The internal mechanisms for backup and recovery have changed very little in the past 20 years. Of course there have been enhancements, but the basic mechanism for basic “hot” or online backups has changed very little.
However, it is the mechanism for restore and recovery that took a great leap forward 10 years ago, when Oracle Recovery Manager (RMAN) was introduced with Oracle8. In a world where misnomers abound, Recovery Manager is quite aptly named. The focus of the product is not on automating backups, but rather on automating the steps of restore and recovery as much as possible.  Much of the early reluctance to adopt RMAN came about not from any failings in the product, but rather from disappointment that the product did not make the job of performing backups any easier.  Since backups are the operation that DBAs see most often, what RMAN does for recovery operations was not fully appreciated.
As I teach people how to use RMAN, I attempt to stress the mindset that RMAN is not just about performing backups. Rather, it is about “feeding” the RMAN “recovery catalog”.  Backups are not ends in themselves, but simply entries in the recovery catalog used by RMAN during restore and recovery operations. If a DBA considers it their duty to feed the recovery catalog with backup operations and other maintenance such as crosschecks, then we have someone who is truly preparing for the eventuality, not just the remote possibility, of restore and recovery. Someone who understands the tool, and is not just applying a different tool to bang in nails the same old way.
The knowledge and capability to recover a database from catastrophic failure is what separates a real DBA apart from someone who found the installer or who know how to do the clickety-clickety thing in Oracle Enterprise Manager. And not just once, by luck, but knowing how to use RMAN to its full advantage, to work around those confusing and misleading error messages, to verify backups and maintain and protect the recovery catalog(s) so as to virtually guarantee recoverability, each and every time.
It is this protective mindset, liberally seasoned with caution and pessimism, which separates DBAs from other technologists. Systems administrators and network administrators have much the same tendencies, but only databases administrators are made responsible for never losing data.  Systems and networks can be made redundant, and if they fail it is only a matter of bringing them back to service, but data loss is forever and is never forgiven.
Years ago, I worked with a very no-nonsense vice-president. She didn’t want to know the details of my job, and rightly so. She simply stated, very clearly, “Failures happen, but don’t EVER tell me that you could not recover my data”. Message received.
This book is written by seasoned professionals who have been using RMAN since its inception. They have recognized that RMAN can be confusing, and feel that everyone should not have to go through the same learning curve in order to arrive at the same conclusions. So, they have gathered together their best practices and tried-and-true procedures and compiled them into this wonderful book.
If you are an Oracle database administrator, this could very well be the most important book you read. Technology books are famous for becoming “shelf-ware”, pristine and unopened books adorning shelves everywhere. This book will be the exception – the book that is dog-eared and worn, the cover falling off and pages smudged, found more often opened face down on a desk than perched serenely on a shelf. The information within this book is the very essence of the job of the Oracle DBA, the most important facet of the job, and I am grateful to Sam, Arup, and Darl for sharing.