What makes a DBA?

Leave a reply

I wrote this article as a foreword for the 2007 Apress book “RMAN Recipes for Oracle Database 11g: A Problem-Solution Approach” by Darl Kuhn, Sam Alapati, and Arup Nanda (ISBN 1590598512), and I’m pleased to learn it will be included in the exciting new Apress update “RMAN Recipes for Oracle Database 12c: A Problem-Solution Approach” (ISBN 143024836X), scheduled for a 30-March 2013 publication.

RMAN Recipes 12c cover
What skills set the database administrator (DBA) apart from other technologists? Of the many responsibilities laid upon a DBA, which cannot be performed by someone else?
Adding database accounts? Creating tables and indexes? Installing and configuring databases? Optimizing the database and the applications that access and manipulate it?
All of these things are regularly performed by people who do not consider themselves database administrators. They consider themselves to be programmer/analysts, to be application developers, to be managers and directors, and they do all these things just to be able to move forward with their own job. Most application developers know how to run the Oracle Universal Installer – it’s just another graphical application, and accepting all the default choices is a perfectly valid way to get the job done, these days. Adding database accounts? That’s easy! Granting database privileges? Just give ‘em “DBA” or “SYSDBA” and no more problems! Creating tables and indexes? C’mon, that’s more of a developer’s job than the DBA’s job, isn’t it? Tuning Oracle databases is mostly about crafting efficient SQL statements, and while this job often falls to DBAs, it is best handled by the developers and programmers who write the SQL in the first place.
While many of these duties are correctly assigned to a DBA, they are not a hallmark of the job.
Think about the people flying airliners. With the degree of automation in aircraft cockpits now, it can be argued (with a lot of merit) that the planes can fly themselves, from take-off, through navigated flight, to touch-down. So, what are the pilots for?
If something goes wrong with the plane, you want the best pilots at the controls of that plane. Because when things go wrong, they go wrong in a hurry, and it takes somebody who knows exactly what all that PlayStation gadgetry is really controlling in that cockpit, and it takes somebody who can intelligently take control and land the thing safely when dozens of lights are flashing and dozens of alarms are buzzing. It’s not too hard to justify the presence of pilots on airplanes, in the end.
Likewise, fifty years ago, at the dawn of the American space program, a debate was underway then, as there is now – should space flights be manned or unmanned? There were good arguments in favor of the latter. The first astronauts weren’t human – they were dogs and chimps. When humans were finally included, the spacecraft engineers assured them that they were redundant, just along for the ride, superfluous, and that they were just “spam in a can”, went the gallows humor.
But it didn’t take long to prove those people wrong. The presence of a well-trained and comprehensively knowledgeable pilot in the spacecraft has proven its worth, time and time again. A classic example is the final 2 minutes of the historic Apollo 11 moon landing, when Neil Armstrong looked out the window of the Eagle lunar module and realized that their automated descent, controlled from Houston via computer, was dropping them into a boulder field. Only a few hundred meters from the lunar surface, Armstrong flipped the controls to manual and pushed the lunar module higher, seeking a more viable landing site. While Houston nervously and repeatedly queried for status, Armstrong calmly replied, “Hold, Houston” until, with only 30 seconds of fuel remaining, he set the lunar module down and declared that the Eagle had landed.
That why we have human astronauts. This is what sets “spam in a can” apart from a pilot. This is why airliners, while heavily automated, have highly-trained pilots at the controls.
Which brings us back to database administrators … I hope!
What sets a DBA apart from an ambitious programmer or a developer doing what needs to be done to move forward?
It is the ability to prepare for trouble and recover from it. Database recovery in the event of failure or mishap is the most vital skill in a DBA’s toolkit.
The Oracle RDBMS has been around now for about 30 years. The internal mechanisms for backup and recovery have changed very little in the past 20 years. Of course there have been enhancements, but the basic mechanism for basic “hot” or online backups has changed very little.
However, it is the mechanism for restore and recovery that took a great leap forward 10 years ago, when Oracle Recovery Manager (RMAN) was introduced with Oracle8. In a world where misnomers abound, Recovery Manager is quite aptly named. The focus of the product is not on automating backups, but rather on automating the steps of restore and recovery as much as possible.  Much of the early reluctance to adopt RMAN came about not from any failings in the product, but rather from disappointment that the product did not make the job of performing backups any easier.  Since backups are the operation that DBAs see most often, what RMAN does for recovery operations was not fully appreciated.
As I teach people how to use RMAN, I attempt to stress the mindset that RMAN is not just about performing backups. Rather, it is about “feeding” the RMAN “recovery catalog”.  Backups are not ends in themselves, but simply entries in the recovery catalog used by RMAN during restore and recovery operations. If a DBA considers it their duty to feed the recovery catalog with backup operations and other maintenance such as crosschecks, then we have someone who is truly preparing for the eventuality, not just the remote possibility, of restore and recovery. Someone who understands the tool, and is not just applying a different tool to bang in nails the same old way.
The knowledge and capability to recover a database from catastrophic failure is what separates a real DBA apart from someone who found the installer or who know how to do the clickety-clickety thing in Oracle Enterprise Manager. And not just once, by luck, but knowing how to use RMAN to its full advantage, to work around those confusing and misleading error messages, to verify backups and maintain and protect the recovery catalog(s) so as to virtually guarantee recoverability, each and every time.
It is this protective mindset, liberally seasoned with caution and pessimism, which separates DBAs from other technologists. Systems administrators and network administrators have much the same tendencies, but only databases administrators are made responsible for never losing data.  Systems and networks can be made redundant, and if they fail it is only a matter of bringing them back to service, but data loss is forever and is never forgiven.
Years ago, I worked with a very no-nonsense vice-president. She didn’t want to know the details of my job, and rightly so. She simply stated, very clearly, “Failures happen, but don’t EVER tell me that you could not recover my data”. Message received.
This book is written by seasoned professionals who have been using RMAN since its inception. They have recognized that RMAN can be confusing, and feel that everyone should not have to go through the same learning curve in order to arrive at the same conclusions. So, they have gathered together their best practices and tried-and-true procedures and compiled them into this wonderful book.
If you are an Oracle database administrator, this could very well be the most important book you read. Technology books are famous for becoming “shelf-ware”, pristine and unopened books adorning shelves everywhere. This book will be the exception – the book that is dog-eared and worn, the cover falling off and pages smudged, found more often opened face down on a desk than perched serenely on a shelf. The information within this book is the very essence of the job of the Oracle DBA, the most important facet of the job, and I am grateful to Sam, Arup, and Darl for sharing.

Remembering Gary Dodge…

Leave a reply

The world lost a remarkable person this week, my friend and mentor Gary Dodge.

He is survived by his wife Luann, to whom he was married 33 years, by his daughter Brigid and by his son Ryan, and by a tight-knit and equally talented and accomplished family.  And by friends and admirers too numerous to count, worldwide.

As long as I knew him, his email signature stated, “Building tomorrow’s legacy systems today, one crisis at a time“, succinctly expressing his dry, lightly-warped sense of humor, suitable even in an uptight business environment.

He is deeply missed.  Thank you, Gary.

Kellyn Pot’vin in Oracle Magazine: “Peer To Peer: Green Is Good”

1 Reply

A new way to think about things…

I’ve always felt that organizations are too quick to KIWI (“Kill It With Iron“) to fix system performance issues, rather than understanding the cause of the performance problem and attempting to address it.

Kellyn provides a thought-provoking perspective in an Oracle Magazine interview on the “green” advantages of tuning and optimizing, making existing infrastructures last longer.

The niche for RAC

1 Reply

Recently, I received the following question from a respected colleague about Oracle’s Real Application Clusters (RAC) software…

I found out from Oracle reps that we have a license for RAC but are not using it. Our DBA confirmed we have no RAC databases.  All 11gR2, but non-RAC.

So what are the primary reasons you would use RAC anyway?

My response was lengthy, which wouldn’t surprise anyone who knows me, but in summary I concluded…

Unless they are encountering the scenario where their current server platform cannot scale to the needs of the application, convert the RAC licenses into more licenses for other Oracle products, if possible.

If not possible, leave the RAC licenses unused, write them off.  The least cost-effective course of action would be to employ the RAC licenses just because they’re there, as that would be throwing more good money after bad.

I know it sounds like I’m a real hater, but is there any validity behind that assessment?

Oracle Corporation markets Real Application Clusters (RAC) as both a high-availability (HA) solution and a high-performance solution.  Let’s address both of those sales points…

The reality is that RAC is not high-availability.  With a non-redundant copy of the database, it wasn’t designed for high-availability.  Any HA capabilities that Oracle RAC might seem to provide are better provided using Oracle Data Guard, which unlike RAC is designed with complete database redundancy for high-availability. So, HA as a reason for using RAC must come off the table completely.  If you’re serious about HA, use Oracle Data Guard.

That leaves performance, for which RAC was obviously designed by employing the resources of two or more servers on a single database. However, let’s define what we mean by “performance”.  It is easy to demonstrate that RAC does not increase the performance of an individual transactions or queries; on the contrary, it reduces the performance of an individual transaction by adding waits on “global cache” (a.k.a. “gc” or “gcs”) class events on top of I/O events, and adding waits on “global enqueue” (a.k.a. “ges”) class events to certain concurrency events. It is quite easy to visualize this RAC overhead on standard non-RAC operations by simply looking at the “Performance Graph” of Oracle Enterprise Manager, and noticing the “gray” representing events of class “Cluster” correlating closely to the “blue” representing events in class “User I/O” and “red” representing events in class “Concurrency” in the stacked chart on that page.

While RAC decreases performance by adding overhead in the form of “cluster” waits, it does permit scalability of many more transactions and queries accessing the database by increasing the pool of server resources, by permitting operations to execute on more than one server.

And this is the niche for RAC.

RAC is not HA, RAC decreases the performance of individual operations, yet RAC enables more operations to execute by scaling across individual servers.

Therefore, the *only* technical reason I would ever employ RAC is when my database workload exceeds the maximum capacity of the resources of a single server of my hardware platform, whether that resource is CPU, I/O, memory, or kernel parameters.

The hey-day of RAC was the era of the commodity 32-bit platforms of Windows and Linux, through the turn of the century up to about 2004.  With drastic limitations on the size and configuration of these 32-bit platforms, clustering servers using RAC was a viable method of employing these commodity servers for larger database workloads.  But with the advent of 64-bit Windows and Linux, these servers can scale larger, providing more CPU and memory, reducing the need for clustering dramatically.

Yet, Oracle continues to promote RAC as a general-purpose solution, striving to convince the market that running Oracle database without RAC is entry-level, while RAC is the advanced solution for visionaries to optimize both availability and performance, obtaining total data processing creaminess, avoiding the chewy chunks of degradation.

So, characterizing the employment of unused RAC licenses as “throwing good money after bad” is not negative hyperbole or product bashing, just common-sense if the technical reason for using RAC isn’t relevant.  In addition to the 50% upllift in licensing and support costs, the total cost of ownership (TCO) for RAC includes the additional hardware resources to compensate for the waits on “cluster” events, the additional time and effort by not only database administrators but also systems administrators to install and maintain shared storage, additional networks, clusterware, and RAC databases.

RAC is the right tool for the job, but be sure it is the right job.  The number of occasions when it is needed is quite small, and getting smaller as the state of the art moves forward.