Presenting “UNIX/Linux Tools For The Oracle DBA” at #C16LV

Going back to the invention of the graphical user interface (GUI) in the 1970s, there has been tension between the advocates of the magical pointy-clickety GUI and the clickety-clackety command-line interface (CLI).

Part of it is stylistic… GUI’s are easier, faster, more productive.

Part of it is ego… CLI’s require more expertise and are endlessly customizable.

Given the evolutionary pressures on technology, the CLI should have gone extinct decades ago, as more and more expertise is packed into better and better GUI’s.  And in fact, that has largely happened, but the persistence of the CLI can be explained by four persistent justifications…

  1. not yet supported in the GUI
  2. need to reproduce in the native OS
  3. automation and scripting
  4. I still drive stick-shift in my car, too

This is the prime motivation for the presentation entitled “UNIX/Linux Tools For The Oracle DBA” which I’m giving at the Collaborate 2016 conference in Las Vegas (#c16lv) on Monday 11-April.

When your monitoring tool gets to the end of the information it can provide, it can be life-saving to be able to go to the UNIX/Linux command-line and get more information.  Knowing how to interpret the output from CLI tools like “top”, “vmtstat”, and “netstat” can be the difference between waiting helplessly for deus ex machina or a flat-out miracle, and obtaining more information to connect the dots to a resolution.

Oftentimes, you have a ticket open with vendor support, and they can’t use screenshots from the GUI tool or report you’re using.  Instead, they need information straight from the horse’s mouth, recognizable, trustworthy, and uninterpreted.  In that case, knowing how to obtain OS information from tools like “ps”, “pmap”, and “ifconfig” can keep a support case moving smoothly toward resolution.

Likewise, the continued survival of CLI’s has a lot to do with custom scripting and automation.  Sure, there are all kinds of formal APIs, but a significant portion of the IT world runs on impromptu and quickly-developed scripts based on shell scripts and other CLI’s.

Last, my car still has a manual transmission, and both my kids drive stick as well.  There is no rational reason for this.  We don’t believe we go faster or do anything better than automatic transmission.  After almost 40 years behind the wheel, it can be argued that I’m too old to change, but that’s certainly not true of my kids.  We just like being a part of the car.

Hopefully all of those reasons help explain why it is useful for Oracle DBAs to learn more about the UNIX/Linux command-line.  You’re not always going to need it, but when you do, you’ll be glad you did.

You can download the slidedeck here.

The DBA is dead. Again.

Mark Twain never said, “Reports of my death are greatly exaggerated.”  Instead, his comment in 1897 was less tongue-in-cheek than matter-of-fact.  Confronted with news reports that he was gravely ill he responded, “James Ross Clemens, a cousin of mine, was seriously ill two or three weeks ago in London, but is well now.  The report of my illness grew out of his illness; the report of my death was an exaggeration.”  I can only hope that, while being equally matter of fact, in the retelling my comments will also grow wittier than they were written.  It is a lot for which to hope, as past experience is that my comments generally provoke unintended offense.

Every few years, when wondrous new automation appears imminent, reports surface about the long-anticipated death of the role of the database administrator.  Sometimes it seems these reports arise out of sheer frustration that DBAs and databases still exist, as seemed to have happened in 2008 during a conversation on the Oak Table email list, which closely followed a similar discussion on the ORACLE-L list.  To whit:  the war is over, and we lost.

Alex Gorbachev commented succinctly at the time:

We have already “lost” the war many times, haven’t we?  We lost it to object-oriented databases (8i?)  We lost to XML databases (9i?)  We lost to grid databases (10g?)  And we are losing to what now with 11g?  The “fusion” will save us all with or *without* databases in the first place?  Yeah right … the end is close.

The focus of discussion on both email lists was a thought-provoking blog post in March 2008 by Dom Brooks entitled “The dea(r)th of Oracle RDBMS and contracting?” He commented that the tide of history had finally turned against the Oracle database and the highly-visible role of database administrator.  Stiff competition from open-source competitors, emerging scalable technologies, absurd license fees, and belt-tightening by many IT shops were the overwhelming trend.  Poor database design exacerbated by immature implementation; if you’re going to produce a disaster, probably best that it not cost as much as Oracle.

My response on both email threads on ORACLE-L and the Oak Table was this…

Back in the 1980s, I worked for a company that had built some really cool applications in the area of travel reservations.  Eventually, the travel providers (i.e. airlines, hotels, car rental agencies, etc) caught on to what we were doing and did it themselves, effectively putting us out of business overnight.  So, it came time to sell the company off in pieces.  We tried to sell the applications, but nobody wanted them — they had their own, or could buy or build better.  We sold the hardware and facilities, but for pennies on the dollar.  Then, when we tried to sell the data, we hit the jackpot — everybody wanted the data, and we were able to sell it over and over again, to multiple buyers.

I never forgot that lesson, and several years later traded being a programmer for being a DBA because (as Michael just said, below) I like working with data.  Data, not programs, is the only thing that matters — applications are transient and have no value except to acquire, manipulate, and display data.  Data is the only thing with value.  The long-term value of data is the reason I’ve moved toward data warehousing and business intelligence, too.

Data is important.  Databases manage data.  DBAs architect, configure, and manage databases.  So, being a skilled database administrator will always be necessary as long as data exists.  If the state of the art ceases advancing, then automation will finally catch up to extinguish the DBA role/job.  But until then, being a DBA is a career.

That’s my story.  And I’m stickin’ to it.

Doug Burns was following both threads and was kind enough to lend his support in a post to his blog entitled “There’s Hope For Us All“, in which he stated “although it doesn’t reflect my personal experience in the slightest, there was something about what he had to say and the way he said it that rung very true to me.”  Kinder words are rarely spoken, and thank you, Doug.  And thank you too Dom, for your follow-up comment to Doug’s post, “Solidarity Brother!  I’m sure Tim’s right and will continue to be right.  I was having an emotional moment… the flat earth society are everywhere!

We all have those moments.

And here we are again, having another moment.

Once again, the topic of discussion on the Oak Table list was a blog post from Kenny Gorman (no relation) entitled “The Database Administrator Is Dead.”  My father, who was a police officer for 25 years, worked in a profession much more dangerous, and certainly several people had wished him harmed or dead over his career and even acted in that direction, but in a general way my chosen profession has received more death threats, it seems.

Now, the forces opposing the DBA are not necessarily cheaper, different, or disruptive technology, but better automation and provisioning.  The role of the DBA will literally be smothered out of existence, as highly-automated management consoles extend to the ultimate capability.  “Database As A Service” or “DBaaS“, cloud provisioning for databases, is the next development to obsolesce the database administrator.

The synchronicity of these discussions is spooky.  During the week previous to the discussion of Mr. [Kenny] Gorman’s blog post, I had related another particular story 4-5 separate times to 4-5 separate people, and now I found that I was relating it yet again, this time to the Oak Table email list.  It was something of a continuation from my earlier story…

In the 1990s, when I chose to move from being a developer to a DBA, the trend of out-sourcing was quite abundantly evident, not quite augmented by the trend of offshoring yet.  In 1999 I did my first ever keynote address at a conference in Portland, Maine to the Maine’S Oracle Users Group (MSOUG) on the topic of being a DBA in a world of out-sourcing.  I described a visualization of one of those water-holes in the Sahara.  A water-hole that is brimming and supporting a lush oasis during the rainy season, but that dries up and shrinks to a small muddy puddle during the dry season, surrounded by dead vegetation and dead animals that didn’t quite make it to the water-hole or another.

Repeating the comments in Doug’s blog, code comes and goes but data is the only thing with lasting value.  I visualized that the dead vegetation and dead animals surrounding the muddy remainders of the water-hole were developers and DBAs whose jobs were outsourced.  Right in the middle of the muddy water were two eyes above the surface, and this was the skilled DBA, guarding the remainder of the water-hole, containing only the most important stuff that couldn’t be outsourced or offshored.  I had long decided that I would be that DBA, and stay as close to the data as I could, and especially the most strategic data (i.e. data warehousing).

I figure y’all might have as much fun as the good folks at MSOUG did with that visualization, especially when subjected to Freudian and Jungian dream analysis.

Though it has nothing to do with why I’ve related this story 4-5 times previously this week, in this context, the author of the article (we’re not related) talks about having been an Oracle DBA 15 years ago, which is about the time I did my keynote for MSOUG.

Perhaps he left the field too early too early?  🙂

I completely agree with his “automate or die” comment, and I might add “keeping learning or die”, and of course the job’s roles are changing, but besides DBaaS being a long way from the pointy-and-clicky utopia that this post implies, the question remains: who sets up the DBaaS environments?  DBaaS isn’t the end of the DBA role, it is more automation.

Who will set up DBaaS environments, if not DBAs?  Don’t get me wrong:  I agree that DBaaS is here.  And I think DBAs will set it up, use it, and improve on it.

That’s my story.  And I’m stickin’ to it.

What makes a DBA?

I wrote this article as a foreword for the 2007 Apress book “RMAN Recipes for Oracle Database 11g: A Problem-Solution Approach” by Darl Kuhn, Sam Alapati, and Arup Nanda (ISBN 1590598512), and I’m pleased to learn it will be included in the exciting new Apress update “RMAN Recipes for Oracle Database 12c: A Problem-Solution Approach” (ISBN 143024836X), scheduled for 14-Aug 2013 publication, assuming that Oracle Database 12c^H^H^HNextGeneration is released prior to then…

RMAN Recipes 12c cover
What skills set the database administrator (DBA) apart from other technologists? Of the many responsibilities laid upon a DBA, which cannot be performed by someone else?
Adding database accounts? Creating tables and indexes? Installing and configuring databases? Optimizing the database and the applications that access and manipulate it?
All of these things are regularly performed by people who do not consider themselves database administrators. They consider themselves to be programmer/analysts, to be application developers, to be managers and directors, and they do all these things just to be able to move forward with their own job. Most application developers know how to run the Oracle Universal Installer – it’s just another graphical application, and accepting all the default choices is a perfectly valid way to get the job done, these days. Adding database accounts? That’s easy! Granting database privileges? Just give ‘em “DBA” or “SYSDBA” and no more problems! Creating tables and indexes? C’mon, that’s more of a developer’s job than the DBA’s job, isn’t it? Tuning Oracle databases is mostly about crafting efficient SQL statements, and while this job often falls to DBAs, it is best handled by the developers and programmers who write the SQL in the first place.
While many of these duties are correctly assigned to a DBA, they are not a hallmark of the job.
Think about the people flying airliners. With the degree of automation in aircraft cockpits now, it can be argued (with a lot of merit) that the planes can fly themselves, from take-off, through navigated flight, to touch-down. So, what are the pilots for?
If something goes wrong with the plane, you want the best pilots at the controls of that plane. Because when things go wrong, they go wrong in a hurry, and it takes somebody who knows exactly what all that PlayStation gadgetry is really controlling in that cockpit, and it takes somebody who can intelligently take control and land the thing safely when dozens of lights are flashing and dozens of alarms are buzzing. It’s not too hard to justify the presence of pilots on airplanes, in the end.
Likewise, fifty years ago, at the dawn of the American space program, a debate was underway then, as there is now – should space flights be manned or unmanned? There were good arguments in favor of the latter. The first astronauts weren’t human – they were dogs and chimps. When humans were finally included, the spacecraft engineers assured them that they were redundant, just along for the ride, superfluous, and that they were just “spam in a can”, went the gallows humor.
But it didn’t take long to prove those people wrong. The presence of a well-trained and comprehensively knowledgeable pilot in the spacecraft has proven its worth, time and time again. A classic example is the final 2 minutes of the historic Apollo 11 moon landing, when Neil Armstrong looked out the window of the Eagle lunar module and realized that their automated descent, controlled from Houston via computer, was dropping them into a boulder field. Only a few hundred meters from the lunar surface, Armstrong flipped the controls to manual and pushed the lunar module higher, seeking a more viable landing site. While Houston nervously and repeatedly queried for status, Armstrong calmly replied, “Hold, Houston” until, with only 30 seconds of fuel remaining, he set the lunar module down and declared that the Eagle had landed.
That why we have human astronauts. This is what sets “spam in a can” apart from a pilot. This is why airliners, while heavily automated, have highly-trained pilots at the controls.
Which brings us back to database administrators … I hope!
What sets a DBA apart from an ambitious programmer or a developer doing what needs to be done to move forward?
It is the ability to prepare for trouble and recover from it. Database recovery in the event of failure or mishap is the most vital skill in a DBA’s toolkit.
The Oracle RDBMS has been around now for about 30 years. The internal mechanisms for backup and recovery have changed very little in the past 20 years. Of course there have been enhancements, but the basic mechanism for basic “hot” or online backups has changed very little.
However, it is the mechanism for restore and recovery that took a great leap forward 10 years ago, when Oracle Recovery Manager (RMAN) was introduced with Oracle8. In a world where misnomers abound, Recovery Manager is quite aptly named. The focus of the product is not on automating backups, but rather on automating the steps of restore and recovery as much as possible.  Much of the early reluctance to adopt RMAN came about not from any failings in the product, but rather from disappointment that the product did not make the job of performing backups any easier.  Since backups are the operation that DBAs see most often, what RMAN does for recovery operations was not fully appreciated.
As I teach people how to use RMAN, I attempt to stress the mindset that RMAN is not just about performing backups. Rather, it is about “feeding” the RMAN “recovery catalog”.  Backups are not ends in themselves, but simply entries in the recovery catalog used by RMAN during restore and recovery operations. If a DBA considers it their duty to feed the recovery catalog with backup operations and other maintenance such as crosschecks, then we have someone who is truly preparing for the eventuality, not just the remote possibility, of restore and recovery. Someone who understands the tool, and is not just applying a different tool to bang in nails the same old way.
The knowledge and capability to recover a database from catastrophic failure is what separates a real DBA apart from someone who found the installer or who know how to do the clickety-clickety thing in Oracle Enterprise Manager. And not just once, by luck, but knowing how to use RMAN to its full advantage, to work around those confusing and misleading error messages, to verify backups and maintain and protect the recovery catalog(s) so as to virtually guarantee recoverability, each and every time.
It is this protective mindset, liberally seasoned with caution and pessimism, which separates DBAs from other technologists. Systems administrators and network administrators have much the same tendencies, but only databases administrators are made responsible for never losing data.  Systems and networks can be made redundant, and if they fail it is only a matter of bringing them back to service, but data loss is forever and is never forgiven.
Years ago, I worked with a very no-nonsense vice-president. She didn’t want to know the details of my job, and rightly so. She simply stated, very clearly, “Failures happen, but don’t EVER tell me that you could not recover my data”. Message received.
This book is written by seasoned professionals who have been using RMAN since its inception. They have recognized that RMAN can be confusing, and feel that everyone should not have to go through the same learning curve in order to arrive at the same conclusions. So, they have gathered together their best practices and tried-and-true procedures and compiled them into this wonderful book.
If you are an Oracle database administrator, this could very well be the most important book you read. Technology books are famous for becoming “shelf-ware”, pristine and unopened books adorning shelves everywhere. This book will be the exception – the book that is dog-eared and worn, the cover falling off and pages smudged, found more often opened face down on a desk than perched serenely on a shelf. The information within this book is the very essence of the job of the Oracle DBA, the most important facet of the job, and I am grateful to Sam, Arup, and Darl for sharing.