EvDBT

After working for a variety of companies in the 1980s, after working for Oracle in the 1990s, after trying (and failing) to build a company with friends at the turn of the century, and after more than a decade working as an independent consultant in this new century, I found myself in a professional dilemma last year.

I know I need to work at least another ten years, probably more like fifteen years, to be able to retire.  I had survived the nastiest economic downturn since the Great Depression, the Great Recession of 2008-2011, while self-employed, and felt ready to take on the economic upswing, so I was confident that I could work steadily as an independent Oracle performance tuning consultant for the next 15 years or more.

Problem was: I was getting bored.

I loved my work.  I enjoy the “sleuthiness” and the forensic challenge of finding performance problems, testing and recommending solutions, and finding a way to describe it clearly so that my customer can make the right decision.  I am confident that I can identify and address any database performance problem facing an application built on Oracle database, and I’ve dozens of successful consulting engagements to bear witness.  I have a legion of happy customers and a seemingly endless supply of referrals.

Being confident is a great feeling, and I had spent the past several years just enjoying that feeling of confidence on each engagement, relishing the challenge, the chase, and the conclusion.

But it was becoming routine.  The best explanation I have is that I felt like a hammer, and I addressed every problem as if it was some form of nail.  I could feel my mental acuity ossifying.

Then, opportunity knocked, in an unexpected form from an unexpected direction.

I have a friend and colleague whom I’ve known for almost 20 years, named Kyle Hailey.  Kyle is one of those notably brilliant people, the kind of person to whom you pay attention immediately, whether you meet online or in person.  We had both worked at Oracle in the 1990s, and I had stayed in touch with him over the years since.

About four years ago, I became aware that he was involved in a new venture, a startup company called Delphix.  I wasn’t sure what it was about, but I paid attention because Kyle was involved.  Then, about 3 years ago, I was working as a DBA for a Colorado-based company who had decided to evaluate Delphix’s product.

Representing my customer, my job was to prevent a disaster from taking place.  I was to determine if the product had any merit, if the problems being experienced were insurmountable, and if so let my customer know so they could kill the project.

Actually, what my boss said was, “Get rid of them.  They’re a pain in my ass“.  I was just trying to be nice in the previous paragraph.

So OK.  I was supposed to give Delphix the bum’s rush, in a valid techie sort of way.  So I called the Delphix person handling the problem, and on the other end of the phone was Kyle.  Hmmm, I can give a stranger the bum’s rush, but not a friend.  Particularly, not someone whom I respect like a god.  So, instead I started working with him to resolve the issue.  And resolve it we did.  As a result, the company bought Delphix, and the rest is history.

Here’s how we did it…

But first, what does Delphix do?  The product itself is a storage appliance on a virtual machine in the data center.  It’s all software.  It uses sophisticated compression, deduplication, and copy-on-write technical to clone production databases for non-production usage such as development and testing.  It does this by importing a copy of the production database into the application, compressing that base copy down to 25-30% of it’s original size.  Then, it provides “virtual databases” from that base copy, each virtual database consuming almost no space at first, since almost everything is read from the base copy.  As changes are made to each virtual database, copy-on-write technology stores only those changes for only that virtual database.  So, each virtual database is presented as a full image of the source database, but costs practically nothing.  Even though the Oracle database instances reside on separate servers, the virtual database actually resides on the Delphix engine appliance and is presented via NFS.

I was asked to understand why the virtual databases were slow.

On the other end of the phone was Kyle, and he was easily able to show me with repeatable tests where and what the nature of the performance problems were, and that they were predictable and resolvable.

But I’m not really writing just about Delphix, even though it is very cool and quite earthshaking.  Rather, I’m writing about something bigger that has stormed into our industry, bringing to fruition something that I had tried — and failed — to accomplish at the turn of the century.  Back then, at the turn of the century, when some colleagues and I tried to build a hosted-application services company, we failed in two ways:  1) we were ahead of our time and 2) we chose the wrong customers.

Being ahead of one’s time is not a failure, strictly speaking.  It shows a clarity of vision, but bad timing.

However, choosing the wrong customers is absolutely a failure.

Before I explain, please be aware that I’m going to change most of the names, to protect any innocent bystanders…

After leaving Oracle in July 1998, I founded my own little company-of-one, called Evergreen Database Technologies (a.k.a. EvDBT).  The dot-com boom was booming, and I wanted to operate as an independent consultant.  I had been living in Evergreen in the foothills west of Denver for years, so the choice of name for my company was by no means a brilliant leap of imagination; I just named it after my hometown.  Business was brisk, even a bit overheated as Y2K approached, and I was busy.  And very happy.

In early 2000, I had been working with another young company called Upstart, and we felt that the information technology (IT) industry was heading in one inescapable direction:  hosted services.  So I joined Upstart and we decided that providing hosted and managed Oracle E-Business Suites (EBS) was a good business.  EBS is the world’s 2nd most prevalent Enterprise Resource Planning (ERP) system and can be dizzyingly complex to deploy.  It is deployed in hundreds of different industries and is infinitely customizable, so in order to avoid being eaten alive by customization requests by our potential customers, we at Upstart would have to focus on a specific industry and pre-customize EBS according to the best practices for that industry.  We chose the telecommunications industry, because it was an even bigger industry in Colorado then, as it is now.  We wanted to focus on small telecommunications companies, being small ourselves.  At that time, small telecommunications companies were plentiful because of governmental deregulation in the industry.  These companies offered DSL internet and phone services, and in June 2000 our market research told us it was a US$9 billion industry segment and growing.

Unfortunately, all of the big primary telecommunications carriers bought the same market research and were just as quick to catch on, and they undercut the DSL providers, provided cheaper and better service, and put the myriad of small startups out of business practically overnight.  “Overnight” is not a big exaggeration.

By October 2000, we at Upstart were stunned to discover that our customer base had literally ceased answering their phones, which is chillingly ominous when you consider they were telecommunication companies.   Of course, the carnage directly impacted the companies who had hoped to make money from those companies, such as Upstart.  Only 4 months in business ourselves, our target market had vanished like smoke.  You might say we were…. a little disconcerted.

We at Upstart had a difference of opinion on how to proceed, with some of the principals arguing to continue sucking down venture-capital funding and stay the course, while others (including me) argued that we had to find some way, any way, to resume generating our own revenue on which to survive.  I advocated returning to Upstart’s previous business model of consulting services, but the principals who wanted to stay the course with the managed services model couldn’t be budged.  By December 2000, Upstart was bankrupt, and the managed-services principals ransacked the bank accounts and took home golden parachutes for themselves.  I jumped back into my own personal lifeboat, my little consulting services company-of-one, Evergreen Database Technologies.  And continued doing what I knew best.

This was a scenario that I’m told was repeated in many companies that flamed out during the “dot-bomb” era.  There are a million stories in the big city, and mine is one of them.

But let’s just suppose that Upstart had chosen a different customer base, one that didn’t disappear completely within months?  Would it still have survived?

There is a good chance that we would still have failed due to being ahead of our times and also ahead of the means to succeed.  Hosted and managed applications, which today are called “cloud services”, were made more difficult back then by the fact that software was (and is) designed with the intention of occupying entire servers.  The E-Business Suites documentation from Oracle assumed so, and support at Oracle assumed the same.  This meant that we, Upstart, had to provision several entire server machines for each customer, which is precisely what they were doing for themselves.  There was little reason we could do it cheaper.  As a result, we could not operate much more cheaply than our customers had, resulting in very thin cost savings or efficiencies in that area, leaving us with ridiculously small profit margins.

Successful new businesses are not founded on incremental improvement.  They must be founded on massive change.

What we needed at that time was server virtualization, which came along a few years later in the form of companies like VMware.  Not until server virtualization permitted us to run enterprise software on virtual machines, which could be stacked en masse on physical server machines, could we have hoped to operated in a manner efficient enough to save costs and make money.

Fast forward to today.

Today, server virtualization is everywhere.  Server virtualization is deeply embedded in every data center.  You can create virtual machines on your laptop, a stack of blade servers, or a mainframe, emulating almost any operating system that has ever existed, and creating them in such a way that finally makes full use of all the resources of real, physical servers.  No longer would system administrators rack, wire, and network physical servers for individual applications using customized specifications for each server.  Instead, virtual machines could be configured according to the customized specifications, and those virtual machines run by the dozens on physical machines.

The advent of virtual machines also brought about the operations paradise of abstracting computer servers completely into software, so that they could be built, configured, operated, and destroyed entirely like the software constructs they were.  No more racking and wiring, one server per application.  Now, banks of “blades” were racked and wired generically, and virtual machines balanced within and across blades, with busier virtual machines moving toward available CPU, memory, and networking resources and quieter virtual machines yielding CPU, memory, and networking to others.  Best of all, all of this virtualization converted hardware into software, and could be programmed and controlled like software.

Everything is virtualized, and all was good.

Except storage.

Think about it.  It is easy and cheap to provision another virtual machine, using fractions of CPU cores and RAM.  But each of those virtual machines needs a full image of operating system, application software, and database.  While server virtualization permitted data centers to use physical servers more efficiently, it caused a positive supernova explosion in storage.  So much so that analysts like Gartner have predicted a “data crisis” before the end of the decade.

This is where Delphix comes in.

By virtualizing data as well as servers, it is now truly fast, cheap, and easy to provision entire virtual environments.  Delphix works with Oracle, and it also works with SQL Server, PostgreSQL, MySQL, DB2, and Sybase.  Even more importantly, it also virtualizes file-systems, so that application software as well as databases can be virtualized.

So back in early 2014, Kyle contacted me and asked if I would be interested in joining Delphix.  My first reaction was the one I always had, which is “no thanks, I’ve already got a job“.  I mean, I’m a successful and flourishing independent consultant.  Why would I consider working within any company anymore.  Business was brisk, I never had downtime, and the economy was improving.  I had successfully been operating as an independent consultant for most of the past 15 years.  Why fix what wasn’t broken?

But here was the crux of the matter…

I wanted to try something new, to expand beyond what I had been doing for the past 25 years since I first joined Oracle.  If it was a large established company beckoning, I wouldn’t have considered it for a second.  But a promising startup company, with a great idea and a four-year track record of success already, and still pre-IPO as well?

I couldn’t resist the gamble.  What’s not to like?

It’s possible I’ve made an enormous mistake, but I don’t think so.

Not to turn excessively morbid, but all of us are just a heartbeat away from our common destination.  I believe that, when I’m at the last moments of my life, the thing I fear will not be death itself, or pain, or leaving life behind.

It is regret that I fear.

And regret can take many forms, but the most painful regret will undoubtedly be what might have been, the opportunities passed or missed.  Career is only one aspect of life, and I don’t want to give it too much significance.  I’ve accumulated regrets in how I’ve lived my life, and how I’ve treated people in my life, and in some cases they are small but there are some which will always haunt me.

But with regards to my professional career, as Robert Frost said, I’ve taken the road less traveled, and that has made all the difference.  No regrets here.