Image courtesy of Shutterstock.
Let's rephrase the question above: The next ticking time bomb is your database. Regardless of whether you run Oracle, Microsoft SQL Server, or MySQL, the odds lean toward your organization having a ticking time bomb. The only question to be posed is whether you are going to become collateral damage or diffuse the bomb before the damage happens.
When we look at the past few weeks of headline news, we see a lot of companies that have chosen to become collateral damage. A short list of recent headlines should suffice:
CNN: United flights resume after computer problem
USA TODAY: NYSE blames software upgrade for outage
The Wall Street Journal: OPM Director Katherine Archuleta Resigns After Massive Personnel Data Breach
CNBC: TD Ameritrade resolves order routing problem
PC Magazine: AWS Glitch Hits Netflix, Pinterest
Many of these organizations settled for being collateral damage. But it should be unambiguously clear that it was a choice they made by their actions and their inactions.
What cost did United Airlines incur after being shutdown for a few hours? How do you think customers feel about flying United Airlines now? Katherine Archuleta the embattled director of the Federal Office of Personal Management lost her job over the recent security breach. How difficult will it be for Katherine to find a new job after such an incident? How many of these businesses could have avoided these problems had they even looked for the ticking time bomb within?
It is standard process for financial concerns to complete an audit of their financial practices. But how many of these same organizations have an external, regular review of the health of their databases? The same databases hold key customer information and business transactions critical to the business. Without this data these companies would cease to exist.
Ntirety published a report titled “Disruptive Epidemic Report” on the state of Microsoft SQL Server databases. Some of the key findings:
- 90% of SQL Server instances failed the disaster recovery review
- 88% of SQL Server Instances failed configuration review
- 40% of SQL Server Instances failed security review
- 39% of SQL Server Instances failed the database backup review
When you couple these findings with the fact that the amount of data now collected is doubling yearly, a dire picture emerges. The technology needed to support these databases continues to grow and becomes more complex. This picture highlights a business environment where management is constantly asking IT personnel to do more with fewer resources. The underlying image revealing itself is that of a perfect storm of catastrophic events; it’s just a matter of when the ticking time bomb explodes.
Will your organization be collateral damage, or will you have taken steps to identify the ticking time bomb and diffuse it?
Indicators You Are Collateral Damage Waiting to Happen
There are a number of early indicators that increase the likelihood that you have a data vulnerability problem within your organization. Review these indicators and do an honest evaluation of your organization.
Understaffed DBA Team
Is your team of database administrators understaffed? Are they continuously working long hours? Is the majority of the team’s effort spent putting out fires? Do you constantly have performance problems? Let’s face it, if the answer is yes to these questions then your staff is cutting corners to get the job done.
Unsupported Versions of the Database
Running an outdated version of the database is a security risk waiting to happen. Today, I know of number of companies running production systems on Oracle version 8 (circa 1997) and yet the latest release is Oracle12c. I know of a number of companies still running SQL Server 2000 (circa 2000), yet the most current release is SQL Server 2014. Today, you can actually get a Microsoft Technology preview of SQL Server 2016.
The most common reason for running an outdated version of the database is that an application you are running requires a specific older version. Running an application that requires such an old version of the database is itself a ticking time bomb all. If you are running unsupported versions of the database, you must purchase extended support. Even if you have done this, the vendor is no longer actively providing updates to the database.
Patching Cycles
Is your staff constantly behind on database patching cycles? Patching databases is critical to preventing database corruption and security breaches. If your organization is not up-to-date on patching, you are collateral damage waiting to happen.
Formal Monitoring & Testing Processes
A database management system is built with some of the most complex software ever written. It is essential that formal monitoring solutions and testing procedures are put in place. This includes full change management procedures. Does your organization have a formal monitoring solution in place? Do you have a formal change management process in place? Do you properly test applications changes before introducing them into production?
Unrealistic Expectations
A few years ago I was working with a healthcare company that monitored critical functions at the bedside. To address a server performance problem, we needed to bounce the database but the client could never give us a 10-minute window to bounce the database. Looking under the covers of the infrastructure, they were running on a single server with no high availably options being used. Management had expectations that the database would never come down, yet they had a physical infrastructure that was unable to meet those requirements. Does this sound like your organization? Why was the company not virtualizing the infrastructure this database was running on? Why didn’t it use a clustered version of this database?
Backup Strategies Deployed
- What is your organizations backup strategy?
- Has it been documented? Is it formally tested?
- Does the backup strategy meet your organizations recovery point objectives?
- Does the backup strategy meet your organizations recovery time objectives?
- Do you have procedure in place that alerts when a database is not being backed up?
- Does every database in your organization allow for a point in time recovery?
- Do your critical systems utilize high availability features of the database you are running?
- Are you virtualizing databases?
- Do you have a geographically dispersed backup of your databases? Anyone who survived hurricane Sandy or Katrina understands the importance of a geographical dispersed backup. Today, it is affordable and easy to leverage the cloud for disaster recovery.
If you don't have good answers to these questions, they are all indicators that you are at risk.
Third-Party Validation
Does your organization use vendors that undergo third-party validations? Good examples of third-party validation include HIPAA compliance, PCI, SOX, and MSPAlliance Cloud/Verify. Organizations that have these certifications must meet rigorous standards and undergo third-party validation they meet these standards. Do your in-house databases ever undergo a third-party reviews?
The old adage “trust but verify” permeates the list. Good business practice requires verification from time to time that the processes and procedures put in place are still working as designed.
Diffusing The Ticking Time Bomb
These past few weeks have been a challenge for IT to say the least. The impact of the outage to United Airlines will mar the travel industry for a long time to come. The New York Stock Exchange could not conduct business for close to 4 hours; make no mistake, people’s jobs within these organization are at risk. Ask Katherine Archuleta who recently resigned from the U.S. Office of Personal Management.
There are a number of steps that can be taken to identify these hidden time bombs and more importantly diffuse them before your business can be affected. You can choose to take preemptive action or you can choose to become collateral damage.
Understaffed DBA Team and Proactive Monitoring
Every business needs to do more with less, and that is a given in today’s world. Sometimes, doing more with less means doing things differently. Today, you have options that did not exist 20 years ago.
Instead of fighting for additional headcount, look at augmenting your team with managed services. For a faction of the money you would have spent for an additional person, you can get access to a team of resources, proactive monitoring solutions, and possibly prevent in-house DBA burnout all in one.
Third-Party Audits & Health Checks
Bringing in qualified vendors to come in to look under the covers and validate your business is following industry best practices. The Disruptive Epidemic Report findings reveal that there is a 39% chance that a database will fail a backup review. Let’s find this internal time bomb and correct it. You don't want your company to be in the next headlines we read. You don't want to be the CIO where the initials mean Career Is Over. Your company will be stronger as a results of a third-party audit of the databases.
Hold All Vendors to High Standards
In the world of the cloud you don't have the luxury of meeting every vendor you do business with. How do you know they are who they say they are? There are a number of third-party accreditations out there to help you with this problem. Look for accreditations that have high standards and require a third-party validation. PCI and MSP Alliance are good examples, where there are clear rigorous standards that must be met and require a detailed third-party validation.
Leverage Technology
Make sure you are leveraging the right technologies in the right places. Are your mission critical systems utilizing the appropriate technologies to meet the needs of the business? What high availability technologies are you utilizing? What virtualization technologies are you leveraging? How do you leveraging the Cloud? The Cloud is a cost effective way to have a geographically dispersed disaster recovery strategy. Don't let the next Hurricane Sandy take you out of business. Do you offer your stakeholder Database as a Service? Working with the right Cloud providers these capabilities are available today.
Fire Drill Your Database Backups.
Every year at Christmas I change all the batteries in my smoke alarms and test them. Until you have tested a database backup, don't be so sure it will work. Have you tested your backups?
Auditing Of Changes & Change Control
Its important to have a proper change control process in place at your organization. Without change control in place someone will put code into production and you will quickly find yourself dealing with the fallout. If it can be logged, then log it. Make sure you take the time to audit all changes to critical portions of your infrastructure.
Database Upgrades, Patching & Security
Keeping your databases current and on the most current patching levels is the best way to keep an environment secure and stable. Running fully supported version of the databases means the vendor is putting out patches to close security vulnerabilities. It important that you have an active patching schedules to keep up with these releases. If it can be encrypted, encrypt it. Just ask the office of Federal Office of Personnel Management the importantance of encrypting data.
The next ticking time bomb is your database. It doesn’t matter if that database is Oracle, Microsoft SQL Server, or MySQL. The odds favor your organization having a ticking time bomb.
So, back to the question: Are you going to be collateral damage or are you going to diffuse the bomb before the damage is done? The choice is yours.
-----
Michael Corey is the president of Ntirety, a division of HOSTING; a Microsoft SQL Server MVP, VMware vExpert and a past president of the Independent Oracle Users Group.
Don Sullivan has been with VMware since 2010 and is the product line marketing manager for Business Critical Applications.