Two columns ago, I described how the TPC benchmarks are useful for getting a general idea of the performance characteristics of your preferred database vendor and hardware platform. And in last month’s column, I described how the published TPC benchmarks can even help with pricing, especially when you don’t have your own quantity discounts in place.
In this column, I want to tell you about one of my favorite aspects of the TPC benchmarks – CHEATING. Keep in mind that I use the term “cheating” in a joking manner with my tongue planted firmly in my cheek. But I’m also half-serious. One of the of the things that is great about the TPC benchmarks is that each of the vendors are required to fully describe all of the shortcuts, tweaks, and special operating configurations they use in order to achieve their spectacular performance numbers. In a sense, the Transaction Processing Counsel requires that all benchmarked platforms declare all of the ways that they cheat in order to achieve peak performance.
Keep in mind that I use the term “cheating” in a joking manner with my tongue planted firmly in my cheek.
To use an analogy, race cars are so heavily customized to that you could never take one out on the street as a daily commuter car and they’d be so uncomfortable, no one would want to even if they could. It’s more or less the same way with the database configuration used for TPC benchmarking. Here’s a quick example. If you’re running a TPC-C benchmark and you have a failure of some sort and need to perform recovery, you’d totally ruin your performance numbers. So rather than recover, you’d be much better off rerunning the entire benchmark. Consequently, since we know we’ll never do recovery during a benchmark, why don’t we disable recoverability features to wring out an extra 1% or 0.5% of performance? Naturally, we’d never do that in the real world. So that’s why I call it cheating.
On the other hand, it’s a good sort of cheating. Why? First of all, it’s an even playing field. Everyone does it. And since everyone does it equally, the TCP benchmark results are equally valuable from a comparative standpoint. Secondly, the Transaction Processing Counsel requires any benchmark to go through a rigorous audit in which the entirety of vendor tweaks are fully disclosed, also known as the Full Disclosure Report. This is actually great news because we can learn from these special tweaks, many of which are minimally documented by the vendor.
Follow me on Twitter and Google.