In the last several articles, I’ve been describing the benefits of reading and analyzing the benchmarking case studies vetted and released by the Transaction Processing Council (www.tpc.org). I’ve given you a broad overview of the TPC benchmarks shown ways that the vendor-published TPC benchmarks can help you save money and how the vendor-published TPC benchmarks must explain in disclaimers how they tweak their workloads in unrealistic ways.
I have described how to run your own benchmarks and explained how to properly prepare your environment for a benchmark test. Now, it is time to show you where the rubber really hits the road, testing and benchmarking tools that can run highly scalable benchmarking workloads against your database servers.
HammerDB
My current favorite tool for performing TPC-like benchmarks is HammerDB, formerly known as HammerOra. HammerDB is a free open-source load testing tool for Oracle (including TimesTen), Microsoft SQL Server, PostgreSQL (including Plus Advanced Server), Greenplum, MySQL and Redis database platforms. It supports the TPC-C and TPC-H workloads, so that you can simulate heavy transaction processing and business intelligence workloads, respectively.
Figure 1: HammerDB testing a TPC-C workload on Microsoft SQL Server
HammerDB is distributed via SourceForge (http://hammerora.sourceforge.net/document.html) and is usually updated several times per year by the all-volunteer development team. It is easy to learn and can drive very heavy workloads against your database servers with minimal configuration or setup. On the other hand, it is also easy to configure and modify. So if you want to drive a heavy workload that is entirely of your own creation, it’s not terribly hard to do assuming you can write SQL code. HammerDB features great documentation and an active discussion forum where the developers respond quickly to questions and are very helpful.
SQL Server 2012 Distributed Replay
Starting in SQL Server 2012, Microsoft added in the Distributed Replay feature set designed to run a multi-threaded workload against SQL Server database. Distributed Replay was initially intended to allow users to assess the impact of upgrading the databases and applications from one version of SQL Server to another. However, it is actually able to offer a fully scalable solution to effectively simulate mission-critical SQL Server workloads, conduct performance testing, and undertake what-if tests and capacity planning.
Distributed Replay is composed of four main components. The administration tool and workload controller make up the first two components, while the client(s) that generate the workload and push it to the target server make up the last two components. Since Distributed Replay uses trace files for its workload, you’ll also need to know how to create or capture SQL Server traces.
More information about the Distributed Replay utility is available at http://msdn.microsoft.com/en-us/library/ff878183.aspx. While it is free for owners of SQL Server, I still prefer HammerDB for straight TCP-C and TPC-H workloads.
Benchmark Factory from Dell Software
(Full disclosure: I am a former employee of Dell Software [formerly Quest Software] and, frankly, always liked this product.)
As a paid commercial product, you should expect more polish, features, and capabilities that what’s available in the public domain and Benchmark Factory (BMF) delivers that in spades. It provides a clean, though somewhat unintuitive, interface to run a wide variety of industry-standard and custom-made benchmark workloads against most of the major commercial database platforms. The workloads it supports include AS3AP, TPC-C TPC-B, TPC-D, and Scalable Hardware benchmarks. And the database platforms it supports include Oracle, Microsoft SQL Server, DB2 UDB, SAP Sybase ASE, MySQL, and any other databases with ODBC connectivity.
Among the tools I’ve mentioned, it also includes the best built-in analysis and assessment features of the bunch. That means that you don’t have to use separate monitoring tools or collection methods to determine the full impact of the workload and benchmark test upon the database you’re testing. The full product is available at www.quest.com/benchmark-factory/, while a low-end freeware version is available at http://benchmarkfactory.inside.quest.com/index.jspa.
Follow Kevin Kline on Twitter and Google.