Any system needs to be tested. And it’s a simple fact that testing is better done by people independent of the system being tested. A different perspective can often highlight new areas of weakness, and there is no conflict of interest in managing a "pass." As a prospective customer, independent third-party audits such as SAS70 can short circuit many approval processes, as can a large customer list, but in some cases there is simply no substitute for looking under the covers directly. From the vendor perspective, far from being a purely negative "cost-of-sale," an enlightened vendor will welcome the prospect of third-party inspection. More, different eyes will often uncover details inadvertently overlooked by the vendor’s own staff, and the whole process will provide the vendor with a wealth of security expertise and enable the vendor to differentiate against their competition.
Elements of an Ethical Hack
An ethical hack is usually performed either by the security audit division of a F-100 company or financial institution, the general IT department of a smaller organization, or a third-party security specialist. A third party may charge $25,000 or more, depending on scope, and it may take two weeks to one month in elapsed time. This includes the audit itself and the review process where various matters are clarified between the auditor and the vendor before the report is released to the client. Internally sourced audits usually deploy a team of two to three personnel over a few days.
Because in commercial SaaS environments, the production system is in use 24x7, the ethical hack is always performed against a parallel system. By using a parallel system you obviate the possibility of any actual customer data being compromised by the audit team, or expose customers to potential performance deterioration during DoS (Denial of Service) testing.
The hack itself is usually limited to electronic attack. Other forms of attack, which might in practice be used by hackers, are typically off-limits as they might even constitute criminal activity. Examples of this include attempts at physical break-ins or other illegal entry, or gaining access through social engineering such as misrepresentation or coercion of vendor staff. While not covered by the ethical hack, the customer should nevertheless satisfy themselves that the vendor has in fact competently protected themselves against such possibilities. This might include visiting the vendor data center to review physical security and interviewing vendor support staff to gauge their compliance to process. Such activities would usually, but not always, be considered additional to the ethical hack process.
Systems Security Configuration Management
Intrusion Detection
Network Security Port Scanning
Denial of Service
Systems Management Review
Application Security Access Control
SQL Injection
Cross Site Scripting
Code Review
Table 1: Elements of an Ethical Hack
The Customer Perspective
On the Web, even small vendors can look impressive. But looks can deceive, and as in any environment, there will always a spectrum of vendor competence. In many cases, there is no substitute for direct testing because what goes on behind the scenes is typically largely hidden from scrutiny in a SaaS business.
Some systems may have fairly low security requirements given the potential impact of security breaches. By contrast, a board portal requires the most stringent security measures because it holds such sensitive information. M&A discussions could be put into jeopardy by inadvertent disclosure. Documents about restructuring could have a profound impact on labor relations. Email taken of out context could result in litigation.
Of course, when it comes to security, one size does not fit all and it is also not true to say that more security is always better. With more security comes trade-offs, typically related to ease of use and systems management. And in most cases some judgment will be required because security must ultimately be proportionate to the task at hand, taking into account data sensitivity, business and organizational impact but also the customer’s attitude, who will likely have varying perceptions and appetites for risk.
The Vendor Perspective
Smart management teams will always look for sources of competitive differentiation. One way that is done is by providing products and services that are "difficult." It results in a barrier to entry. Security is certainly in that class of "difficult" solutions, where every day, even the largest and most established companies can slip up. Effective security is much easier to claim than to demonstrate, and unfortunately there is no guarantee that all those good practices are actually observed by the vendor.
A third-party ethical hack, while carrying with it risks of failure and expense, can be an effective way to make the vendor’s security expertise more visible. It can act as a barrier to competing solutions that "talk the talk" but cannot "walk the walk."
Finally, and perhaps most importantly, if a vendor is serious about security, ethical hacks are a wonderful source of customer feedback. Third parties, immune from internal politics, can make observations that might be difficult for internal QA departments and security teams. By subjecting oneself to a plethora of different tests by different teams, the vendor dramatically increases the coverage of possible exposures.
Common Issues
Ethical hacks typically cost around $25,000. An internally resourced ethical hack such as those implemented by the security teams found within large financial organizations, while not separately budgeted, will require equivalent resource. Clearly, this cost needs to be considered within the context of the overall solution, and the relative importance of your security requirements. Also, few companies would choose to conduct ethical hacks against three competing vendors to select the best. Instead, the ethical hack becomes a final hurdle for the chosen vendor.
Other forms of validation are also possible. If the vendor has a long list of financial customers (typically amongst the most security-conscious), then this may be taken as a proxy for a certain level of competence. Likewise, existing third-party audits, if available, as well as SAS70 Type II compliance can cover many important aspects.
There are some issues to consider for a vendor, most clearly in terms of the amount of resources needed to support an ethical hack. Also, especially with new functionality and new markets (which may utilize existing functionality in new ways and also have different security requirements), it is likely that faults will be found. No vendor wants a press release announcing "X fails on Y," so confidentiality is important in order to build trust between the vendor and the auditor and ensure complete disclosure.
A risk for the vendor arises when a genuine disagreement arises regarding a particular feature of process. Given that security is often about balancing a large number of corner cases, each of unknown but small probability, against the known and in some cases large impact on usability, a black and white representation of right and wrong will not help the vendor or the customer.
Summary
While an expensive investment for both customers and vendors, ethical hacks remain the best and most effective way to verify the security of the SaaS vendor. In addition ethical hacks remain one of the best ways for a vendor to maintain high security standards, and seen from a competitive perspective, the substantial costs involved can act as a powerful differentiator.