Is Black Box Testing dead?


When I was reading through the OWASP ASVS 4.0 PDF a few weeks ago, I was stopped in my tracks as early as page 10. Specifically by the following quotes : "Black box testing is not effective assurance and must stop." "Over the last 30+ years, black box testing has proven over and over again
to miss critical security issues that led directly to ever more massive breaches."

Now, let me preface this with the clarification that this post is not an attack on the effort put into the ASVS project. The output by the volunteers behind the project, a few of which are personal friends, is useful and provides value to a wider audience. However, over the last few years the advocacy against "black box testing" has increased to the point where either I am completely out of sync with what is best practice in terms of penetration testing or we're seeing another case of erosion of a definition. This post is me pondering about the misconceptions and how we can do better.

Penetration Testing. Definitions and somesuch

By and large, in penetration testing, we usually see 3 types of offerings:

  1. Black box testing : the testers do not have any information about the target(s) besides an entry point (usually a URL, an IP address, or a binary).
  2. White box testing : the testers have a lot of information about the target(s), including documentation, access to developers and source code, and credentials.
  3. Grey box testing : Something in between. Testers have access to some information but do not operate in full knowledge.

These are good descriptions of what those different services are. With a black box test, it is pretty difficult to know what you will encounter. Is it a single login page? Are there some public APIs you'll be able to tamper with? Will it even be possible to make a dent from the outside?

There is no doubt that the challenge of a black box test is much greater. There is also no guarantee that the tester(s) will find anything of significance in the target(s). But is this the ultimate goal of this type of test? To me, it isn't.

I think we can all agree on the fact that any application will break given enough attention from a motivated adversary (simulated or not). Black box testing is not meant to bring to light any and all weaknesses in a target. It is meant to verify whether a target can sustain X time of pressure from a motivated and skilled adversary.

The reality of black box testing

For some reason the continuum between black box testing and white box testing has become synonymous with cheap vs. pricey. Where white box testing comes at a significant cost, black box testing is the cheap alternative offered when there is no budget, no time, or no interest. Today, when you buy a black box test you are likely to receive a run-off-the-mill report, based on a bunch of automated tools by one or more junior penetration testers. It is kind of funny.

Personally, I don't think we have already lost the meaning of black box testing but we've definitely failed to sell it. Black box testing is not supposed to be your cheapest offering. To the contrary, it requires the most resources and is the most labor-intensive. Allow me to illustrate with a few examples.

The Set Top Box

A few years ago, I was engaged to test the security of a set top digital video box. I received the box with no documentation, no credentials, and no instructions beyond "Go at it. You have 10 days".

I was lucky that they sent me 2 of them, because at least one would be rendered unusable by the end of the engagement :)

There is nothing more fun than jumping into the rabbit hole that is hardware analysis. The reality is that it requires a complex set of skills that is rarely found in one person (definitely not myself). So, to complete this project it was necessary to tear the device down carefully and document potential avenues of compromise. From there, I had to pull in the right people for certain sub-tasks and at certain points bounce ideas of others based on what I found. There was no way that a single person, for a set amount of time, would be even close to successful on this type of project but thanks to the collaboration of multiple skilled individuals, it didn't take us more than 5 days to compromise the target.

A useful reminder might be that our goal is not to make the targets we test unhackable. We aim to make it more expensive for an adversary to gain control. After a few iterations both the customer and us were confident that the primary adversaries in their threat model would need considerable resources to compromise the system. Goal: achieved.

The Android App

This example came in through an investment company. They were in advanced talks to acquire an interest in a startup company. That meant that they had 10 days exclusivity to engage with the startup on the negotiated terms. Time, of course, was of the essence. The investor was interested in a few things:

  1. Does this application do what it says it does?
  2. Is this application secure? (for some values of secure)
  3. Is this application maintainable?

Very similar to the scenario described above, we were given 5 days to explore the application with zero information and no credentials. It took us a little over 2 days to answer all three questions with the result being that the investor withdrew their investment.

Both of these engagements were black box tests by definition, however they weren't sold cheap. The reality of black box testing is that you can't tell beforehand what the resources are that you'll need to achieve the goal. You'll have to play it by ear as the scenario unfolds itself. Do you need a malware writer? A firmware reverse engineer with experience on a certain platform? Additional hardware or tools? A cryptographer? It is all open and your budget needs to be able to cover it without a change order. If that's not what you're doing, you're not black box testing. You're testing for compliance or aiming for the lowest bar, without any consideration of the adversary landscape. Why we call the latter "black box testing" is something I haven't been able to understand yet.

There is a lot to say about developing, offering, and delivering testing services but one thing I would say to buyers is that if black box testing is the cheapest item on the menu, it is unlikely that it is black box testing as it is intended to be. It should, by all means, be the most expensive item on the menu. Always.


Black box testing is not without value, nor is it cheap. I disagree vehemently with the position, as taken by the ASVS and some providers in the industry, that "black box testing must stop". I believe that black box testing must be understood, and clearly communicated. Attacking an undocumented target requires a lot of resources and it should always happen in the context of the threat model of the owner. I'm sincerely hoping that we don't discard the practice as worthless, because to me the true black box tests are few and far between and - at the same time - provide significant value to clients.

Share this post: