Penetration Test != Audit != Assessment

If someone is selling you a network penetration test, and then running a vulnerability scanner and handing you a report, you’re not getting what you paid for, period.

Back around the turn of the century when I founded a small security consulting firm in North Texas, we had to explain the difference between these three types of engagements to most new customers and watch the color in their faces drain as they realized they hadn’t been getting what they were paying for from other firms.  Many times they just didn’t know how to ask for what they actually needed, so our first step in business development became customer education.  The differences between a network penetration test, an audit, and an assessment, can be summed up as follows:

A Penetration Test is generally exactly that. The target is attacked as a real attacker would; the approach is more stealthy, profiling is done to identify weak targets, those targets are enumerated and surgically attacked until one or more are successfully compromised, then the test is over. Penetration tests are generally limited in scope and comprehensiveness by nature, and on many occasions produce new, never-before known vulnerabilities in the systems and software the customer employs. Effort is focused on compromising the target in any way possible.  Penetration tests are generally the most expensive of the three types of tests I’m outlining because they by far require the most skill.  The differentiators here are the VARIABLE SCOPE and LENGTH of the test via selective targeting and the end of the test as soon as the goal is reached; successful compromise.

An Audit is generally a test for some form of compliance. The scope is defined exactly by whatever documentation outlines the requirements for compliance. This can be anything from multiple security compliance documents the size of HIPAA and SOX  to a simple checklist of specific vulnerabilities. If all the requirements are met, the target is compliant and has passed the audit. These are usually the next most expensive tests of the three; while they don’t require as skilled personnel as Penetration Tests, they’re more often than not extremely tedious.  The differentiation here is that you have requirements to audit the target AGAINST.

An Assessment is what you get from stock tools like vulnerability scanners, custom tools to identify vulnerabilities, and nice pretty reporting software to tie all the results together and provide references, mitigation and remediation guidance, executive summaries, a network health graph, threat metrics, and any number of other bells and whistles that can be compiled from the breadth of available information about the vulnerabilities found. The important differentiation here is that Assessments are essentially a test of the target for KNOWN vulnerabilities, hence the amount of available information to compile a pre-generated report from.  Any monkey can run packaged tools and generate reports from canned data, which results in Assessments generally being the least expensive of the three tests.

My firm was happy to provide all three (yes, we had various Nessus-flinging primates on call), given that the customer actually understood what they would be getting when they paid for one or the other.

An earlier version of this article originally appeared as a comment to a post over on Ari Takanen’s VoIP Security blog entitled “VoIP security auditing is becoming more and more complex … Not!”.  After posting the comment, I realized I had typed for quite a while and the content would actually make a decent blog post over here with the addition of a little more context information.

4 Responses to “Penetration Test != Audit != Assessment”

  1. Joany Boutet Says:

    Hello Mr Trammell,

    I am a student trying to test rtpinsertsound tool but that doesn’t work for me !

    I have two Cisco 7960 SIP phone with the last firmware, x and y, running on an asterisk server.

    On backtrack, plugged on PC port on x phone, I run wireshark to know ports used during RTP packets exchange between x and y:

    RTP port for x: 30136 IP:
    RTP port for y: 21150 IP:

    On backtrack i have a view on all RTP packets during the call, so I run ./rtpinsersound -i eth0 -a -A 21150 -b -B 30136 test.wav -f 1 -j 80

    Normally I have to heard the wav file on x; but that doesn’t work !

    I have already tried with a pcap file qnd VoIP Sound Board but that doesn’t work too !

    Is there codec error ?

    Thanks in advance

  2. Ari Takanen Says:

    The point I was trying to make in my original post at ITworld was that all these services do not really solve any issues people have in VoIP assessments. A proactive (call-it-what-you-like) security service should not:

    a) look for known issues only
    b) look for just single vulnerability (and exploit it)
    c) go through a check list to pass a compliance against whatever

    A person buying these types of services expect that you will use all possible means to find all possible vulnerabilities, and help them close those issues. Services that are built around tools (such as vulnerability scanners) or around certification requirements (whether they are PCI, SOX or whatever useless stuff around there) do nothing to resolve the real issues people have with VoIP security. A hacking assignment also (pentest?) also only works as a demo to use for building a more significant budget for a real security assessment (or whatever it should be called).

    Maybe it calls for a new name for such services. I dunno.

  3. Dustin D. Trammell Says:


    I answered your questions on the VoIPSec mailing list where I first saw them, please check there for my reply.

  4. Dustin D. Trammell Says:


    While the three types of engagements I outlined do have some individual value to some customers, you’re right in that all three have their obvious limitations. We would generally begin a customer relationship by finding out exactly which of the three type’s results they were actually looking for.

    The problem with the ‘find it all’ approach, at least what we discovered back when we were performing such services, is that the amount of time and skilled manpower it takes to properly perform that service was generally viewed as being cost prohibitive and is as such was an extremely tough sell. We did however on more than one occasion leverage a Pen Test, as you mentioned, to up-sell a longer Pen Test to find multiple successful attack vectors or the other two services. We actually didn’t have a name for the ‘find it all’ service because it was actually just performing all three of the named services and extending out the Pen Test as long as they would agree to pay for, as well as recurring one or more of the services on a regular schedule.

    While the Assessment approach may find some initial low-hanging fruit, the real value to testing VoIP networks lies in an extended, or multiple, Pen Tests performed by people intimately familiar with the technology. Once the Pen Test is successful, the customer can always fix the issue found and then say ‘Now do it again’. Through repeating this process the more intricate and subtle vulnerabilities should eventually be uncovered.

    Anyhow, the primary point I took away from your original article was that there aren’t very many people performing these types of services that are familiar enough with the technology to do a decent job… Is that correct or did I miss the mark there?

Leave a Reply

%d bloggers like this: