Key Factors to Consider When Hiring a Third-Party Penetration Testing Organization

This post might come across as self-aggrandizing; after all, Rotas Security is a third-party penetration testing company. While there’s some truth to that, my aim is to provide general insights that could be helpful in assessing the technical prowess, professionalism, and expertise of pentesting companies out there.

Throughout my career, I’ve had the privilege of wearing many hats. I’ve served as a Soldier, a DoD contractor, and a civil servant. In the private sector, my roles have ranged from security analyst and vulnerability manager to consultant, as well as managerial and directorial positions. My experience isn’t limited to executing and overseeing penetration tests; I’ve also been on the other side, responsible for hiring professionals to conduct them. I’ve also been a practice director for one the largest pentest shops in North America, and spent a lot of time getting grilled by many types of organizations; from small school districts to fortune 500’s.

I hope the principles shared in this post will be valuable to those in need of pentesting services.

A solid pentesting organization isn’t just adept at identifying, validating, and exploiting weaknesses; they should also be capable of guiding you through their process in an understandable way.

 

The Wild Wild West

When everyone is special no one is

Pentesting is like the wild, wild west: an unregulated industry. To practice law, you have to pass the bar; to be a certified public accountant, you must hold a license; and medical licensing board exams are required to practice medicine. However, to be hired by companies to hack into their systems and networks professionally, you only have to… offer to do so. There are certainly professional certifications individual practitioners can obtain, and in the UK, companies can acquire the CREST certification. But there is no official oversight and governing body presiding over pentesting in many parts of the world, including the United States.

Some may argue that pentesting is “self-regulating,” in that poor work will tarnish your reputation, and people will “just know” you’re not a good company. However, this presents a paradox. To recognize “good work vs. bad,” one must have a depth of expertise and understanding in adversarial techniques. If you’re hiring a third-party company for reasons other than regulatory compulsion for an external perspective, you may not have staff with that depth of expertise. That’s why you’re hiring a third party. So, how can you ascertain if pentesting organizations are any good? Is it based on the depth of their report, the number of flaws they find, or whether they “get in” or not?

I propose that you first need to decide what you aim to achieve from a pentest. Then, leverage some of the tips below to help form your own opinions when choosing a partner.

Reputation Score

When you do good work, people are willing to let others know

Earlier, I mentioned that some believe the self-regulating nature of the infosec industry, particularly pentesting organizations, helps keep it in check, and I agree there’s truth to that. However, I believe this should not be the only factor considered. It’s similar to when an individual passes on another’s resume for consideration for a role. You take into account the person making the recommendation, as they are willing to put their name behind the candidate. But you still interview the candidate. The same principle applies to pentesting. If several colleagues in your professional sphere mention having received good work from an organization, it certainly merits consideration.

Penetration testing is more than just hacking and finding flaws; it’s about effectively communicating those flaws, helping to triage and prioritize with expert input, and going beyond what tooling and automation can provide. Some organizations are so profit minded they want to churn through pentest work as rapidly as possible, so they can move on to the next gig to keep the coins flowing. Even leveraging generative AI, breach and attack-sim tooling, and expert automation techniques, getting this activity executed thoroughly takes time, and the level of effort and time is commensurate with the size and scope of the environment.

If you’re only running scans and using tools, without innovating or finding novel ways to circumvent security protections, or extend the viable attack surface, you’re not delivering the value you’ve promised to the organization when offering a penetration test. If pentesting were merely about billing someone to run scans and fire off a few scripts, then everyone would be doing it… and, sadly, that is often what happens.

All this is to say, consider what other trusted individuals in your network have to say about the organizations they have used.

Method is in the Madness

What is it you say you do here?

When you’re on the hunt for a pentesting company, one key aspect to look out for is their ability to clearly explain their assessment methodology. A solid pentesting organization isn’t just adept at identifying, validating, and exploiting weaknesses; they should also be capable of guiding you through their process in an understandable way. This involves outlining their planned actions, the tools and techniques they’ll employ, and how they’ll safely simulate real-world cyberattacks. During discussions about their testing methodology, the organization seeking pentesting services should present unique scenarios. In response, the prospective pentesting team should demonstrate how their methodology can be effectively applied to these specific situations.

This level of openness and detail in their approach goes beyond mere professional jargon; it’s indicative of a pentesting company that truly knows its craft and takes its responsibilities seriously. It’s about offering you, the partner, a solution that’s tailored to the unique security needs of your business, rather than a generic, one-size-fits-all approach. This method fosters a relationship built on trust and understanding, positioning the pentesting team not just as a vendor, but as a committed partner in your security. The right pentesting team will instill confidence, assuring you that they’re not only testing your defenses but also fortifying them for the future.

Community Involvement

You get out what you put in

A defining characteristic of pentesting organizations that value excellence in execution is their involvement in the InfoSec community. This isn’t just about attending conferences or being present in industry events; it’s about actively contributing and shaping the cybersecurity landscape. These organizations often go beyond passive participation, taking on roles as sponsors and key speakers at InfoSec conferences. This level of engagement demonstrates their commitment to staying at the forefront of the latest security trends and technologies. Moreover, they share their expertise and research, whether through publishing insightful white papers, contributing to open-source projects, or presenting interesting findings at conferences. They also provide training workshops so that others can garner more understanding and learn from them. This sharing of knowledge not only cements their status as experts but also shows a dedication to advancing the field as a whole. Networking and collaboration are also integral, as they exchange ideas and foster innovations that benefit the wider community. For any business looking for a pentesting partner, a firm’s active role in the InfoSec community is a strong indicator of their expertise, commitment, and the value they bring not just to clients but to the advancement of cybersecurity itself.

The type of conferences that organizations sponsor and attend can also paint a clear picture of their priority. There’s nothing wrong with places that are successful, and have the budgets to be able to sponsor and attend some of the very high-profile (and often expensive) grandiose events. Some of these events bring in excellent folks who can train, speak, share and engage very well. However, due to the cost, many professionals and technologists are unable to attend these events. I think there’s a lot of value in seeing organizations who not only attend/support/sponsor the “big cons” but also support smaller, sometimes regional, conferences. For example, various BSides events, or conferences with names like <something>con. Conferences that include colors (bonus points for red or blue) should also score high marks on your list. That last sentence is in jest. The point being these organizations recognize the value in conferences beyond just marketing purposes. Showcasing not only domain eminence, but focusing on activities that enrich the InfoSec community as whole, offering training, and being actively engaged with the community.

Showcasing Expertise

Put Up or Shut Up

An organization that has been performing professional pentesting for any length of time should be able to quickly produce an example report (like this Rotas example pentest report). This should highlight their expertise, so they’d put their best foot forward. Based on what they are willing to call their best work, you can determine, at some level, how good they are. Is the report thorough, does it contain data that you’d expect in a format you’d like to consume? Is their reporting strategy dynamic or static?

Asking for professional references is another good source of truth. Again, it’s expected that companies will only put organizations that they feel they have had a good relationship with out there, but if can give you the chance to follow-up on the reference, ask some questions about the previous pentesting engagements.

Although many pentesting organizations are not at liberty to disclose their clients, they can provide examples of some of the work or projects they have done, at a high level. Look for organizations that can produce examples that are relevant to your industry or your IT ecosystem. In conversation you can ask the prospective pentesting partner to provide some anecdotes of common issues they’ve seen that organizations (similar to yours or within your industry/sector) face.

I’ll go somewhat against the grain and say that asking for a resume of the tester(s) is a bit of a waste of time. There’s no way for you to know, at that level, if that will be the real human turning screws behind the scenes. It’s on them to put up qualified talent. Although, same as an example report, asking for some example resumes can be another data point. Due to the nature of project schedules, it’s typically impossible for them to guarantee the resource availability until a schedule is being built, and the project is much farther along than “just kicking the company tires”. You can get an idea of the type of folks that work there.

Having the company provide a list of relevant industry certifications that their testers hold may be another good way to determine if they are staying relevant and engaging and training their people. Certifications certainly don’t mean their people are advanced or truly skilled, but it shows an investment in continuing education and trying to stay relevant with emerging techniques.

It is helpful to sleuth on the company on social media outlets. That can give you somewhat unfiltered (or less filtered) access to see their personnel, their GitHub projects, and social media postings.

Focus

A mile wide and an inch deep

Try to determine if they have a dedicated focus on penetration testing. Regardless of the organization’s size, the absence of dedicated teams, divisions, or a specialized boutique approach could result in a commoditized, “best effort” pentest conducted by smart technology generalists. True pentesting is not just an “other duty as assigned.” It’s not a skill that can be passively acquired or picked up casually. Pentesting is a unique blend of deep technological understanding, diverse logical environment exposure, and a mindset honed through experimentation and mentorship.

There is also the matter of specialization to consider. Certain facets of the technology landscape are more complex than others. For instance, application security is nuanced and requires a high degree of expertise. Areas such as Operational Technology (OT), industrial control systems, mobile platforms, code reviews, hybrid cloud network infrastructure, and physical security are just a few examples where experienced professionals are particularly valuable. Some pentesting companies may not possess the requisite personnel with the necessary skill sets to execute specialized work effectively.

I’ll admit my bias: I believe that opting for shops solely focused on providing offensive security services ensures that you are engaging with serious players in the space, whose entire focus is on performing adversarial simulations. However, I’ve also experienced being part of a dedicated attack & pen team within a very large, multifaceted technology company. In that setting, our division operated like a small, specialized company within the larger entity. Really it doesn’t matter if it’s a group, a department or division, or the company’s entire purpose. The key takeaway is this: regardless of the organization’s size if there is a demonstrable focus and emphasis on performing offensive security services you should do well.