When working with clients, often one of the first things I am asked to discuss is what services I have to offer them. My niche is application security. It is important for me to explain that the type of testing I do is different from what they may have received in the past.

Most likely, the client at some point has had a network security assessment or "pentest". I use the term assessment throughout this article. It aligns with what I offer more than other familiar testing terms. I also don't consider the auditing of a network or web application a penetration test. For a more thorough breakdown of assessment types, I recommend reading Daniel Miessler's article, Information Security Assessment Types. Regardless of what the client has had in the past, it is certain they've had someone poking around their network. But, in reality this was not in-depth application assessment.

During a recent client assessment, I discovered a Cross-Site Scripting (XSS) issue. This issue had been on the site for many years through many pentests. I'd like to say I found it due to my  technical knowledge of JavaScript and browser behavior, but that's rarely the case. Anyone with minimal knowledge of application vulnerabilities could have found it. This raised questions from the client asking why it wasn't found during any of these tests.

I find myself explaining these testing differences to clients often. So I decided to explain how I approach an assessment. This is so the client understands the scope of what will, and will not be, part of the assessment.

How I Start My Engagements

An engagement for me starts with listening. A majority of my work comes through referrals or other security companies. I take the time to understand what a client is looking for. I need to make sure that we're a good fit and that I can meet their expectations.

It's common to have a client who has never received an in-depth application assessment. Often though the application was part of a previous network assessment. Previous findings may include insecure TLS configurations or missing security headers. These are "low hanging fruit" per se and important, but only scratch the surface.

Clients may approach me wanting an application assessment and a network assessment. I'd never do both and it's important that they understand that I won't be looking at the network. These types of discussions allow me to determine if I can deliver a quality assessment. For me the primary difference between these two assessments comes down to scope.

How I Scope an Engagement

For most network assessments, the tester receives a list of IP addresses. This could be a few dozen, hundreds, or even thousands. Among these addresses there might be a few web applications, such as a company website or a client portal. Generally, the tester only has a couple of weeks to complete the assessment. This includes the analysis, review, and reporting for the test. During these, any time spent on the web applications is only going to be a very superficial, surface look.

If you’re hiring someone to test a single application, that entire test may last 60-80 hours (or more). This time is all spent on the assessment of the application and always include testing for  common issues. When I'm doing work with a client for the first time, my preference is to do what I'll call a baseline test. In this test I'm looking at all areas of the application. This allows me to gain an understanding of the application and cover all the common issues. As time allows I'll go deeper into certain areas where it makes sense. I allot eighty hours over two weeks with per-project pricing for these assessments.

I've also had assessments that focus on certain functionality for more in-depth testing. Authentication and authorization systems are common for this type of in-depth testing. Role-based access controls and privileges get complex and get out of hand real quick.

It takes time to understand the purpose of an application, what it's used for, and who the typical users are. Some questions I ask when I start assessing an application:

  • How does the application behave under normal conditions?
  • How does it behave when something goes wrong?
  • What pieces of functionality do users use the most?
  • Which pieces aren't used that often?

The time spent learning the application has a huge benefit for the client: in-depth testing of the application's business logic. It's not uncommon for testers to find the most critical issues near the end of the testing window. These are issues that automated testing or "quick looks" will never find. They are also most often the issues that will cause real damage to the client. For example, testing a financial application would be different then a medical portal. The financial application allows users to check balances or send money. The medical portal shows appointments, conversations, and exposes sensitive medical tests.

Both applications may rely on the same technologies, but they need different functional testing. This all takes time and diligence. Automated or "off the shelf" tests won't find flaws at this level. This brings me to the next point which is the skillset of the tester do the actual work.

The Tester’s Expertise

In most cases, vendors have different teams handling application security and network security. If not different teams, then people that specialize in one area or the other. If you want a network assessment, you want someone skilled in network security. Someone that understands the low-level networking protocols. That know the ins and outs of how traffic flows through the network and best practices gained from experience.

For an application test, you want someone who knows the frameworks in use. It could be a ReactJS frontend with a Rails backend or an Angular .NET Core application. All frameworks use the same underlying protocols, but vulnerabilities can look different. How an XSS or mass assignment issue manfiests in each can be very different.

There's often a lot of cross-over between the skillsets of any tester and sometimes there's not. We can't all be good at all areas of security. In the past I've done network assessments and helped out other teams. While I'd do my best to ensure I gave an adequate test, it's not where my skills lie. Same with social engineering or physical assessments.

The field of application security is growing and there's a need for a lot more skilled testers.

What to Know Beforehand?

I've approached this from the viewpoint of a consultant. What if you do need an application assessment? How can you be sure if you're paying for one that you are getting a skilled professional on the other side?

It helps to come at this with a bit of knowledge. Familiarize yourself at high level with the Open Web Application Security Project (OWASP). The following are three documents I recommend to be familiar with:

These are a great place to start and serve as the baseline for all my testing. You don't need to understand any of these in-depth so I'll explain each at a high level.

The Top 10 Web Application Security Risks

This is the OWASP flagship project. Every three years the organization reviews their data. Data they receive from security professionals, developers, and organizations. From this data they identify what the trends are and rank the issues. This forms the OWASP Top 10. These issues include injection attacks, authentication issues, Cross-Site Scripting (XSS), and many others.

This is a great starting place to gain an understanding of the common application security flaws.

The Top 10 Proactive Controls

Throughout the years the OWASP Top 10 Risks have changed. What doesn't very often though is how to address these risks. This is where the Proactive Controls comes in. Each of the Top 10 Risks has a complementary Proactive Control.

These controls are fundamental to the development of secure applications. Modern development frameworks have many built in controls for handling security issues. There's no reason to not have any of the proactive controls in place.

The Application Security Verification Standard (ASVS)

The ASVS is also an incredible resource that I recommend all  teams become familiar with. This standard touches on Authentication, Design, Session Management, and many other areas.

Before spending money on an external assessment I'd suggest to have an internal one for a few days. Have your development team run through the ASVS checklist. Fix or have a plan for anything on it. This won't be conclusive, but you'll at least have an idea of what type of issues you might see. This will also help you identify where gaps might be in the testing.

It's all around a good idea to do this regardless of whether you're planning an assessment. The other bonus is that the ASVS covers a majority of issues that the tester will be looking for. If you're team can address issues before the test, you'll have external validation that the fixes are adequate. The ASVS also  as the backbone for all my testing. I also use it when developing client specific secure guidelines.

What to Ask The Vendor?

Ask the vendor for sample reports. In all cases make sure you understand what you're getting. I have example reports for that I keep updated for this reason. Compare the findings in the sample reports with the above. Do they mention these and map findings back to the appropriate categories? Do their recommendations make sense? Involve your development team in this and get their feedback. Many development teams don't like to be on the receiving end of a security review. Involving them in the process is a great way to circumvent any adversarial feelings. Any good development team should look forward to anything that will make them better.

What about certifications? This is a hot topic and there's plenty of debate on both sides. I'm not going to dive into the merits of having or not having certifications here. When it comes to application security, there's very few relevant ones. There's not a single certification I can look for and know the tester has the necessary know how. While many certifications might cover areas of application security, it'll be broad.

The big three cloud providers all have security certifications. While not specific to application security, they cover the necessary service technologies in-depth. For anything cloud specific, look to see if the tester has any certifications from the vendor. Vendors provide a lot of learning material for these certifications for free. The exams themselves aren't expensive and any vendor should be more than willing to help offset the costs for their consultants.

Also, talk with the person who will be performing the actual testing. Have one of your lead developers on the phone too and have a conversation. Don't worry too much about titles here or whether someone is a junior or senior. I've worked with more than capable people in appsec with all titles.

Conclusion


Web applications increase the attack surface of any organization. A single application flaw can lead to a breach of data or the organization. An application security assessment can expose risks that other testing won't. Risks that all the time and money spent on securing the network and perimeter won't prevent. These risks underscore the importance and need for solid testing.

I hope this article sheds light on the differences between these  testing strategies. Both testing strategies are neccesary but serve different purposes. If you're having trouble finding a vendor or want to make sure you're on the right path, feel free to reach out.

Photo by Pankaj Patel on Unsplash