Penetration testing is basically an audit of a company’s network security defences. Like it or not, there are real dangers on the Internet and security is becoming a more prominent issue, especially as more and more people work from home using the Internet to link to their organisational network. And although security scares are sometimes over-hyped, the dangers are genuine. So will penetration testing give organisations peace of mind?

We’re pleased to welcome Stephen Bishop, Head of Penetration Testing at IDsec, to clarify matters for us.

Once a year check-ups on security have become a bit like having your central heating or your boiler maintained – is that how we should look at it?

Not really. I think the central heating system is far more detached, whereas your security, or the systems your security is involved with, are really part of your business. They’re much more part of the process. So whereas while the central heating could be switched off and you could just plug in an electric fire, your systems that you’re worried about, they’ve got to be running 24 hours a day.

So this analogy of an annual event – that’s not right?

No, I think that’s far too simple. Perhaps on average annual, but really there are occasions when you’ve just got to bring forward the testing because something’s changed – if you’re changing your servers or launching a new product or whatever on the Internet – then you may have to do another test, say three months after the previous. Whereas if nothing changes then perhaps you can let it slide for a bit. So annual on average but I think you’ve got to be smarter – you’ve got to match the schedule to what’s going on in the business.

Right, so make it relevant to what’s going on. What can go wrong between tests?

Well, if you don’t change things yourself, if you don’t actually make any mistakes yourself, there are things going on outside that really can influence your security. For example, few systems are isolated – often you’re sharing the infrastructure, you’re sharing networks, you’re sharing connections with other systems – and people will make changes behind your back, if you like, that can affect you without you knowing it. And similarly, the world outside can change. There can be, in the rather refined world of networking, there can be security scares just like there are in the real world. And so the level of vigilance suddenly might have to rise and so, therefore, it brings on a test.

I know your view is that non-technical staff should understand the importance of penetration testing, and the results of that testing – why do they need to do that?

Well, to be blunt, because they pay for it. That’s perhaps the main reason that the budget is often signed off by a non-technical manager and, quite rightly, they want to know what they’re getting for their money. Also, we’re not totally divorced from the functional world and sometimes we find things, as a result of our testing, that has an impact on how systems work. For example, sometimes we find that things are actually broken, they don’t actually work properly and all that goes into the report – so it shouldn’t just go to the rather cloistered security personnel. It has to go to the people that are going to fix problems, whether they are security problems or general problems.

But presumably the problems will get fixed if you point them out, won’t they?

Well, yes, we like to think so, but it can vary enormously. We’ve had real examples (no names, no pack drill), where we’ve gone back to do testing in the same place two or three years on, and the recommendations that we put so much effort into two or three years before have not been acted on in any way at all. And that’s quite dispiriting. But to be positive, there are cases where people have fixed problems literally while we’re testing – I mean, they’ve stood beside us and said, ‘is it better now?’ and ‘good’. That’s perhaps a bit extreme, but you’ve got to put it into perspective.

Fixing problems has to be done as part of a process. In other words, you don’t want to fix perhaps a relatively minor security problem but bring the whole house down because, after all, for Internet service, keeping it going is more important than, frankly, security. So fixing it has to be thought about carefully.

Let’s bear in mind that to manage expectations, it’s important to agree the scope and the definition of the testing, as these affect the results and consequently the actions taken by management.

Stephen, depth and width testing – what is the difference and do organisations really need to know?

It is an important difference, yes. Depth is really penetration testing in the raw, where the target really is to break into a system by any means and to come away, if you like, with a trophy. Whereas width is more like the auditor’s role in a traditional financial environment, taking a broader view, collecting a broader range of evidence, and perhaps having a more structured approach, but there’s room for the two of them. I don’t really see why there shouldn’t be, perhaps, a wide framework but, within that, some depth approach so that, if nothing else, you can get some examples out and evidence.

Because, I mean, there are organisations, there are sort of hackers, who will go in in-depth to try and see what they can achieve really.

Yes. That’s no bad thing but I think one of the problems with that is they tend to be very much focused on their own interests. So, for example, they may be very interested in name servers or whatever, some technical area, and they go into that and they’re very, very focused and they may get results. But they’re not looking at all the other things that might go wrong or the other areas that might be important for your business.

So from a security point of view, we’re talking about width really.

Yes, there’s got to be a wide framework.

So then, from hackers to suits, those who understand IT but not business, to management consultants who understand business but whose IT understanding is based on applying automated tools. Not all penetration test providers are created equal. Here is a summary of the main providers:

• Independent providers – these are specialists who provide penetration testing as a standalone service.

• Systems integrators – these often throw in penetration testing as part of other services. The customer, usually a big corporate, is paying hundreds of thousands of pounds anyway, so appearing to give the client a free penetration test, worth a few thousand pounds, is easy.

• Internal services – some companies choose to use their own staff to carry out penetration testing.

• Hired hands – typically freelance IT consultants, often all-rounders who know something about penetration testing, but often not enough.

• And reformed hackers – those who really could fall into the hired hands category. While they’ll be able to carry out thorough testing, they’re unlikely to have the experience to then translate the results into management actions.

So it really is a case of ‘you pay your money and you make your choice’. So Stephen, what advice can you give to organisations to help them make that choice?

I think the main thing is that you’ve really got to get external help – not because the people inside, not because the people in the IT department, are incompetent in any way, but they really do have a vested interest and they can’t focus. They’ve got all the usual day-to-day routine to manage, whereas the people coming in from the outside, they have a more objective view and they obviously have no axes to grind.

So the golden rule – external. Anything else?

Yes. I think they should be independent in the sense that, for example, if you’re getting in an external supplier to provide, say, half a million pounds worth of firewalls, servers, networking equipment, they will often offer to throw in a free penetration test and why not? But that really isn’t good practice because, after all, it’s being tested by the people that are putting the systems in the first place.

So hardly independent?

Not at all.

So external, independence, and presumably knowing what they’re talking about? I mean, you’ve got to check what their background is.

Yes, people should have some experience in a way that they want to give you the benefits of all the things that they’ve learnt doing similar work for other people in the past. Obviously, you can’t expect all your testers to be grey-haired people who have been doing this for a long time. But, nevertheless, the team as a whole that’s coming in, and it’s often good if it’s a team, they should have, between them, experience of your types of systems, your type of industry, your sort of expectation.

So the three golden guidelines then – external, independent and experienced. Anything else?

I think you should look at their credentials in the sense of asking perhaps for some sample reports – I mean, after all, reports are the most important thing of the deliverable. You can watch them doing their work but, at the end of the day, they’re going to deliver a 20-page report. And when choosing your suppliers, I think it’s a very good idea to look at some samples and see exactly how they’re going to express themselves. So however good they are technically, whether they can put across to you their findings in a way that you can do something with them.

Well, for most managers watching this programme, those reports will be key. Best practice format dictates three sections – a summary overview, a context (setting out the scope, purpose and caveats) and a detailed section on the findings.

Stephen, tell us more about the relevance of the context section.

Well, I suppose this goes back to scientific method. It’s very tempting to get on with all the juicy findings, but it’s very important up-front to set the scene, to say how you did the testing, when you did the testing, who did the testing and the tools you used, details like that. It might be thought of as boilerplate text that you just thumb through on your way to find the exciting results. But, for example, knowing the date of the testing is quite important, because if the test was done before a major change, then obviously the results might not be quite so important.

Explaining the findings quite succinctly must be a fair challenge in itself – what’s really the best way to begin there?

Well, you’ve got to have a clear structure that separates the issues, so you’re dealing with one issue at a time, not trying to tackle them all but, more to the point, you’ve got to tease out from the mass of working results the real issues. In any report, I don’t think you can have more than about 10 issues that people can deal with. And with those you’ve got to describe what they are, how you found them, why you believe they are important to be considered at this stage and, finally, how to fix them.

How does the tester determine and communicate the severity of the results they find? Because every organisation’s set-up, is going to be different.

That’s very hard. I think we have a duty, as testers, to make an effort to categorise them but it does involve some knowledge of the importance of the systems, the importance of data held on them and, frankly, even the culture of the organisation. Although we talk a lot about black box testing, in a technical sense, I think you can’t divorce the customer’s expectations from the reporting. So, therefore, you’ve got to tailor the report to that customer because, frankly, we want people to act on reports, we don’t just want them paying.

But, of course, the bottom line is that each security issue has to be clearly reported in a consistent manner. Now this could consist of:

• The issue – a clear description of the vulnerability, using industry accepted terms, or if the vulnerability has not been previously documented, a good description of their own.

• The severity – an initial assessment by the tester, assigning consistent security levels to each hole found.

• The evidence – the tester’s reasons for believing that the vulnerability exists.

• The references – pointers to relevant external sources.

• The actions – recommendations for resolving the issue.

The key item for each issue will be the evidence.

So Stephen, how important is it for the tester to clearly and unambiguously document that evidence?

Well, it’s very important if we want things to be acted on. We’ve got make sure that possibly hostile readers see that we’ve got clear reasoning behind our findings. In a sense we’ve got to take a legalistic approach and separate out factual evidence from opinion. Also, and I suppose this is a practical point in a way, we’ve got to be able to demonstrate, if necessary, that these problems are repeatable. We’ve got to be able to show somebody who may be sceptical, or possibly after they’ve tried to fix it, how we’ve found this problem, how they themselves could repeat it and how it can be shown that the security hole has gone away.

So how could a tester demonstrate vulnerability?

Well, let’s give a very, very simple example. We do often find, even nowadays with all the awareness there is, that people use bad passwords. And so that’s a very direct way of demonstrating to somebody, if we found a username and password that’s easily guessed. We can just show them, or even get them to see on their own PC or laptop, that they can go to a server and get access to information just by simply being prompted for username and password. They put in something that we’ve guessed or we’ve deduced from circumstances: and bingo – there it is.

So it can be as simple as that to point it out?

Yes.

Most vulnerabilities are difficult to demonstrate, as they’re discovered by inference, I gather. Now what exactly is meant by that?

Well, that’s where can’t do a direct demonstration, we can’t say ‘oh look, you do this, and here we are, we’ve found a hole, we’ve walked through that hole’. It’s where the general properties, if you like, of the system we’re looking at – in other words, things like version strings or even your manufacturer’s details and so on – lead us to deduce that there is a vulnerability. So it’s circumstantial, we’re looking at something, we see that it’s Apache Version 4.3, and we know that 4.3 is out of date. We know that that’s two years old, that people have found 10 vulnerabilities into it and the likelihood is that they’re vulnerable, but unfortunately we, for various reasons, can’t demonstrate it.

But it’s still a vulnerability all the same.

So the customer has got their penetration report – now does it get filed and the box ticked or does it actually get actioned? In your experience, what ensures that actions follow the report?

Well, that’s the skill of it. We have to tailor the report to the customer and in some ways we have to make sure that it’s the sort of report that they want. For example, if we’re dealing with an existing customer, somebody that we’ve worked for many, many times before, then they don’t need all the preaching about security.

They just want the facts, they want very simple pointers, they don’t want huge amounts of reference material, they want something short and sharp. But with a new customer, or a more naïve customer, we do have to give much more material. We have to give background, we have to explain why, we have to not exactly scare them but we have to set the scene for the results. We have to put everything in perspective. So they might get a 30-page report, whereas pretty much the same work for an existing customer who’s more savvy, they might get five or 10 – but that’s right for them.

So some customers will want what? Literally a quick chat through?

Well, in the most extreme case, we’ve had an example where a customer, after us doing four or five days’ work, wanted just a five-minute telephone report. So really I said, ‘we’ll send you a full written report’, but he wanted a five-minute conversation, and at the end of it he’d got all he wanted out of it. And I think I had to send him an email, I felt guilty not sending him an email. But that’s what he wanted and he’s the customer.

And for other customers I imagine splitting it up into parts so they can get the bit they want perhaps?

Yes, very much so. That’s a practical point. In a large organisation with a large system, we don’t know what we’re going to find. But often our findings come from, or are the responsibility of, different departments. And the customers and the commissioning IT security manager may want to, almost literally, separate out the report and give this part to that team and so on, because he doesn’t really want them all to see the whole thing.

So find out what the customer wants – what’s relevant to them is important.

Yes, very much so.

Let me ask you, finally I suppose really, where do you think the greatest threats lie now and how do you think it’s going to change in the few years ahead?

Well, I think that the greatest threat really is that security, as a function, hasn’t got the prominence that it ought to have. We know that there are companies out there that do take things very, very seriously but often security is a bit of a bolt-on. I mean whether it’s penetration testing is sometimes a bit of an afterthought, are whether there are active mechanisms, their components, are just added on pretty much after the application has been sewn up and that doesn’t really work. I think that the problem is that security is often said to be one of the top four issues but unfortunately never becomes the top issue and that’s been the experience over the years.

And, of course, those causing problems are becoming more professional and have done over the years. Computing is becoming more mobile and so on. All these issues I imagine you’ve got to deal with.

Yes, I mean very much so. The bad guys out there are actually getting smarter – they are very smart – and I think it’s true to say, I mean we don’t have any real figures of this, that most of them are now doing it for financial gain or possibly as real acts of vandalism. Not so long ago they were just doing these sorts of things to prove how smart they were. And we are all making the problem worse for ourselves by wanting to adopt mobile computing – to use the laptops on the train, to connect them up to customers’ networks, to connect them up to home networks, to have information on the mobile phone. And that’s quite a nightmare for IT security managers but, of course, the benefits are huge.

So clearly a lot to be aware of. Stephen, thank you very much indeed for that. Penetration testing then – a key component of risk assessment for all organisations, making extensive use of the Internet within their communications network. Bottom line is – follow best practice, ensure that you and your customers sleep soundly at night.

Many thanks to Stephen Bishop – there’s more from him on this subject in the study notes. But for now, from all of us here at Einstein Network, goodbye.