Thoughts on CTF, CCDC and Simulations

I’ve been sitting on this topic for a while since I read the articles, now’s as good a time as any to put this to print. Capture The Flag competitions, like Black Hat’s renowned competition, have long been a staple of hacker conventions and way to get some industry bragging rights under your belt. Conversely, defensive competitions like the Collegiate Cyber Defense Competition (CCDC) have caught a bit of flak about not being “real enough”. I’ve personally never found criticism of one event, using the other as a basis for that criticism, to be necessarily valid because they tend to dig deep, split hairs and otherwise ignore the obvious – the obvious being that the competitions have entirely different purposes.

In the interest of disclosure, I’ll say now that I have not (as of yet), participated in a CTF challenge, team or individual. I have nothing against them and I agree with some of the points about them being more ‘holistic’ approaches to information security as general practice. I have; however, competed in CCDC when I was in university, leading my team to a respectable finish. I’m no stranger to the criticism between the two types of events – we had some heated discussion during our preparations for the CCDC on my own team about which was “better”.

The articles that got me thinking about this topic is Matt Weeks’s from ScriptJunkies’s Thoughts on Security and a response from Carnegie-Mellon’s CTF team, the Plaid Parliament of Pwning. Weeks’s article is itself a response to a conference keynote and presentation. While their points and criticisms are fair, like earlier, they fall into missing the obvious point until their conclusions, that each competition has different goals immediately from its outset.

The CCDC is a defensive competition for college/university students that is intended as a simulation for entering corporate or enterprise environments and securing the business. Student is implied in the name of the competition but defense is explicit. Those pillars factor heavily into the goal of preparing teams for “99% of InfoSec professionals in business”. As this is an educational exercise, carried out between institutions with that defined goal, there’s a lot more structure (aka, “rules”) on them. To my knowledge and memory, CCDC isn’t claiming to produce the 1337est hackers. It is providing a simulated experience to grads and undergrads on what their most typical security experiences will be out of school. In an economic environment where truly “entry-level” positions are a rarity for skilled and white collar careers, CCDC is helping close the gap for students actually having some qualified experience to whittle away that 2 -3 years experience request from the job recruiters and interviewers.

CTFs have also a predefined goal – capture the identified target. And from what I gather, using a borderline MAD policy to keep your opponents from obtaining the target are a lot more flexible than CCDC’s strict “No Hacking” rule. I agree that these competitions have a more “holistic” view of information security field – systems, networking, cryptography, penetration testing (receiving and committing) to name some highlights. But the goal of CTF is still to get the target. I’m hard-pressed to come up with an overarching purpose, other than educating oneself or building a resume that already has some accomplishments, than obtaining the target. And that’s OK!

With two competitions, with two different outcome goals – obviously, one must be better than the other, right!? Not quite. I would never eschew anyone from competing in a CTF at any stage of their career. There’s a lot of stuff that folks who compete in CTFs can do that I could ever hope to (or, at least, that’s what I’ve built myself up for). However, I would also recommend people to compete in CCDC if they are given the opportunity. Complaining that CCDC and CTF don’t have the same outcome due to their different nature is complaining that all that time on the treadmill isn’t doing a lot for muscle definition. But this leads to a broader objective that I think is being neglected when we get caught up in these discussions: the importance of simulations.

Simulations are key for implementing what we don’t want implemented: an attack that brings the business down. While there might be well-crafted policy to address an incident, without incident simulation, we do not have a complete set of data against which to test and improve that policy or run-book. And ultimately, that is the take away for each type of competition – each is a type of simulation with immediate benefit to its participants. But, are there other benefits for non-individuals for say, a company organizing their own simulation?

I would argue that yes, there’s a benefit to the corporate organization beyond individual skills. By running a more structured simulation, a la CCDC, Security and Operations staffs can tune their existing procedures and update their run-books for more established threat vectors (that do not occur regularly) and have a very effective post-mortem sessions. Well scoped vulnerability assessments and penetration tests can serve for this – a previous company I worked for had several clients specifically outline VA/PT engagements to test our response to security incidents. From there, we would be involved in the post-mortem for what improvements were suggested in the final report. Particularly for an company that feels short on time and cash, incorporating this into a vulnerability assessment or penetration test that is already budgeted, is an advantageous way to schedule and organize this simulation.

Likewise, an appropriately scoped penetration test can serve for an easy implementation for a no-holds-barred CTF-styled test for your Operations and Security staff, without the trouble of having to face compromise at the hand of a non-simulated threat! With the CTF styled attack, the organization is not only testing its controls and responses for established threat vectors but also hopefully finding an organization’s technical Maginot Lines.

While incorporating either simulation into routine vulnerability assessment/penetration test is the easiest way to perform simulations like CTFs or CCDCs within your own staff, scheduling your own internal simulations along these guidelines are worth pursing as well. When an organization designs its own tests and takes its own simulations to practice, that is where the true learning and improvement of the organization as a whole will come from.