Running a capture the flag event (for testers)

We are knowledgeable enough to at least mitigate the most common security risks through refinement and testing while we develop. To do this we have a lot of resources at our disposal, first and foremost those offered by the Open Web Application Security Project (OWASP).

OWASP has several projects of interest to testers. First and most well-known is the OWASP Top Ten, listing the ten most critical security risks. This document also includes a section on “What’s Next for Security Testers”, which is also interesting reading for non-security testers. It offers links to further reading, advice on test strategies, and ways to get started. One of the documents referenced is the OWASP Testing Guide which describes vulnerabilities and how to test for them. Besides its use in manual tests, the document can also be used to create automated security tests.

To make my fellow testers aware of these resources, and to lower the barrier to “scary” security testing, I organised a small capture the flag (CTF) event for my colleagues at Centric Test & Security Professionals. My aim was to have them find some common security holes so they could see how, with some simple tests – which in some cases can quite easily be automated – major security risks can be found, even by non-security testers like us.

To do this I used the OWASP Juice Shop application, developed to be “the most bug-free vulnerable application” by Björn Kimminich. This application, which can be run locally (using Docker, Node.JS, or Vagrant) or online for free, makes learning about security fun.

Because I used (and tweaked) some online guides on how to run your own CTF event, most notably this post by Josh Grossman and the book Pwning OWASP Juice Shop by Björn Kimminich, I would like to offer a run down on how I set up the CTF, what I used, what I learned, and tips if you want to do it yourself.

The set up

For the CTF event I prepared four components for the evening itself. I made a presentation of about 10 minutes, just going over what security testing is, the types we run into in our daily work, the laws (in the Netherlands) that mandate us to do something about security, and where to find resources to help us testers do this.

I did look into converting my site from Hugo to Gitbook so I could use a fork of Kimminich’s book repository to make my own, but I couldn’t get the legacy version of Gitbook (the one that still can be run locally) working on my machine, so I decided to stick with the (to me) more familiar Hugo site generator instead. Because I used Hugo I had to redo some of the internal links in the pages I got from Kimminich’s book, but with a quick find-and-replace action this was not a lot of work.

Making the Juice Shop available

I previously hosted a workshop similar to this one where I asked all participants to install Docker or Node.JS on their machines so they could run the Juice Shop locally. However, as expected, during the workshop we still spent a lot of time getting the Juice Shop running for everyone, and because the laptops varied in capacity, some participants couldn’t even run the Juice Shop because of storage limitations.

I had read about the Heroku option in Kimminich’s book, but was hesitant to use it. Sure, it said it was free, but I needed a dozen instances, and I was certain it wouldn’t be free for me. However, Grossman’s post about his CTF event convinced me to try it. It couldn’t have been easier (and it was indeed free).

When the dyno is not used for a while it stops running. So, because I knew it would take a while for the dyno and app to start up again when I was ready to use the instance, just before the event I visited all my instances just to get that start up time out of the way.

The CTF platform

A CTF platform like CTFd enables challenge tracking and score keeping during the event. I found the CTF platform, how it works, and how it is used the most difficult part of setting up the event. It took me a while to get my head around the fact that the CTF platform is “just” another site available to all participants where they log in and register the challenges they find. The CTF platform has no link to the Juice Shop, and the Juice Shop has no link to the CTF platform. The Juice Shop offers a tool to create a “backup” of the CTF platform containing all challenge information needed, but this backup is also not linked to a specific instance of the Juice Shop.

From the documentation available, and from Grossman’s post, I got the idea that CTFd is a pretty ok platform to use, so I picked that to install. I don’t have my own publicly available server, so I tried to get a company server to host CTFd. However, the only one available was running an old version of Windows Server and CTFd needs Linux to run. Luckily Grossman ran CTFd on Amazon AWS, so I figured I’d give that a try.

After changing the docker-compose.yml I used Docker Compose to get CTFd running.

On my own computer (so not the Heroku dyno nor the AWS instance) I used the Juice Shop command line tool to create a “backup” of the CTFd settings needed to use CTFd for a CTF event using Juice Shop. I made all the hints free (because I was giving them away anyway in my support site).

Uploading the backup to my CTFd instance was a bit confusing. You have to make an admin account, upload the file, restart, and then create another admin account (because your first was overwritten by the backup file I guess). After doing all that I had the Juice Shop challenges loaded in my CTFd instance. I then customised the front page with our own logo and a link to the support site.

I use Netlify to manage the DNS settings for my personal site, and I set up a subdomain pointing to the AWS IP. When I turned on my AWS instance again just before the event, I updated the IP in the DNS settings and found that it worked nearly instantaneously. I chose not to use TLS, both because setting up everything else had cost me a lot of time and effort already and because this sounded pretty difficult too. I realise that by omitting TLS, despite it being relatively simple due to the service of Let’s Encrypt and the EFF, I am doing exactly what OWASP is trying to prevent, and not making my app as secure as it could be. Mea culpa: next time I will surely try to figure out how to also set up TLS myself.

Organising the event

Before the event started we gathered a list of all participants. We created teams by pairing up testers based on technical ability: a tester with more technical knowledge was paired with one who had less experience with code, automation and security. The hope was that the first tester could share knowledge and experience with the latter. Inspired by the Juice Shop theme I created twelve Juice Shop instances, each with their own fruit-based name. I also created twelve similarly named accounts in CTFd. I made place cards with the team names on the front, and the relevant links to the Juice Shop and support site as well as the login details to their CTFd account on the back. Once we were certain who would attend, I also wrote the team member names on the back of the cards. I had created two extra teams as a backup in case an instance got corrupted or was otherwise not available.

Each participant brought their own laptop so it was just a matter of guiding them to the right seats and team partner and getting started. I had connected my own laptop to the beamer so I could project the scoreboard (automatically updated every 10 seconds by using the Chrome extension Super Auto Refresh Plus), one sheet with the relevant links, and a timer counting down the remaining time. I had another laptop with me for troubleshooting.

I set some ground rules before we started:

• No googling the answers or any Juice Shop related info
• No DDoS or extensive tool usage to attack the Juice Shop instance (because my credit card is linked, so be nice)
• No attacking CTFd or the other teams’ Juice Shops
• Start time is 18:30; end time is 20:30. Challenges can only be registered between these times (this is a setting in CTFd you can use)
• Work together, read the support docs well (especially the chapters “Walking the happy path” and “Hacking exercise rules”)
• Asking me questions is allowed.

I walked them through the first challenge of finding the score board, and then set them loose.

Experience and lessons learned

When organising an event like this, especially one outside your comfort zone – like hacking is for us testers – the question is always, how well will it be received? I am happy to report that the participants all took it very seriously (you could hear a pin drop during the two hours of the challenge) and did their very best. Of course, some participants took to it better than others, but all reported that they found it a very enjoyable way to get to know security testing and ethical hacking a bit better.

I noticed several minor things during the event that I might do differently were I to do it again:

• The support site I set up was missing some internal links (notably those from the challenge overview to the challenge descriptions). This was because of the difference between Hugo and Gitbook and how they format those links. Luckily the fix was done pretty quickly thanks to my setup of GitLab and Netlify, but I should have tested these links better.
• Late arrivals found it hard to get started, so I needed to get them up to speed before they could begin. The next time I would make a dedicated page on the support site with the basic getting-started information.
• I had registered the teams in CTFd using just their fruity team name, which was also the name that was shown on the scoreboard. This wasn’t much fun as you couldn’t see who was beating you. During the challenge I changed the team names to include the members’ first names. Luckily CTFd didn’t kick out those who’d already logged in with the old team name.
• Some teams had a hard time getting started. Either they did not understand how to use CTFd, or they were solving challenges but not registering the codes, or they were unable to solve the challenges altogether. Each team should be able to register an easy 100 points from the demoed “Find the Score Board” challenge, so any team that failed to do that I helped. If a team did not have more points after an hour I started helping them with hints.
• The participants took the “No googling the Juice Shop” rule to mean “Don’t google at all”. This held them back because some of the terminology was totally new to them. I had to clarify this rule.

All in all, this was a very fun event to organise and run. I learned a lot, too, while organising the event, mainly about cloud services like Heroku and AWS that I had never used before. I have to admit, the pricing structure of these providers scares me (I like a fixed price), but I was pleasantly surprised to find that running this event with 18 participants cost me nothing in the end. I’m also happy that my colleagues took the event seriously and had fun that evening just trying a bit of hacking. I know that raising security awareness and enabling “ordinary” testers to do their own security tests will entail a lot more than just one CTF event, but at least we got started. In the future, I might organise more knowledge sharing based on the OWASP Top 10 to help testers with security testing.