Sunday, 22 May 2016

What's the point of (InfoSec) Certifications?

Quite recently, my GSE was up for renewal.

I'm currently in the middle of transporting my family to another continent and I've slightly more responsibilities work-wise in 2016 versus 2012. However, given the effort and study that it took to get the cert the first time (and to a lesser degree the expense), I figured it was a no-brainer to renew. For me, I've always been a huge fan of the GSE and considered it the epitome of InfoSec certifications, much like the CCIE for (Cisco) networking.

Personally, I learn better by "doing" and consider it as the evidence that someone knows their stuff so the "2-day lab" element in the GSE was a both a huge goal and challenge that I was excited about. I talked about the value of "doing" when trying to learn about yourself previously here with the infamous Security Ninja and here on my own blog so there's no point in repeating myself.

When I did the GSE, I absolutely loved the hands-on lab more than anything-else I'd done in the world of SANS or GIAC, outside of Mike Poor's 503 Packet Work book (if you like packets, this is heaven, literally :) ) and the "Capture the Flag" exercises created by Ed Skoudis in 504 and 560. I've also had some amazing instructors like Arrigo Triulzi (Arrigo teaching SEC504 actually convinced me that my future was in InfoSec) and Stephen Sims, however, I am questioning more than ever the value of certifications and to a lesser degree the training courses (which are priced to be exclusive to a tiny minority who are already fairly well off or lucky - I often recommend Coursera or the Offensive Security stuff to candidates when cost is a real issue).

I do truly value the education that I have received through SANS as well as the support from the companies I've worked at and more importantly my family in order to be able to do those certifications and ultimately get the GSE but now I'm questioning the value of many of them, more than ever. Maybe it's the fact that I've seen people turn up with certifications and know absolutely nothing (the CEH and CISSP-holders are more often than not jokers), though I've never seen a GCIA-holder turn up and not know packets :) I haven't encountered too many OCSP or OCSE holders to have full context on those but having read their syllabuses, the practical element sounds awesome.

As for SANS, as I've said, I've learned hugely there but it truly does seem over-priced. I recently did the FOR572 with Phil Hagen, who also seems to be a great instructor and an awesome dude (based on my limited interactions on email), however, I paid (or in this case Riot did) practically the same price for the course in "offline" format versus "attending the course physically for a week". I am truly baffled as to why there's such a minute difference but I've never seen or heard a good explanation. As for the course, it was very good although I was very familiar with the vast majority of the topic.The exam for this was easily the worst GIAC exam I've ever done - the questions were not clear, occasionally the answers were incorrect and there was too much ambiguity. SANS did reach out but the discussion didn't go too far, I suppose this is natural in comparison to the GSEC but both exams are around $400 to challenge on top or the SANS course or $1149 outright to challenge so surely the standard should be much closer.

Additionally, sadly SANS and GIAC still seemed to be heavily geared to the US Market. For example, with the GSE renewal, the cost is $399 and you get one set of books (i.e. either the GSEC, GCIA or GCIH) included. Sounds sweet, right? Unfortunately if you live outside of the US, you've to pay $199 to ship that one set outside of the US. Being a tight-ass, I used my old books :)

I'm immensely proud of passing the GSE back in 2012 and realise that I am also incredibly lucky to have a company (thanks AIB) to support me in the burden of the financial costs for the vast majority of the per-requisites. However, in 2016 I'm not so proud of passing my renewal. On the other hand, this could simply be me getting old, cynical and grumpy but I'm pretty sure I won't be renewing my GSE in 4 years, we shall see! Maybe if it was to be the practical 2-day lab again, I'd fail :)

I then see articles like this and this, which are two of many pushing people to certifications :( Two of the smartest and hardest-working people I've ever worked with that regularly kick my ass daily don't have degrees. If I ever started a company, they're the first people I'd hire again :)

Here's the complete list of InfoSec certifications, which is pretty incredible and many I've never heard of (it must have been so painful to compile this list). Seeing so many certifications is little scary for me but really, it's just indicative of the bloat in the InfoSec industry in general.

This is either a rant or random thinking aloud but I do feel better after getting it out of my head....

Subsequent Edit:

Some links to "Getting into InfoSec" as it came up in Twitter convos:

Tuesday, 3 May 2016

Being the Bug in Bug Bounty :: Fail & Tell

Late in December 2015, I sent the email below to all of "Engineering" across Riot Games. I want to share this externally because it's core to the security culture that we want to build in Riot, i.e. one of accountability and responsibility, where we aren't afraid to talk about our screw-ups.

Hey Folks,

So as most of you know, I'm a bit of a perfectionist with high standards :)

Well recently, I screwed up and didn't come close to meeting my own expectations.

What did I do?

Well, when testing (locally) Netflix's Security Monkey in 2014, I copied over some aws-related scripts I was using and found useful to a local directory on  my work laptop, where my Github repo was stored. I also mistakenly copied over a flask configuration file ( from the local version of Security Monkey that I was testing at the time.

To compound this mistake, prior to committing files and pushing to my Github repo, I performed a "git add" on the sub-directory (aws), not the specific files I'd added. As a result, was added to the changes to the repo and consequently, committed and pushed up to my personal Github.

What Happened Next?

Nothing for a year until I got a random late-night text from a colleague, nothing unusual there :)
  • Bug Bounty Ticket - xxxx
  • Internal Vulnerability Ticket - xxxx
However, this text was advised me to check our Bug Bounty system. The shear panic and "shit, shit, shit" went around in my head :-/ So unsurprisingly, I immediately logged on to  read the bug report.

What was the impact? 

Fortunately, close to nothing:
  • the file was a test file with no production credentials of any sort
  • we have publicly committed to the Security Monkey repo
  • we have publicly spoke about using Security Monkey
  • sheer embarrassment and annoyance on my behalf
What's the point of this email?
  • I'd like to apologise to you guys for screwing up.
  • To let you know that I've removed the personal repo and modified the controls on any public github repo that's been cloned to my system
  • This researcher was actually very cool (despite initially claiming it was an awesome vulnerability and significant information disclosure).
    • Validation is important on all (vulnerability/bug) reports coming in through our Bug Bounty or any other medium.
  • I feel like I got lucky
  • Run "git status" before "git push origin master"
  • .gitignore doesn't always save you :(
  • I got great support from the InfoSec team on this, which meant a lot and reminded me that we're a team at all times (success & failures).
  • Hopefully others will realise that it's not the end of the world to make mistakes and it's important that we don't try to hide them but share them and learn from them.
If you've any questions or feedback for me, please let me know.

This episode was a huge learning experience for me and I believe I can only improve from it (whilst, yeah, I was lucky). It was ultimately a very humbling experience, a nice reminder that we in #InfoSec make mistakes, that we should not be advocating for security from the our perches up high, feeling perfect.

We have a "no asshole" rule and I believe this is one reason (of many) why I had a pretty awesome and very supportive response from many across Riot, a nice reminder that it's often better to talk about your failures than your successes.

Saturday, 2 April 2016

Reading List

At BruCon last year, one of the audience came up afterwards and started asking about the books that I read. He was fascinated by some of the books that I'd mentioned in my talk (massive surprise to me) so I promised to publish a list on my blog.

Granted it has taken me around 7 months but I've started to compile such a list on Good Reads, now I just have to find out who it was :-/

Saturday, 29 August 2015

Team Building @ Riot

Back in early July 2015, on a uniquely dry Thursday evening, the Riot InfoSec team ran a small meet-up at our new Dublin office as I previously mentioned here.

The goal of the event was to engage with the local security community, several of whom are also huge League players as I noticed by negative KDA :'( My main memories of the gaming that night being constantly head-butted away by 60 minute Alistair (also spamming his heal) and an extremely fed, axe-wielding and catching Draven, killing at will.

As a team, we felt that the event went well and it seems to have been well received by those who attended. My talk was about the lessons that I have learned from hiring a team into Riot Engineering, which is the first time I've ever had the opportunity to build a team from scratch. It's not only be incredibly educational and exciting, but also humbling as I've had phenomenal support from so many folk in both the Dublin and LA :)

Given there's probably quite a bit of context missing from my talk (if you only look at the slides here) followed the format below:

  • Who I am and who Riot Games are
  • Set context on Riot, League of Legends and the security challenges we face
  • Levelling up the team
    • How I went about it
    • What mistakes I made
    • How I learned
    • That "exciting" feeling when you interview a game-changer
    • How we've improved the process and continue to improve through
      • being data-informed
      • conscious of bias
      • training
      • incorporating practical elements
    • Team (alignment)
In addition to the presentations, we also showed off some tools that we've written to help raise that security bar.

Given we are a diverse team, split across multiple countries and timezones, with different backgrounds, alignment is a challenge because we are all ultimately remoties. As a result, we are always thinking of how we can improve in terms of communication, collaboration and yes, alignment. We took the opportunity with the open-evening event to bring over many of our North America-based InfoSec team to

  • hang out for the week
  • deliver some cool awareness and social hacking presentations to the Riot Dublin office
  • build a lego deathstar (yes, a deathstar :) )

Sadly, one of our Darth figures was stolen and taken back to St Louis by a malicious insider :(

All feedback on the slides is greatly appreciated.

Hopefully the next meet-up will be in early 2016. If there's anything in particular you'd like to hear about at the next meet-up, let us know (@markofu or @davidrook).

If you want to help level the team up, we're still hiring (multiple locations), so please reach out!!!

Monday, 8 June 2015

Riot Games Dublin InfoSec Meetup

On July 9th, the Riot InfoSec team will be hosting a security event in our new office, which will include:

  • talks from Rioters
  • the chance for you to see and play with some of our security tools
  • time to play some games with us!
Riot is dedicated to positively engaging with the industry and its community, and the Riot InfoSec team are psyched to further that mission in Dublin - we want to share and to learn. If you work in Riot or engage with Rioters, you may hear the term “default to trust” and this is a huge part of our culture. It’s not just “talk”, it’s something that I see on a daily basis and to me, this is very special. I've linked two posts that touch on this aspect below -

As a result, we (in Riot InfoSec) wanted to “walk the walk” also and share some of our failings and successes, and learnings while building some relationships in the community.

The event will have three security talks from David Rook, myself and Peter Tillotson. We hope that these talks will give you an insight into -

  • how we approach application security
  • what we’ve learnt from trying to hire into InfoSec
  • our security-focused big data analysis
We would also like to give you the chance to check out some of our security tools we use. We will have several demo booths setup so you can play with these tools and learn more about the unique challenges we face at Riot.

Finally, did I mention that you can hang around and play games :D

Thursday, 12 February 2015

Applying for AppSec Engineer @ Riot

It's no surprise that attackers will use recruiters as targets for a compromise and like many companies, we've seen the usual applications with XSS and macros. Today I received something slightly different which I figured was worth sharing -

As you'd expect, the candidate details are fabricated so we can't progress :(

P.S. We are actually hiring in Dublin, Istanbul, St Louis and LA for security engineers:) 

Saturday, 24 January 2015

Socialising Security @ Riot

Socialising Security @ Riot

Quick Link: Presentation here.


In late November last year, I had the honour of following the illustrious David Rook (ex-SecurityNinja :) ) in the Owasp Dublin Chapter meeting (thanks Owen & Owasp Ireland). Quite a few people (mostly Chris John Riley) reached out and said:

“The presentation looks cool and I'm jealous of the cool artwork but context, need MOAR context!”

From an OpSec perspective, it's not always possible to include all the context when it comes to publicising security presentations, but @Riot, the goal of the InfoSec team is to socialise security within Riot, our players, the gaming community and the security community.


  • Each Rioter is responsible for their own security  
  • Riot has posed very new challenges (for me) -
    • Scale
    • Volume of Incidents (i.e. a successful compromise, a leak, a ddos attack)
    • Open policy to security ( this is the bit that will draw the crowd )
  • We want to be more active in the community in future
  • How do the InfoSec team engage with the players and Rioters
  • How we have leveled-up
  • Some futures :)

Without further delay, some more detailed context -

Who am I 

At this point, I gave a quick introduction to myself, my role within Riot (basically, I’m incredibly lucky to have a very varied InfoSec role, at Riot, with loads of challenges and a lot of fun ). I also provide context to the audience on:

Game of Security


I discussed our three values (Security, Agility, Quality) that we follow in order to move fast but we haven't always nailed the "security" element and to be honest, we are still learning new things there and improving. Riot grew faster than anyone expected and, as a result, quick responses, fixes and upgrades were required to meet that organic growth and fulfil our mission statement - "player focus". These responses resulted in such a solution as presented by Marty and Jonathan at AWS re:Invent where such a level of growth requires innovation and responsiveness but can also result in spaghetti junctions:

With this evolutionary process going on, InfoSec were still very much at the firefighting stage:

This initial section also covered how every single Rioter goes through initial security training with a twist where

  • It is explained to each Rioter that security is their own responsibility (I don’t mean that each Rioter patches their own machine but when working, engaging with players, talking to vendors, developing new application, the Rioter “owns” the security of what he/she is doing)

  • Technical advice and solutions are provided (e.g. how to construct a strong but memorable password, how to store your multiple passwords)

  • A few real-life stories are told and we do this in a passionate, fun way with cool images

Finally, the second section came to a close with a discussion on three different type of incidents that we've experienced -

  • DDoS

    • This has been very well documented in the past few weeks with the DDoS attacks on Xbox and PSN but it's been going on much longer than that. As with everything, it's harder to defend than break but we've made substantial advances in the past year. I talked about -

      • Some of these advances
      • The attacks we experienced (ntp/ssdp/chargen amplification, syn floods, booters, botnets etc)
      • The attack size (the biggest amplification took down multiple upstream Tier 1's and was well over 400gb/s)
      • Future plans, though I didn't mention this, which is currently in the works and hugely exciting :)
  • Leakers

    • I mentioned some recent leaks and explained how such unfortunate leaks player-focused and whilst it may get the leaker huge kudos on Reddit, he/she are eventually caught and it does no-one any good, most certainly not our players, as it will spoil the launch event and surprises.
    • One attendee asked for details on how we have caught leakers and I explained what usually leads to the leaker's downfall, i.e. he/she, more often than not, has very poor OpSec and leaks without taking precaution or has to "tell" someone.

  • Infrastructure Hack

    • In mid 2013, Riot was hacked and lost a password database.

      • I explained how unfortunate this event was, how it hurt our players and how tough and draining it was for the InfoSec team itself.

  • I discussed how the team performed Incident Response (primarily identification, containment, recovery and eradication). Additionally, I explained how we split tasks between Subject Matter Experts in terms of analysis, forensics, investigation, communications and review.

I tried to share our lessons learned from these type of incidents because there was clearly a lot. For example:

  • Our Incident Response (IR) processes were not clear, either to new Rioters or outside of the InfoSec team.

  • IR in a geographically-diverse team is hard, especially with over 90% of the team in one place. Some big reasons are -

    • Communication
    • Burn-out

Hiring obviously helps fix these issues but internal processes must be re-edited, communicated, clarified and most importantly agreed upon first.

  • There were technical gaps that hindered our Incident Response. These gaps subsequently formed a large part of immediate roadmap.

  • We needed to level-up our awareness program but more importantly, we needed to engage much more closely with the broader Riot Engineering discipline. There were valuable lessons to be learned from the incident but the message could not be delivered top-down with a hammer, it had to be a two-way process with full alignment.

Level Up

In this section, I talked about some of the things that the Riot InfoSec team have done to "level up" security within Riot over the past year or so. I focused primarily on three areas -

  • RFCs - we have our internal RFC system that (a little like the well-known public version). This process/system has been driving by our Engineering dudes and it's incredibly valuable. You can think of it like the building-blocks to how we should do things and it works on adoption through a hierarchical scope system.

    • We haven’t been doing this for long but we’ve found it very it to be very effective and the RFC that we end up with (after discussions and modifications) is always significantly better than that “perfect” version we first publish ;-)

    • A RFC should also be iterative. For example, if SHA1 was originally the recommended hashing algorithm but then it becomes vulnerable to collisions, a new algorithm should be recommended and the RFC updated with those adoptees updating anything affected under their scope.

    • If you don't know me, let's just say, I'm clearly not a programmer. This system has very much been a change for me as I'm traditionally from a non-agile, operational type environment but I think it is amazing and most importantly, we as a team have learned so much on so many levels (e.g. we now know more about how our developers work and what's important to them). You may not get perfect security but 70% security with 100% alignment is better than 100% security with 0% alignment. As Zane Lackey once said -

Don't be a jerk!

    • To help understand and learn how our Engineers work, I personally have reached out to a lot of folk internally within Riot - Product Owners, Product Managers, Engineers, Engineering Managers. As a non-programmer, this has been incredibly helpful and I have been able to put myself in their shoes. We similar engage with other areas of Riot to increase our understanding of what a Rioter needs to do their job and enhance the player experience.

  • Visibility - this is something others have also spoke more eloquently than me on. Our philosophy here is similar to those espoused by Zane Lackey (here and here) and frequently by Richard Bejtlich, here and here. The goal here is not to prevent attacks but to detect and respond to attacks, we constantly want to be working on our networking knowledge and understanding so we can detect and respond as quickly as possible.

  • Red Team - we've moved from thinking of vulnerability penetration tests to additionally conducting proper attack simulations (without limits) where the attacker has already penetrated the perimeter (sadly we have seen that this is possible) and giving the attacker specific targets (quite often, our partners have better ideas so we rip up our attack simulation plan and goals, instead going with theirs). Penetration tests and vulnerability scans still have a place obviously but there's lots of terminology confusion in the industry on this :)

Bonus Levels

Security Week

Shortly after I gave the presentation at Owasp, we ran our first "security week" event. The main goal here was again to "socialise security" in that we made security fun for Rioters and provided some interesting challenges. We had internal blogs, presentations (hacking the audience and gaming-specific risks) and it was a success, though we had some failings and this gives us some awesome things to work on in 2015.

Standards & Verify 

In Riot, we "default to trust" and yes, there can be security issues with that but this philosophy is key to Riot culture. Our goal is to continue driving the RFC process, iterating on what we learn, checking adoption and verifying that there aren't issues and undue or misunderstood risks.

Tech & Tools

  • Continue to build solutions and tools. 
  • Contribute to open-source solutions such as Security Monkey (from Netflix) and release our own. 
  • Research
  • Work with other Engineering teams on innovative solutions such as this.


  • More active participation in the community
  • Conference Talks
  • Bug Bounty
    • This has been an interesting project as we have moved from non-public to public. We made a lot of mistakes and we have had to ramp up quickly. 
    • Check out some of the presentations from Katie Moussouris, from HackerOne, as she has a lot of interesting bug bounty learnings and stories. 
    • Hopefully some day soon, we can share more. To again quote Zane Lackey - 

"You're on the Internet, so you get a free pen test already, now you get the report!"


Obligatory mention, right :)

Seriously though, we're looking for awesome dudes or dudettes to help us so please see here. Help us build!!!

My personal philosophy is to focus on having/retaining/hiring awesome team members over products, solutions or anything-else (possibly why I'm losing some of my tech skills, I want the team to grow and be autonomous otherwise we can't scale or be effective and folk won't stick around). 


Hopefully this extra context was helpful to someone other than Chris ;-)

I think most of what I've discussed is self-explanatory - keep learning, communicating, iterating and reminding yourself that you don't know everything. Security is the responsibility of everyone in your company, not just you (as the security person) so you're not alone :)

I've probably forgotten someone, some thing or some event so apologies on that - let me know if I have (if you read this) and I'll modify the post asap.



On the developer perspective, there are quite a few non-Riot presentations that have helped me understand how developers work, how developers think about “code” and how we can build security into their processes and life-cycles. Here’s a small sample -

  • Spotify Engineering Culture - here and here 

Oh yeah, Rookie's presentation from the same Owasp Chapter meeting can be found here.