Saturday, 2 April 2016

Reading List

At BruCon last year, one of the audience came up afterwards and started asking about the books that I read. He was fascinated by some of the books that I'd mentioned in my talk (massive surprise to me) so I promised to publish a list on my blog.

Granted it has taken me around 7 months but I've started to compile such a list on Good Reads, now I just have to find out who it was :-/

Saturday, 29 August 2015

Team Building @ Riot

Back in early July 2015, on a uniquely dry Thursday evening, the Riot InfoSec team ran a small meet-up at our new Dublin office as I previously mentioned here.



The goal of the event was to engage with the local security community, several of whom are also huge League players as I noticed by negative KDA :'( My main memories of the gaming that night being constantly head-butted away by 60 minute Alistair (also spamming his heal) and an extremely fed, axe-wielding and catching Draven, killing at will.

As a team, we felt that the event went well and it seems to have been well received by those who attended. My talk was about the lessons that I have learned from hiring a team into Riot Engineering, which is the first time I've ever had the opportunity to build a team from scratch. It's not only be incredibly educational and exciting, but also humbling as I've had phenomenal support from so many folk in both the Dublin and LA :)



Given there's probably quite a bit of context missing from my talk (if you only look at the slides here) followed the format below:

  • Who I am and who Riot Games are
  • Set context on Riot, League of Legends and the security challenges we face
  • Levelling up the team
    • How I went about it
    • What mistakes I made
    • How I learned
    • That "exciting" feeling when you interview a game-changer
    • How we've improved the process and continue to improve through
      • being data-informed
      • conscious of bias
      • training
      • incorporating practical elements
    • Team (alignment)
In addition to the presentations, we also showed off some tools that we've written to help raise that security bar.



Given we are a diverse team, split across multiple countries and timezones, with different backgrounds, alignment is a challenge because we are all ultimately remoties. As a result, we are always thinking of how we can improve in terms of communication, collaboration and yes, alignment. We took the opportunity with the open-evening event to bring over many of our North America-based InfoSec team to

  • hang out for the week
  • deliver some cool awareness and social hacking presentations to the Riot Dublin office
  • build a lego deathstar (yes, a deathstar :) )

Sadly, one of our Darth figures was stolen and taken back to St Louis by a malicious insider :(




All feedback on the slides is greatly appreciated.

Hopefully the next meet-up will be in early 2016. If there's anything in particular you'd like to hear about at the next meet-up, let us know (@markofu or @davidrook).

If you want to help level the team up, we're still hiring (multiple locations), so please reach out!!!

Monday, 8 June 2015

Riot Games Dublin InfoSec Meetup

On July 9th, the Riot InfoSec team will be hosting a security event in our new office, which will include:



  • talks from Rioters
  • the chance for you to see and play with some of our security tools
  • time to play some games with us!
Riot is dedicated to positively engaging with the industry and its community, and the Riot InfoSec team are psyched to further that mission in Dublin - we want to share and to learn. If you work in Riot or engage with Rioters, you may hear the term “default to trust” and this is a huge part of our culture. It’s not just “talk”, it’s something that I see on a daily basis and to me, this is very special. I've linked two posts that touch on this aspect below -




As a result, we (in Riot InfoSec) wanted to “walk the walk” also and share some of our failings and successes, and learnings while building some relationships in the community.


The event will have three security talks from David Rook, myself and Peter Tillotson. We hope that these talks will give you an insight into -


  • how we approach application security
  • what we’ve learnt from trying to hire into InfoSec
  • our security-focused big data analysis
We would also like to give you the chance to check out some of our security tools we use. We will have several demo booths setup so you can play with these tools and learn more about the unique challenges we face at Riot.


Finally, did I mention that you can hang around and play games :D

Thursday, 12 February 2015

Applying for AppSec Engineer @ Riot

It's no surprise that attackers will use recruiters as targets for a compromise and like many companies, we've seen the usual applications with XSS and macros. Today I received something slightly different which I figured was worth sharing -

As you'd expect, the candidate details are fabricated so we can't progress :(

P.S. We are actually hiring in Dublin, Istanbul, St Louis and LA for security engineers:) 

Saturday, 24 January 2015

Socialising Security @ Riot

Socialising Security @ Riot


Quick Link: Presentation here.

Background

In late November last year, I had the honour of following the illustrious David Rook (ex-SecurityNinja :) ) in the Owasp Dublin Chapter meeting (thanks Owen & Owasp Ireland). Quite a few people (mostly Chris John Riley) reached out and said:


“The presentation looks cool and I'm jealous of the cool artwork but context, need MOAR context!”


From an OpSec perspective, it's not always possible to include all the context when it comes to publicising security presentations, but @Riot, the goal of the InfoSec team is to socialise security within Riot, our players, the gaming community and the security community.

Tl;dr

  • Each Rioter is responsible for their own security  
  • Riot has posed very new challenges (for me) -
    • Scale
    • Volume of Incidents (i.e. a successful compromise, a leak, a ddos attack)
    • Open policy to security ( this is the bit that will draw the crowd )
  • We want to be more active in the community in future
  • How do the InfoSec team engage with the players and Rioters
  • How we have leveled-up
  • Some futures :)

Without further delay, some more detailed context -

Who am I 

At this point, I gave a quick introduction to myself, my role within Riot (basically, I’m incredibly lucky to have a very varied InfoSec role, at Riot, with loads of challenges and a lot of fun ). I also provide context to the audience on:


Game of Security

Philosophies


I discussed our three values (Security, Agility, Quality) that we follow in order to move fast but we haven't always nailed the "security" element and to be honest, we are still learning new things there and improving. Riot grew faster than anyone expected and, as a result, quick responses, fixes and upgrades were required to meet that organic growth and fulfil our mission statement - "player focus". These responses resulted in such a solution as presented by Marty and Jonathan at AWS re:Invent where such a level of growth requires innovation and responsiveness but can also result in spaghetti junctions:



With this evolutionary process going on, InfoSec were still very much at the firefighting stage:



This initial section also covered how every single Rioter goes through initial security training with a twist where

  • It is explained to each Rioter that security is their own responsibility (I don’t mean that each Rioter patches their own machine but when working, engaging with players, talking to vendors, developing new application, the Rioter “owns” the security of what he/she is doing)

  • Technical advice and solutions are provided (e.g. how to construct a strong but memorable password, how to store your multiple passwords)

  • A few real-life stories are told and we do this in a passionate, fun way with cool images

Finally, the second section came to a close with a discussion on three different type of incidents that we've experienced -

  • DDoS

    • This has been very well documented in the past few weeks with the DDoS attacks on Xbox and PSN but it's been going on much longer than that. As with everything, it's harder to defend than break but we've made substantial advances in the past year. I talked about -

      • Some of these advances
      • The attacks we experienced (ntp/ssdp/chargen amplification, syn floods, booters, botnets etc)
      • The attack size (the biggest amplification took down multiple upstream Tier 1's and was well over 400gb/s)
      • Future plans, though I didn't mention this, which is currently in the works and hugely exciting :)
  • Leakers

    • I mentioned some recent leaks and explained how such unfortunate leaks player-focused and whilst it may get the leaker huge kudos on Reddit, he/she are eventually caught and it does no-one any good, most certainly not our players, as it will spoil the launch event and surprises.
    • One attendee asked for details on how we have caught leakers and I explained what usually leads to the leaker's downfall, i.e. he/she, more often than not, has very poor OpSec and leaks without taking precaution or has to "tell" someone.

  • Infrastructure Hack

    • In mid 2013, Riot was hacked and lost a password database.

      • I explained how unfortunate this event was, how it hurt our players and how tough and draining it was for the InfoSec team itself.


  • I discussed how the team performed Incident Response (primarily identification, containment, recovery and eradication). Additionally, I explained how we split tasks between Subject Matter Experts in terms of analysis, forensics, investigation, communications and review.

I tried to share our lessons learned from these type of incidents because there was clearly a lot. For example:

  • Our Incident Response (IR) processes were not clear, either to new Rioters or outside of the InfoSec team.

  • IR in a geographically-diverse team is hard, especially with over 90% of the team in one place. Some big reasons are -

    • Communication
    • Burn-out

Hiring obviously helps fix these issues but internal processes must be re-edited, communicated, clarified and most importantly agreed upon first.

  • There were technical gaps that hindered our Incident Response. These gaps subsequently formed a large part of immediate roadmap.

  • We needed to level-up our awareness program but more importantly, we needed to engage much more closely with the broader Riot Engineering discipline. There were valuable lessons to be learned from the incident but the message could not be delivered top-down with a hammer, it had to be a two-way process with full alignment.


Level Up


In this section, I talked about some of the things that the Riot InfoSec team have done to "level up" security within Riot over the past year or so. I focused primarily on three areas -

  • RFCs - we have our internal RFC system that (a little like the well-known public version). This process/system has been driving by our Engineering dudes and it's incredibly valuable. You can think of it like the building-blocks to how we should do things and it works on adoption through a hierarchical scope system.

    • We haven’t been doing this for long but we’ve found it very it to be very effective and the RFC that we end up with (after discussions and modifications) is always significantly better than that “perfect” version we first publish ;-)

    • A RFC should also be iterative. For example, if SHA1 was originally the recommended hashing algorithm but then it becomes vulnerable to collisions, a new algorithm should be recommended and the RFC updated with those adoptees updating anything affected under their scope.

    • If you don't know me, let's just say, I'm clearly not a programmer. This system has very much been a change for me as I'm traditionally from a non-agile, operational type environment but I think it is amazing and most importantly, we as a team have learned so much on so many levels (e.g. we now know more about how our developers work and what's important to them). You may not get perfect security but 70% security with 100% alignment is better than 100% security with 0% alignment. As Zane Lackey once said -

Don't be a jerk!

    • To help understand and learn how our Engineers work, I personally have reached out to a lot of folk internally within Riot - Product Owners, Product Managers, Engineers, Engineering Managers. As a non-programmer, this has been incredibly helpful and I have been able to put myself in their shoes. We similar engage with other areas of Riot to increase our understanding of what a Rioter needs to do their job and enhance the player experience.

  • Visibility - this is something others have also spoke more eloquently than me on. Our philosophy here is similar to those espoused by Zane Lackey (here and here) and frequently by Richard Bejtlich, here and here. The goal here is not to prevent attacks but to detect and respond to attacks, we constantly want to be working on our networking knowledge and understanding so we can detect and respond as quickly as possible.

  • Red Team - we've moved from thinking of vulnerability penetration tests to additionally conducting proper attack simulations (without limits) where the attacker has already penetrated the perimeter (sadly we have seen that this is possible) and giving the attacker specific targets (quite often, our partners have better ideas so we rip up our attack simulation plan and goals, instead going with theirs). Penetration tests and vulnerability scans still have a place obviously but there's lots of terminology confusion in the industry on this :)

Bonus Levels

Security Week


Shortly after I gave the presentation at Owasp, we ran our first "security week" event. The main goal here was again to "socialise security" in that we made security fun for Rioters and provided some interesting challenges. We had internal blogs, presentations (hacking the audience and gaming-specific risks) and it was a success, though we had some failings and this gives us some awesome things to work on in 2015.

Standards & Verify 

In Riot, we "default to trust" and yes, there can be security issues with that but this philosophy is key to Riot culture. Our goal is to continue driving the RFC process, iterating on what we learn, checking adoption and verifying that there aren't issues and undue or misunderstood risks.


Tech & Tools


  • Continue to build solutions and tools. 
  • Contribute to open-source solutions such as Security Monkey (from Netflix) and release our own. 
  • Research
  • Work with other Engineering teams on innovative solutions such as this.

Community

  • More active participation in the community
  • Conference Talks
  • Bug Bounty
    • This has been an interesting project as we have moved from non-public to public. We made a lot of mistakes and we have had to ramp up quickly. 
    • Check out some of the presentations from Katie Moussouris, from HackerOne, as she has a lot of interesting bug bounty learnings and stories. 
    • Hopefully some day soon, we can share more. To again quote Zane Lackey - 

"You're on the Internet, so you get a free pen test already, now you get the report!"

Hiring

Obligatory mention, right :)


Seriously though, we're looking for awesome dudes or dudettes to help us so please see here. Help us build!!!



My personal philosophy is to focus on having/retaining/hiring awesome team members over products, solutions or anything-else (possibly why I'm losing some of my tech skills, I want the team to grow and be autonomous otherwise we can't scale or be effective and folk won't stick around). 


Conclusion

Hopefully this extra context was helpful to someone other than Chris ;-)

I think most of what I've discussed is self-explanatory - keep learning, communicating, iterating and reminding yourself that you don't know everything. Security is the responsibility of everyone in your company, not just you (as the security person) so you're not alone :)

I've probably forgotten someone, some thing or some event so apologies on that - let me know if I have (if you read this) and I'll modify the post asap.


########

P.S. A FYI 

On the developer perspective, there are quite a few non-Riot presentations that have helped me understand how developers work, how developers think about “code” and how we can build security into their processes and life-cycles. Here’s a small sample -

  • Spotify Engineering Culture - here and here 

Oh yeah, Rookie's presentation from the same Owasp Chapter meeting can be found here.