Socialising Security @ Riot

Socialising Security @ Riot

Quick Link: Presentation here.

Background

In late November last year, I had the honour of following the illustrious David Rook (ex-SecurityNinja :) ) in the Owasp Dublin Chapter meeting (thanks Owen & Owasp Ireland). Quite a few people (mostly Chris John Riley) reached out and said:

“The presentation looks cool and I'm jealous of the cool artwork but context, need MOAR context!”

From an OpSec perspective, it's not always possible to include all the context when it comes to publicising security presentations, but @Riot, the goal of the InfoSec team is to socialise security within Riot, our players, the gaming community and the security community.

Tl;dr

• Each Rioter is responsible for their own security  
• Riot has posed very new challenges (for me) -
• Scale
• Volume of Incidents (i.e. a successful compromise, a leak, a ddos attack)
• Open policy to security ( this is the bit that will draw the crowd )
• We want to be more active in the community in future
• How do the InfoSec team engage with the players and Rioters
• How we have leveled-up
• Some futures :)

Without further delay, some more detailed context -

Who am I 

At this point, I gave a quick introduction to myself, my role within Riot (basically, I’m incredibly lucky to have a very varied InfoSec role, at Riot, with loads of challenges and a lot of fun ). I also provide context to the audience on:

• Riot Games
• The Riot Manifesto and Mission Statement [one of Riot's founders explained the Riot principles last year on Reddit]
• How it affects the Riot InfoSec team
• The Scale at which Riot operates 
• Which makes Security Engineering hard, but also easy
• Yes, continuous deployment can be good for security but for someone more eloquent, go listen to Zane Lackey talk about it. 
•  My #LOL career, tl;dr "I'm not very good :("

Game of Security

Philosophies

I discussed our three values (Security, Agility, Quality) that we follow in order to move fast but we haven't always nailed the "security" element and to be honest, we are still learning new things there and improving. Riot grew faster than anyone expected and, as a result, quick responses, fixes and upgrades were required to meet that organic growth and fulfil our mission statement - "player focus". These responses resulted in such a solution as presented by Marty and Jonathan at AWS re:Invent where such a level of growth requires innovation and responsiveness but can also result in spaghetti junctions:

With this evolutionary process going on, InfoSec were still very much at the firefighting stage:

This initial section also covered how every single Rioter goes through initial security training with a twist where

• It is explained to each Rioter that security is their own responsibility (I don’t mean that each Rioter patches their own machine but when working, engaging with players, talking to vendors, developing new application, the Rioter “owns” the security of what he/she is doing)

• Technical advice and solutions are provided (e.g. how to construct a strong but memorable password, how to store your multiple passwords)

• A few real-life stories are told and we do this in a passionate, fun way with cool images

Finally, the second section came to a close with a discussion on three different type of incidents that we've experienced -

• DDoS

• This has been very well documented in the past few weeks with the DDoS attacks on Xbox and PSN but it's been going on much longer than that. As with everything, it's harder to defend than break but we've made substantial advances in the past year. I talked about -

• Some of these advances
• The attacks we experienced (ntp/ssdp/chargen amplification, syn floods, booters, botnets etc)
• The attack size (the biggest amplification took down multiple upstream Tier 1's and was well over 400gb/s)
• Future plans, though I didn't mention this, which is currently in the works and hugely exciting :)

• Leakers

• I mentioned some recent leaks and explained how such unfortunate leaks player-focused and whilst it may get the leaker huge kudos on Reddit, he/she are eventually caught and it does no-one any good, most certainly not our players, as it will spoil the launch event and surprises.
• One attendee asked for details on how we have caught leakers and I explained what usually leads to the leaker's downfall, i.e. he/she, more often than not, has very poor OpSec and leaks without taking precaution or has to "tell" someone.

• Infrastructure Hack

• In mid 2013, Riot was hacked and lost a password database.

• I explained how unfortunate this event was, how it hurt our players and how tough and draining it was for the InfoSec team itself.

• I discussed how the team performed Incident Response (primarily identification, containment, recovery and eradication). Additionally, I explained how we split tasks between Subject Matter Experts in terms of analysis, forensics, investigation, communications and review.

I tried to share our lessons learned from these type of incidents because there was clearly a lot. For example:

• Our Incident Response (IR) processes were not clear, either to new Rioters or outside of the InfoSec team.

• IR in a geographically-diverse team is hard, especially with over 90% of the team in one place. Some big reasons are -

• Communication
• Burn-out

Hiring obviously helps fix these issues but internal processes must be re-edited, communicated, clarified and most importantly agreed upon first.

• There were technical gaps that hindered our Incident Response. These gaps subsequently formed a large part of immediate roadmap.

• We needed to level-up our awareness program but more importantly, we needed to engage much more closely with the broader Riot Engineering discipline. There were valuable lessons to be learned from the incident but the message could not be delivered top-down with a hammer, it had to be a two-way process with full alignment.

Level Up

In this section, I talked about some of the things that the Riot InfoSec team have done to "level up" security within Riot over the past year or so. I focused primarily on three areas -

• RFCs - we have our internal RFC system that (a little like the well-known public version). This process/system has been driving by our Engineering dudes and it's incredibly valuable. You can think of it like the building-blocks to how we should do things and it works on adoption through a hierarchical scope system.

• We haven’t been doing this for long but we’ve found it very it to be very effective and the RFC that we end up with (after discussions and modifications) is always significantly better than that “perfect” version we first publish ;-)

• A RFC should also be iterative. For example, if SHA1 was originally the recommended hashing algorithm but then it becomes vulnerable to collisions, a new algorithm should be recommended and the RFC updated with those adoptees updating anything affected under their scope.

• If you don't know me, let's just say, I'm clearly not a programmer. This system has very much been a change for me as I'm traditionally from a non-agile, operational type environment but I think it is amazing and most importantly, we as a team have learned so much on so many levels (e.g. we now know more about how our developers work and what's important to them). You may not get perfect security but 70% security with 100% alignment is better than 100% security with 0% alignment. As Zane Lackey once said -

Don't be a jerk!

• To help understand and learn how our Engineers work, I personally have reached out to a lot of folk internally within Riot - Product Owners, Product Managers, Engineers, Engineering Managers. As a non-programmer, this has been incredibly helpful and I have been able to put myself in their shoes. We similar engage with other areas of Riot to increase our understanding of what a Rioter needs to do their job and enhance the player experience.

• Visibility - this is something others have also spoke more eloquently than me on. Our philosophy here is similar to those espoused by Zane Lackey (here and here) and frequently by Richard Bejtlich, here and here. The goal here is not to prevent attacks but to detect and respond to attacks, we constantly want to be working on our networking knowledge and understanding so we can detect and respond as quickly as possible.

• Red Team - we've moved from thinking of vulnerability penetration tests to additionally conducting proper attack simulations (without limits) where the attacker has already penetrated the perimeter (sadly we have seen that this is possible) and giving the attacker specific targets (quite often, our partners have better ideas so we rip up our attack simulation plan and goals, instead going with theirs). Penetration tests and vulnerability scans still have a place obviously but there's lots of terminology confusion in the industry on this :)

Bonus Levels

Security Week

Shortly after I gave the presentation at Owasp, we ran our first "security week" event. The main goal here was again to "socialise security" in that we made security fun for Rioters and provided some interesting challenges. We had internal blogs, presentations (hacking the audience and gaming-specific risks) and it was a success, though we had some failings and this gives us some awesome things to work on in 2015.

Standards & Verify 

In Riot, we "default to trust" and yes, there can be security issues with that but this philosophy is key to Riot culture. Our goal is to continue driving the RFC process, iterating on what we learn, checking adoption and verifying that there aren't issues and undue or misunderstood risks.

Tech & Tools

• Continue to build solutions and tools. 
• Contribute to open-source solutions such as Security Monkey (from Netflix) and release our own. 
• Research
• Work with other Engineering teams on innovative solutions such as this.

Community

• More active participation in the community
• Conference Talks
• Bug Bounty
• This has been an interesting project as we have moved from non-public to public. We made a lot of mistakes and we have had to ramp up quickly. 
• Check out some of the presentations from Katie Moussouris, from HackerOne, as she has a lot of interesting bug bounty learnings and stories. 
• Hopefully some day soon, we can share more. To again quote Zane Lackey - 

"You're on the Internet, so you get a free pen test already, now you get the report!"

Hiring

Obligatory mention, right :)

Seriously though, we're looking for awesome dudes or dudettes to help us so please see here. Help us build!!!

My personal philosophy is to focus on having/retaining/hiring awesome team members over products, solutions or anything-else (possibly why I'm losing some of my tech skills, I want the team to grow and be autonomous otherwise we can't scale or be effective and folk won't stick around). 

Conclusion

Hopefully this extra context was helpful to someone other than Chris ;-)

I think most of what I've discussed is self-explanatory - keep learning, communicating, iterating and reminding yourself that you don't know everything. Security is the responsibility of everyone in your company, not just you (as the security person) so you're not alone :)

I've probably forgotten someone, some thing or some event so apologies on that - let me know if I have (if you read this) and I'll modify the post asap.

########

P.S. A FYI 

On the developer perspective, there are quite a few non-Riot presentations that have helped me understand how developers work, how developers think about “code” and how we can build security into their processes and life-cycles. Here’s a small sample -

• Spotify Engineering Culture - here and here 

• Etsy - Engineering Rotation & Day One Deploy 

• Risk IO - Security & DevOps 

• And obviously, The Phoenix Project

Oh yeah, Rookie's presentation from the same Owasp Chapter meeting can be found here.