Finding a New Infosec Job

I went through some rounds interviewing for Senior Detection Engineering and Incident Response roles with mostly larger or more well known tech companies and thought I would share my thoughts and experience. I ended up accepting an offer somewhere you’ve probably heard of and will start my new role soon. I am not going to name specific companies, but I do want to talk about some processes I went through, things that I liked, things that I didn’t like, and thoughts for anyone facing a similar situation soon. This has turned into a multi-part post that already totals close to 6000 words, so I’ll breaking it up into several different posts.

Who I Am

I understand any number of people can read what I write and think it doesn’t apply to them, but I am writing this from my perspective and with my background, not intending to speak for anyone else. To get a bit more personal than I usually am on this blog, I am a hispanic male, came from a middle class family, my degree is not in tech or computer science but I did graduate from college paid for largely by loans, scholarships and jobs I had with some help (20% or so? I’m guessing) from my parents, I worked my way up from a help desk (see this gist for more background info) into security, I have tech certifications either previous employers or a previous employer paid for, some I bought myself, taught myself and learned from others cloud security, AWS, python, a bunch of other things, had a series of lucky breaks, and plenty of other details.

None of this means my experience will or will not necessarily apply to you, but I do understand I have certain privileges and statistics working for and against me throughout my career and that means everyone should take my story and advice with a grain of salt. I won’t be giving you any secrets to landing a dream job or anything, and even though I got an offer in under a month of looking and interviewing, I really want to drive home the point that this was a multi-year process for me and it began way before I had my first call with any recruiter.

Before you even apply

I think your next job begins to come together before you even plan to leave your current one. I don’t take any job intending to leave, but I knew I wanted to keep learning and growing if my new role was outside or inside the company, I needed to gain and add new skills and experience to get there. I had taken intentional steps at my current role to learn new skills, focus on a subset of security (blue team to incident detection and response and even more narrowly to detection engineering), and be appealing to my current and future employers. A saying in sales is “Find a need and fill it” and I think that’s what I did with my focus on detection engineering.

Detection Engineering and Incident Response as a focus seems particularly spicy right now and big places you might want to work for are hiring people into those roles at very competitive salaries. I got lucky I think because I had a very passionate interest into something that is suddenly in demand and requires quite a broad skillset. At least at the role I’m leaving, it combined coding and scripting, infrastructure management and automation, incident response, incident detection, threat intelligence, and a few other skills. This is an extremely broad skill set that could easily comprise several roles. I am not claiming to be an expert in all of these, but this article, Introduction to DFIR really describes how I approached what skills to gain and develop.

My Work History In This Job

I joined my previous company about 4 years ago and there was not a dedicated detection engineering focused role. My focus was on incident response, but we tended to respond to incidents we detected. Building and maintaining detections to support that seemed critical and something we needed someone who was solely responsible for it on the team. I thought that if you had someone who did it as part of their job, there was too much of a risk of it becoming something they did as time allowed instead of as their primary responsibility. I think this work is important and needs someone dedicated to it, but its easy to put it aside to handle oncoming incidents.

I saw a need for a dedicated role supporting incident response efforts by creating custom detection content, onboarding and tuning existing “out of the box” content, and automating incident response processes. Doing this as a third job alongside incident response and a brief stint in cloud security I spent the last few years really dedicated to getting better at this while building support internally for a dedicated team. By constantly gaining new skills and developing existing ones for four years, I set myself up to be a candidate other companies would want to hire.

Know where you want to be. Have rough salary targets, think about pain points in your current role, and think about what you wish you had and what you want your day to day work to look like. Decide early what you will and won’t compromise on. For example, I kind of wanted to move away from hands on incident response, but I was also willing to do it full time if it meant I got to get much deeper in the practice (getting heavily invested in memory/disk forensics or specializing some other way).

Persistence is Key

I should also mention I literally did not shut up about this role and how I wanted to do it almost the entire time I was at my last job. I did my assigned roles well, did extra projects, worked with other teams, and all that, but I did not miss an opportunity to bring up to whoever was managing me or managing my manager that I thought a Detection Engineering role was a dedicated role the team needed and I also happened to know the perfect person for the job (me). I do not think my job would have played out for me the way it did if I was not constantly bringing it up and asking for my career to move in that direction.

Ask for what you want and make concrete steps to make it happen. Feel out and understand the people you’re asking to know when to push a topic and when not to push an issue. I’ve had managers in my past where pushing on an issue can put you on their bad side and have them focus negative attention on you or make your job harder. I don’t have good advice there except get out of that situation, whether its being promoted out or moving on. It took 2 years of asking for the role I wanted to be created, so I can’t overstate how important persistence and patience was for me.

In the end, I got the role I wanted but it worked out in a way that I had to leave. I guess that’s fine, I learned a lot along the way, did a lot of really cool work I am proud of, got a bunch of new skills and things to put on my resumes, and when I did interview with a bunch of major companies, I had what felt like were very easy interviews.

Knowing What You Want

I knew a rough salary target and decided that once a job arrived somewhere close to that target, I wasn’t going to try and negotiate or use other offers to increase the one I wanted. While that’s a perfectly valid strategy, it’s not for everyone and you deserve as much as employers are willing to pay you for work that matters to them. Some employers make an offer and won’t negotiate, some make an offer expecting counter offers and negotiations, and I’ve even heard of some companies cancel offers if you try to negotiate. I don’t have all the answers here, but the answer that worked for me was set a list of things I wanted and if somewhere can meet 80% of them, take the offer.

As for the job and company I was looking for, I wanted:

work in my focus area I was passionate about
work I could see myself still doing in 4-5 years (I personally do not like changing jobs and companies)
a company with a big enough footprint to make the work interesting
a company that had supporting teams (one does not engineer detections in a vacuum)
a company willing to budget for required tooling and infrastructure
a manager that seemed to “get” the role of detection engineering and incident response
a manager with hands on experience doing the work and a vision for the team
a security organization that wasn’t fighting its organizational structure
senior/indirect managers that understood the mission of my team
coworkers that I got good vibes about
a very appealing total compensation package
competitive benefits (low premiums or fully covered, 401k matching, etc)
a company I would expect to be there in 10 years
remote (open to compromise here) OR part time in office with local companies (no relocation and no compromise on relocation)

That’s a big long list, there were some other things, but if someone could give me most of that, I knew I would take the offer.

Ready your resume

Your resume will serve 2 big purposes: getting your foot in the door and setting the stage for conversations during interviews. Expect to be able to talk to everything in your resume so obviously don’t lie or exaggerate in it, but do highlight things you want to be able to talk about.

For example, I knew the type of company I wanted to apply at would likely have relatively large Linux and macOS footprints and have a preference for managing their detections and tools as code. So front and center, top line of my current roles responsibilities, I include the skill “Applying engineering principles and best practices to detecting security incidents in a macOS and Linux environment”. There’s 3 or 4 good things in my career history I can very easily talk about, including managing detections and documentation as tested code, experience detecting security incidents, and macOS and Linux.

Maybe those are fluffy words to describe what I did, but that’s good in a way because it prompted interviewers to ask “what do you mean here?” and you get a great chance to talk about yourself. Part of their job in an interview is finding out if you really know what you say you know and part of your job is making sure your strengths shine through, so I think this serves both of you. It, of course, helped that that line accurately described what I did and processes I built.

Again, Find a need and Fill it. Security as an industry is so heavily focused on Windows environments, which, while extremely important and pervasive, leaves a huge gap in defender’s skill sets around macOS and Linux. The kinds of companies I wanted to work for almost always had large linux and macOS deployments and those skills are harder to find in defenders. It’s a skillset that we could use more a lot more of.

When you’re applying

Apply for jobs that match your focus and make sure your resume reflects what you’re focusing on. Also, once the process begins, be willing to ask direct questions about where the team is what challenges they face, especially since more teams have you talk to the hiring manager very early in the process. This whole process is a two way street and it sounds corny but you’re interviewing them too. I didn’t want to work somewhere I wasn’t excited about just because they made me an offer.

I knew I wanted to work for a largish, modern company with a more mature security team that could use my focus. Imagine a security team that doesn’t even have an incident response policy or defined process hiring a detection engineer. That would set the company and the employee up for failure since there’s likely are not infrastructure or processes in place to support such specialized roles. It would likely be more appropriate to hire generalists that specialize with time or intentionally pick T shaped specialists that fill immediate needs and are ready when medium and longer term goals become a reality.

I asked questions early and identified some companies I was interviewing with that I likely would not want to accept an offer from but wanted to continue the process. I continued with them because I didn’t have any screaming red flags about the company or “deal-breakers”, but thought I could learn something later in the process that might help change my opinion from “I probably don’t wanna work there” to “OK, now with a better understanding I can see myself fitting in there well.” Also, maybe if the offer was outlandish and I didn’t want to work there, I could use to negotiate on a role I did want.

Some Lessons Learned

Ultimately, I know I did not do a good job of drawing red lines early. As much as I knew about what I wanted in a company, I didn’t have a list of things that would be deal-breakers even though I had a fuzzy idea of that in my head. In the future, if I am looking for another role, I think I will put more thought into that and I will likely find myself ending more interviews earlier.

I know this outlook is a luxury I had but not everyone has. Not everyone is in a position where they can turn down good, steady job offers, but if you do have that luxury, lean into it and make it so you find yourself in the kind environment that suits you best.

Thats Enough For Now

I hope this give you some insight into how I approached this process. I don’t think my approach or timeline will work for everyone and I know that I have a lot of things working in my favor that many won’t have. If you take anything from this series of posts, it’s that this can be a long and involved process and for some of us it isn’t easy. It’s not easy to start and it’s not easy to follow through. It can be uncomfortable and weird, but if you do it you might come out on the other end much happier.

Finding a New Infosec Job - Part 1 - Looking