Marcus Crane, Senior Site Reliability Engineer at Vend by Lightspeed

Not knowing what he wanted to do when finishing high school and thinking he has missed the boat on becoming a programmer, Marcus Crane, Senior Site Reliability Engineer at Vend by Lightspeed eventually found his way into the industry by continuing to expand his education and persistently tinkering with his own projects.

We caught up with Marcus to learn more about his career journey, what a Site Reliability Engineer actually does day-to-day and some great advice that has helped him along the way.

We loved hearing such an honest account of the struggle that many people can identify with - picking a career path to pursue. Marcus also has his own blog that details things he is currently exploring for anyone curious. Thanks for sharing your story, Marcus.

Firstly, how would you explain to a five year old what it is you do?

Here at Lightspeed, we help people run their shops.

Our products help store owners with everything they need to do each day like selling things to people, keeping a list of all the items that are available to buy, counting up how much money was made from sales and more.

While there are lots of people who work on those products as their day job, my team and I make sure everything is running at its best.

That might mean regularly oiling the machinery, helping with computer accidents and more.

And for the adults, what does that translate to in regards to your day-to-day?

The general gist of site reliability engineering (SRE) is "What happens if you take a software engineer and have them work on infrastructure/operational problems".

With that in mind, we generally work across a range of areas from assisting product teams in keeping their systems stable and performant, aiding in incident response and generally reducing the amount of manual work or "toil" required for tasks that can be automated.

To provide a concrete example, let's take a closer look at that last point through the lens of a real reliability issue we used to have: "canarying".

For the unfamiliar, "canarying" here refers to the concept of testing software by deploying it such that only a small percentage of users reach it. If something goes wrong, it can be quickly disabled and a bad software deployment can be avoided.

The vast majority of this process is automated internally but until recently, we had a handful of services running on older style load balancers. This meant that our internal tooling wasn't compatible with them.

As a result, any developer wanting to perform a canary would need to manually configure a load balancer to redirect some portion of traffic towards the canary deployment.

The worst case scenario here is that if it wasn't done correctly, you could potentially send some traffic into a black hole if say; you turned off the canary deployment but didn't remove the load balancer configuration.

Changes happen all the time with modern SaaS products behind the scenes but human error is unavoidable. By investing in automating highly repeatable processes, we can ensure a greater level of stability overall which is our ultimate goal.

A key thing to note here is that in a traditional ops model, you might ultimately perform various tasks over and over without the freedom or flexibility to deal with them once and for all.

With the SRE approach, we have the software background to take these sorts of tasks and automate them such that we can avoid human error and also free up time that developers can better invest back into their products.

What are some of the common misconceptions about working as a Site Reliability Engineer

Perhaps that we're solely operations focused?

I'll speak for myself here but I've found myself all over the place (which I quite enjoy) from the edge of SPAs right back to the bowels of infrastructure so for better or worse, I like to think I'm mostly a jack of all trades.

I think there are definitely companies who simply rename their existing operations team to SRE and call it a day which is essentially what you aren't meant to do.

Reading some of the job descriptions that get sent around with SRE in the title is a bit confusing because they don't appear to reflect what an SRE does at all.

Was working in tech something you dreamed about doing as a kid, if not what was?

I remember that we got our first computer when I was about ten. My dad is a builder by trade so we bought one to make it easier to file his tax returns or something along those lines. It probably just looked cool too.

It came in a huge box and I was fascinated with how it worked. How do icons end up on your desktop? Does someone just draw images of every possible variation, like Icon 1 in the top left corner, Icon 1 in the top right corner and so on? I didn't understand as a kid but it really baffled me.

Despite that, I never saw myself being in the industry. I didn't even really know what that meant or what programs were. I wasn't one of those kids who learned programming at a young age or anything like that. The school I went to had basically nothing but Microsoft Word and Powerpoint when it came to computing topics.

I was still fascinated in my teenage years but most of my interest was through video games. I would read about game engines and how the game development process worked through watching interviews. I didn't want to be a game developer mind you, I just found it all interesting stuff.

If I'm honest, I was pretty well resigned to the idea that I would never be a programmer and that I had "missed the boat".

I had the wrong idea that because I hadn't been introduced to programming at a young age or I wasn't some math genius that I had dropped the ball forever.

Tell us a little bit more about your career journey and ultimately about how you ended up working at Vend?

As mentioned, I was fascinated with technology and computers but I never really enjoyed programming.

I had tried a few online courses but I didn't really understand any of the concepts. I also didn't know anyone that could break things down into layman's terms for me.

In my last year of school, I was pretty bored so I ended up skipping statistics to bash out HTML on my laptop. I had a small game review site that I worked on as writing practice, but also an excuse to fiddle with Wordpress. No one read it mind you, and I didn't really care about having an audience. It was just an excuse to produce something.

Ultimately my grades were pretty bad towards the end of the year and while I had applied to a Computer Science course, I was rejected for not having enough Math credits.

I had no idea what I wanted to do so the first year after school ended, I just stayed at home until my parents told me I should get a job in the meantime.

I started working at a local supermarket where I learned a lot but it was still pretty mind numbing. I remember there used to be this giant chart on the wall which was like 6 pieces of A4 paper stapled together with all of the incoming delivery dates.

I took a crack at making a small PHP website that lets you search by retailer. I showed the store owners who... didn't get it. I think I shelved what little remained of my programming ambitions after that.

After three years of retail, I was pretty depressed and admittedly not in a great place mentally. Something had to change so on the advice of a counsellor, I looked at what was out there and applied to Enspiral Dev Academy which only had the Wellington branch at the time.

With a goal for once, and months of saving repurposed that was meant to be for a car, I moved to Wellington where I attended EDA and came out the other side. It took me about another year to find my first gig and that first year was pretty miserable. I mostly spent it cooped up in my room churning out side projects and burning through the last of my savings.

To cut a long story short, I did a few odds and ends in Wellington before ultimately getting accepted to the Xero Graduate Program, despite not technically being a real graduate. I joined the SRE team in Auckland as their first graduate and from there, the rest is history really.

After about three years at Xero, I worked at Mercedes-Benz for a bit. Funnily enough, Vend initially reached out to me because unbeknownst to me, an old colleague from Xero had joined and referred me as a fit for the SRE team. As I learned more, it all sounded like great stuff and it genuinely ended up being a great match!

What is the best piece of career advice you have ever received?

I often revisit a YouTube upload titled Ira Glass on Storytelling 3.

The gist of it is that Ira Glass, host of This American Life, recounts when he first started out and he talks about the idea of taste.

When you start out, you might feel that what you're producing isn't very good. To be able to identify that feeling means that your taste, what you consider "good", exceeds what you're able to create at present.

The key part is that your taste is good enough to guide you forward, and as you produce more and more over a span of years, the gap between your taste and your abilities starts to close.

Off the back of that idea, I also think that the idea of being passionate can be backwards in terms of cause and effect. When you start out, it's hard to be passionate about a skill if you're not very good.

In my personal experience, I've found that when the gap between skill and taste starts to close, then I get more passionate because I can actually bring ideas to life that I wasn't able to before.

The hardest part of course is getting started.

What do you love about working at Vend?

I guess the proper word would be "pragmatic" but really; people don't care about technology past a certain point and that's a good thing.

As a technologist, it can be tempting to get excited about a particular technology to the point that you're just kind of pushing something for the sake of it and not because it has any value for the business.

At the end of the day, someone running a store doesn't care what's under the hood, just that it works and gets out of their way.

We also have a pretty good balance with experimenting. Lightspeed has a global hackathon every quarter (with prizes up for grabs) and development teams often have a cooldown period to pause and potentially look into new tooling.

Some of the more interesting ideas get converted into RFC, or Requests for Comments, internally. They're basically a README document that details a proposal and if it gets accepted, it can become a new standard for how we do things internally.

I probably can't speak much about them but there are some really cool proposals I've seen floating around from some smart people.

Lastly, Vend continues to grow and evolve, what are the key traits and characteristics of people that will be well placed to work there?

Now that we're part of the Lightspeed family, Vend will soon be home to many more customers and with that comes the task of scaling up to meet increasing demand, both in terms of people and infrastructure.

No single person has all the answers, and there are plenty of unknowns we'll face as a team so being comfortable with not knowing things is a very good trait!