13mins

Enabling Developer Velocity

Our philosophy is simple: hire smart people, help them move very fast, and good things come as a result.

Cord is a startup. We started with a strong vision: all sorts of people doing their jobs with less friction, more context, and easy collaboration. Making that abstract vision a reality is hard work. Exactly how much work it is depends. A major factor is how much the the company is optimised for developer velocity. At Cord, we don't mess around about developer velocity. Startups like us are lucky to attract great talent. We want to honour that by making work really, really fast.

Full-stack Onboarding

One of the best ways we boost developer speed is by taking away all the friction of learning the tech stack and processes. At Cord, when someone joins the team, we take time to do hour-long sessions on every layer of our tech stack and operations. We walk them through the client code, the server code, the full system architecture. We do a session on the DB schemas and migration logic. Our deployment and devops setup. We onboard them to the company-level iteration process. We bring them to user interviews. At the end of her first month, an engineer on our team has a seen everything there is to see of our tech. They know the code review process, the continuous integration. They know who to tap for feedback or questions.

It does take time away from each of our team members to do these onboarding sessions, but that time is paid back many times over by every new team member feeling really grounded and informed. It shows them that we trust them and want them to have access to all the knowledge they need to succeed. I've been on a lot of teams where even a year into their jobs, developers still see aspects of their tech stack as mysterious and hazy unknowns. Where there is ambiguity about who owns what or how to solve certain kinds of problems. I can say with high confidence that empowered and informed developers produce way more and way better work, so that's what we ensure we have.

No Agile

Hot Take: Sod retrospectives. Banish sprint planning. Burn down the burn down charts. Leave the manifestos to the revolutionaries. None of the process. Why not? It doesn't match our needs. The world has forgotten, but Agile replaced an even more outdated (but equally flogged) methodology called Waterfall. Agile was a huge innovation in its day. So, too, was the telephone. But now we have the internet.

Despite what various Agile zealots and coaches will vouchsafe to you, Agile doesn't solve all ails. The opposite in fact. Sadly, it's most commonly a symptom of deeper company-level problems which subsequently go unaddressed. Agile is like logging all your food, taking a daily blood sample, carefully regulating your sleep, but actually your leg was broken the whole time.

Maybe it's a flakey product owner who doesn't know what they want. Maybe it's a CTO or VP of Engineering who isn't providing strong technical leadership. Maybe the company has hired weak developers who don't get good work done. Whatever the situation, process-heavy methodologies typically only add drag on developer speed because they create new work without fixing the root causes of the original problem. You go from writing code with unclear requirements to endlessly curating tasks which tediously detail those unclear requirements and then still have to write code against those well-documented unclear requirements.

When you have a problem with your dev team and you try to solve it with more process, you typically only succeed at triggering the Tech Company Doom Loop:

Source unknown, inspired by the Jim Collins (2001) "Tech Company Doom Loop"

We take a different approach. Specifically, we derive priorities from direct user engagement. Then we trust our team to do important work. We rely on solid tech leads and engineering management to give people a nudge if they're veering off piste. Instead of two week sprints, we work in roughly 6-week blocks, setting high level priorities based on our company strategy. From there, we figure out what the rough shape of those priorities looks like as technical execution, then we let our developers do what they do best -- solve the problems. Great people just want to get into the technical work, not wade through 3 page specification documents. Clear priorities and mature technical leadership enable this.

Our team includes stellar people who have track records of delivering projects for Oculus, Google, GoCardless, Facebook, among other great companies. Do we need to chase our people for t-shirt sizing on their tasks? Nope! Should I interrupt a senior engineer mid-flow to let him know he hasn't filed a prioritised ticket that explains his current scope of work? No, no, no. What we do instead is set clear top-level priorities and let our tech leads drive the right outcomes, supporting mid-level and junior developers to rip through the tasks. Because we've got a great team, this works.

One Repository, No Barriers

Every time you prematurely shard out a microservice, the whole internet cries a small tear.

When I talk with engineers at other companies about their work, I often hear about how they juggle multiple codebases. I hear about small teams with 10 different Golang microservices, each with its own repo. I've seen an application frontend split into 3 different Git repositories for... reasons. The worst I've seen so far was a company with sub-30 engineers and 250 repositories. Friends, hear me -- this is madness.

Forcing talented engineers to divide their efforts across N number of repositories has become so common that we take it for granted. Tech teams often assume that a company must have a different repository for each piece of tech. They often assume that they need to carefully curate access to the sacred, private, precious code. We've all been that poor engineer needing to solve a problem, but getting stuck at the first step because we don't have access to the repository. It's dreadful.

At Cord, we don't mistake complicatedness for technical achievement. You know what is a monumental achievement? A fast, well-maintained monorepo. On day one, every engineer we hire has access to all of our code. There are no sacred, secret repositories. We do this because we hire great people and want to empower them to do big, bold things. If one of our engineers sees that she can improve all of our code by adopting a smart, idiomatic code abstraction, why on earth would we force her to fix up the code in 10... or 20... or 50 repositories? At Cord, you can improve the entire stack with a single commit.

Is it harder? Sure. We have to think carefully about how to make everything work well and fast when it all lives in a single repository. And that investment pays a dividend every single time our of developers makes the client side change and the server side change in a single, atomic commit. Or when a junior engineer can do a cross-stack refactor in a single commit. If we needed to roll something back, we can do it just as easily. We don't have to think about asynchronous release processes or complex branching models.

It's a symptom of an operationally weak engineering team to impose boundaries and force dependencies on engineers. We work hard to break down those boundaries and dependencies. This makes the default workflow the efficient one. It means that people building products rarely or never have to think about the operations layer. They don't have to worry about synchronising their work across teams or services or repositories. It's hard work and it pays off in developer speed.

Very Fast Continuous Integration

Build pipelines are a crucial piece of the software engineering puzzle. We want to know if our code works ASAP. If something is broken, the sooner we find out, the better. We've invested in the speed of our CI pipelines such that the time between commit and deployment is less than five minutes. This meant ripping out one CI toolchain and replacing it with another one. It's meant changing the order of operations in deployment multiple times. In another post, we'll describe the drastic step of ditching Webpack entirely because it's just too slow. Why did we do all of this? Because we want the development iteration loop to be as fast as humanly possible.

We take an all-access approach to our CI pipelines, too. If your commit breaks the build, you can see that. You can see the error and debug it. It's seems wild to have to make this point explicitly, but many great coders are stuck on teams where access to the CI tooling is kept behind lock and key of the operations teams or "senior" engineers. We don't believe in gatekeepers. We don't believe in protected enclaves or blessed, untouchable pipelines. We do believe in developer empowerment. We want every coder to be able to do their work and fix their shit without arbitrary hurdles to jump.

GitHub, no. Git wizardry, yes.

Let's have some real talk. The vast majority of software developers in the world never actually learn how to use Git. Many engineers will bristle indignantly at that statement, but a small subset will nod quietly to themselves and sigh and stare solemnly into the distance. Part of why we don't learn to use Git properly is because the Pull Request development style has become the standard for code review in the vast majority of tech companies. There are enough Stack Overflow snippets out there that most engineers can just copypasta their way through enough Git commands to get a pull request out.

There's a huge hidden cost to this. Pull Requests are actually *terrible* for code quality. Every merge commit is a little stopper on the productivity of the whole team. 99 out of 100 Git repositories are so wrecked by wildly branching and merging histories that senior engineers couldn't Git bisect to debug the codebase if they wanted to.

At Cord, we know that a clean, linear history in a repository is crucial. We believe every commit should build. Every commit should have a test plan. Every commit should be reviewed. Every single commit. Now I hear you screaming, "But wait... this is a post about developer velocity?! That sounds painfully slow." Au contraire, my pear! There is a smarter way.

There's a much, much less-known working style for code review called 'patch-based development' or 'diff-based development' which includes a specific variant called the stacked diff workflow. Working this way means that instead of piling 10-1000 commits into a branch and then merging it all into the main branch, you can land each commit when it's ready to land. The commits are smaller and more coherent. They move faster through the CI pipeline. They get tested sooner. This means we actually significantly decrease the risk per-commit. When you can see the impact of each change minute by minute, you know immediately when/if something is broken. By contrast, Pull Requests encourage huge, messy merges that take all the risk of each commit and smash it together into one high-risk merge. That's no bueno, my... dang, what rhymes with bueno?

This does mean that our team are basically all Git badasses. We like engineers who want to be masters of their tools. Copypasta mercenaries aren't really our jam. So, for every engineer, we do a Git master class. We break things down all the way to objects and refs. We cover rebases and merges and the reflog. And the result is that we move way faster than most engineering teams while also keeping the quality super high.

No Kubernetes

You heard me. Now put your pitchforks down. This also doesn't require those torches. Look, it's not that K8s is bad. It's just cargoculted and implemented when it absolutely doesn't make sense. We've lost the plot a bit on this one, as an industry. An engineer shouldn't need a PhD in Kubernetes to run the whole stack. You should be able to run everything locally and not because you're running Minikube, two proxies, and a partridge in a pear tree.

When Cord scales to the point of hundreds of server instances and dozens of components in the system, we'll reassess. We just don't need much infra now and won't need it for a year or more. So, so many companies buy into the complexity of K8s as a technical achievement divorced from any real need for that complexity. Just like Agile, K8s seems to have spawned many acolytes and zealots. They come in with bold promises of isolation, templatisation, and -- most of all -- reliability. They're not wrong exactly. But they always dramatically underplay the primary cost: developer velocity. Once you've got your Megatron of docker containers and subnets and clusters, your application does run. And (probably) more reliably. But... what at what cost?

Picture it -- you've K8s-ified your teams' whole stack, it's a glorious masterpiece. Like the inside of a Swiss watch. Then a troublesome product engineer comes along and says, 'Hey, how do I launch a new process to do real-time image encoding for this experimental feature we're building? We need production DB access.' You look at your Swiss watch. Its seamless edges. Its scratchless crystal face. And now this peasant dares to ask you to rip it open to bodge in an experiment? The bloody cheek!

The important point implied by the above is that once you've k8s-ified your stack, the average engineer is no longer capable of doing anything with the infrastructure. No room for launching a quick experimental cache layer. No easy way side-load data for analytics pipelines or *gasp* debugging production... . Too many security layers and subnets and permissions issues. To say nothing of the 50 layers of YAML config files they'd have to wade through. It works directly against your most investigative engineers. Instead, all the keys are held by the SRE or DevOps lead. This is a great place to be if you're an SRE lead. It's an absolutely terrible place to be if you're a good engineer trying to help a startup succeed. Importantly, there's usually a 10:1 or more ratio between product engineers and DevOps folks. So, what's the smart optimisation for the whole team? Hint: It's not a Swiss watch.

At Cord, we've avoided this pitfall. We may or may not ever take the K8s plunge. If we do, it will be for carefully weighed reasons. One of those key tradeoffs we'll weigh carefully is how our infrastructure choices impact individual engineer productivity.

TL;DR: Developer velocity dictates company velocity so we optimise for it heavily

This post is about practical examples of optimisations we've made to enable maximum developer velocity. We do this because developers do the majority of the work in any tech company. If we make our work slow and clumsy, our company will be slow and clumsy. If we make the day-to-day workflow for developers super fast, our company will be super fast.

Btw, we're hiring. If you think you'd like to work at a company with these values, we'd love to talk. Shoot me an email: jack@cord.com.

Author

Jackson Gabbard

Categories

  • Engineering