The problem with rapid evolution: Blind spots and siloed tools

In biology, rapid evolution can bring about convoluted ways to solve problems. Peacocks' enormous tails are a burdensome mating signal, making the bird clumsier and an easier prey as a result. The wires of the human eye go in front of the photo-receptive cells and lead back through a hole in the middle, giving us all a tiny blind spot where the nerve fibers occlude our vision. It works, but it's not perfect.

 using computers also saw rapid evolution: We moved from desktop apps and file-based sharing to cloud-based web apps, with the average company now using over 88 different web apps, 

. In building on the shoulders of giants, copying and mutating, subsequent generations of software settled on some paradigms: In browser. Based on HTML. Log-in with Google. Open APIs.

What hasn’t evolved yet, though, is a consistent way to work together in these tools. So when we need our teammates for help, approval, feedback, or for them to do their bit, we have no choice but to resort to Slack.

The problems with this method can be hard to see, until you reimagine how you would've built it if you could control everything. Like fish in water, we're so used to the lack of communication in our work tools that we don't even notice the friction it causes. But when we're in 'deep work' mode, trying to finish, say, a marketing campaign that has to go out 

, and we're unsure we put in the right graphic - why do we have to take a screenshot, context switch, find a relevant Slack channel, re-attach the screenshot, write a message, paste the link, and then keep checking the channel back for answers, as our question gets buried under new messages from other people with different asks? We have tools where we get shit done, and other tools for communication, and there's no connection between the two. Other smart folks 

This is akin to trying to build a piece of IKEA furniture with your partner, but without being able to communicate verbally or with gestures. When you want to ask them to pass the screwdriver, or where this wooden bit fits, you have to go to another room, away from where the half-assembled cabinet is, wait for them to come in, talk about it, and go back. Except now, we're on a factory floor trying to assemble 100s of different cabinets, asynchronously, and with one room where everyone is shouting.

The magic happens when world-class communications meets context, inside the tools you use

There are great innovations in communication tools. Simplified, multiplayer, rich-text editing like Notion, voice chat apps, and even 'plain old' Slack is a major improvement over email. But SaaS tools are built for a purpose, and they do their specific job very, very well. Marketers relying on an ad-tech tool, or Devops engineers using a management console cannot and do not want to work elsewhere, so trying to force all workflows into limited capacity tools like Google Docs or your task-manager-meets-wiki-meets-chat can't be the answer. We believe the right answer is empowering these specialized tools. We want to infuse the entire stack with world-class communication.

World-class communication isn't a hyperbole. Most communication that's tacked onto a product as a lower priority feature sucks. Just try and use LinkedIn messaging for more than answering casual recruiting reach-outs for an example of that. The commenting sections of most SaaS apps are even worse. No surprise they aren’t used, causing the vendors to invest even less in improving the feature.

But imagine if communication in SaaS apps was as good, or better, than Whatsapp. They'd have search, mentions, presence, typing and 'seen' indicators, audio messages and the rest of the long list of nuances that make chat products like Whatsapp shine. Suddenly, just chatting where you're already working becomes much more convenient than Whatsapp.

If you can then interact - directly manipulate and collaborate in the tool you're having the conversation in - annotating bits of the page, editing together and creating tasks that track the work that's left to do, you become more effective. You stay in flow and leave a trail that helps others find their way and see what decisions were made and how. You don't have to go far to imagine that: No one using Google Docs ever sends you a link and says "

 - they just leave an annotation there. And if this miraculous chat is accessed and used the same way across any tool you use, with a unified inbox for all your notifications, it's easier to adopt the habit and stay informed.

A unified inbox that’s not everyone’s inbox

Another reason why Slack and the other communication 'destinations' are so distracting and a poor fit for 'contextual conversations' - where there's an item to review, a transaction to complete, or an incident to investigate across several pages - is that in these channels you're essentially in everyone's inbox. In the tools you work in, there's a set of people who are relevant to the task at hand. Once you're working with them on that task, there's no reason for everyone to continually see that conversation. But no mapping exists between the people you work with on a particular task and the channels you have available in Slack: You either spam the entire #design team, or you have to find the exact group - Juan, Michelle, Dan, Celeste (but don't accidentally choose Juan, Michelle, Dan, Diane), in which case they can't ever pull new people in or find where the decision about that marketing campaign was. And once you're in that chat destination, you'll also find you now have 6 other unread channels. In all likelihood nothing there really matters to you (it's probably all people spamming the entire team to get one item done) - but there went your entire concentration and flow.

There's a very good reason and a very good place for the townhall and phone book that Slack, Teams, etc. provide: Announcements like the company's new strategy, and DMs like "

 that are unrelated to a work item in a tool. But stuffing every single request for approval, help or feedback down that channel never works. As the company scales, you end up with 

10,400 page long guides on communications

We want to empower you to focus on the work, and collaborate inside your virtual workshop or atelier, or your own product, internal tool or landing page. That's what you're talking about, so it might as well be where you talk. That's why we built 

. To curve the arc of our tools' evolution towards their collaborative potential.

The way in which computing evolved led us to find ourselves working across different, single-player web-based tools, and communicating in a separate 'destination' like email or Slack. We can design a better way to collaborate.

How in-app collaboration is changing the way we work

Building apps – internal tools or otherwise – is 

. But actually working together with your teammates when code is involved is easier said than done. Taking screenshots, adding annotations, sending a flurry of Slack messages, and then adding Jira tickets is 


In this post, we’ll walk through how to use 

 (a fast way of building internal tools) with 

. Cord lets you add comments, annotations, and guides directly in your favorite apps – so you can talk with your team in the apps you’re working in. 

 and project management tools (Jira, Asana, Monday, etc.) so you can create and manage tickets without leaving your app.

Save time by building internal apps with Retool and Cord

Here’s a few suggestions for how teams use Retool and Cord to build better internal tools without all the back and forth.

Write onboarding guides directly in the app. 

Onboard new members of your team by creating annotations and comments with Cord. Attach relevant files to document how the internal tool works, right inside the tool. This is a lot easier than outdated walkthroughs lost in Confluence or Coda. Plus, teammates can @mention you if they have a question and you can chat directly in Retool– almost like sharing your screen on Zoom, without having to sit through another video call.

Write onboarding guides directly in the Retool app with Cord.

Create tasks that will automatically appear in Jira or Asana. 

Anytime you write a comment using Cord, whether it’s to investigate a bug or change a dropdown to a search input, you can make it an Asana or Jira task so it fits right into your workflow (and it will actually get done!).

Create tasks that will automatically appear in Jira or Asana with Cord.

Your internal ops team can log edge cases and leave a record of the conversation. 

The users of your app can record edge cases and leave notes about what was decided inside Retool to increase efficiency and speed up onboarding for new team members who may run into the same issue.

Add Cord to your browser to start collaborating in Retool

Cord is a Chrome extension that adds a collapsible sidebar to your browser that shows up when you are using Cord supported tools. Install Cord in a few quick steps:

2. Click “Add Cord to Chrome for free”

3. Sign-in with Slack (Or, sign-up to be notified when we add Google and Microsoft Teams support)

4. Complete onboarding: Watch a 30 second video, install the browser extension from the Chrome store, and import relevant Slack messages by adding the Cord Slackbot

Complete Cord onboarding

5. Then, go directly to your Retool app and start annotating with your team!

Go directly to your Retool app and start annotating with your team!

If you don’t have a Retool account yet, you can sign up on 

Build together in Retool with Cord comments and annotations

Cord is a startup. We started with a strong vision: all sorts of people doing their jobs with less friction, more context, and easy collaboration. Making that abstract vision a reality is hard work. Exactly how much work it is depends. A major factor is how much the the company is optimised for developer velocity. At Cord, we don't mess around about developer velocity. Startups like us are lucky to attract great talent. We want to honour that by making work really, really fast.

One of the best ways we boost developer speed is by taking away all the friction of learning the tech stack and processes. At Cord, when someone joins the team, we take time to do hour-long sessions on every layer of our tech stack and operations. We walk them through the client code, the server code, the full system architecture. We do a session on the DB schemas and migration logic. Our deployment and devops setup. We onboard them to the company-level iteration process. We bring them to user interviews. At the end of her first month, an engineer on our team has a seen everything there is to see of our tech. They know the code review process, the continuous integration. They know who to tap for feedback or questions.

It does take time away from each of our team members to do these onboarding sessions, but that time is paid back many times over by every new team member feeling really grounded and informed. It shows them that we trust them and want them to have access to all the knowledge they need to succeed. I've been on a lot of teams where even a year into their jobs, developers still see aspects of their tech stack as mysterious and hazy unknowns. Where there is ambiguity about who owns what or how to solve certain kinds of problems. I can say with high confidence that empowered and informed developers produce way more and way better work, so that's what we ensure we have.

An image of a dangerous looking man with the text, "No agile. No sprints." 

No Agile, No Sprints.

Hot Take: Sod retrospectives. Banish sprint planning. Burn down the burn down charts. Leave the manifestos to the revolutionaries. None of the process. Why not? It doesn't match our needs. The world has forgotten, but Agile replaced an even more outdated (but equally flogged) methodology called 

. Agile was a huge innovation in its day. So, too, was the telephone. But now we have the internet.

Despite what various Agile zealots and coaches will vouchsafe to you, Agile doesn't solve all ails. The opposite in fact. Sadly, it's most commonly a symptom of deeper company-level problems which subsequently go unaddressed. Agile is like logging all your food, taking a daily blood sample, carefully regulating your sleep, but actually your leg was broken the whole time.

Maybe it's a flakey product owner who doesn't know what they want. Maybe it's a CTO or VP of Engineering who isn't providing strong technical leadership. Maybe the company has hired weak developers who don't get good work done. Whatever the situation, process-heavy methodologies typically only add drag on developer speed because they create new work without fixing the root causes of the original problem. You go from writing code with unclear requirements to endlessly curating tasks which tediously detail those unclear requirements and then still have to write code against those well-documented unclear requirements. 

When you have a problem with your dev team and you try to solve it with more process, you typically only succeed at triggering the Tech Company Doom Loop: 

Behold, the tech company doom loop

We take a different approach. Specifically, we derive priorities from direct user engagement. Then we trust our team to do important work. We rely on solid tech leads and engineering management to give people a nudge if they're veering off piste. Instead of two week sprints, we work in roughly 6-week blocks, setting high level priorities based on our company strategy. From there, we figure out what the rough shape of those priorities looks like as technical execution, then we let our developers do what they do best -- solve the problems. Great people just want to get into the technical work, not wade through 3 page specification documents. Clear priorities and mature technical leadership enable this.

Our team includes stellar people who have track records of delivering projects for Oculus, Google, GoCardless, Facebook, among other great companies. Do we need to chase our people for t-shirt sizing on their tasks? Nope! Should I interrupt a senior engineer mid-flow to let him know he hasn't filed a prioritised ticket that explains his current scope of work? No, no, no. What we do instead is set clear top-level priorities and let our tech leads drive the right outcomes, supporting mid-level and junior developers to rip through the tasks. Because we've got a great team, this works.

An image of some guy getting struck by lightning or being abducted by aliens or something.

There can be only one... repository.

When I talk with engineers at other companies about their work, I often hear about how they juggle multiple codebases. I hear about small teams with 10 different Golang microservices, each with its own repo. I've seen an application frontend split into 3 different Git repositories for... reasons. The worst I've seen so far was a company with sub-30 engineers and 250 repositories. Friends, hear me -- this is madness.

Forcing talented engineers to divide their efforts across N number of repositories has become so common that we take it for granted. Tech teams often assume that a company must have a different repository for each piece of tech. They often assume that they need to carefully curate access to the sacred, private, precious code. We've all been that poor engineer needing to solve a problem, but getting stuck at the first step because we don't have access to the repository. It's dreadful.

At Cord, we don't mistake complicatedness for technical achievement. You know what is a monumental achievement? A fast, well-maintained 

. On day one, every engineer we hire has access to all of our code. There are no sacred, secret repositories. We do this because we hire great people and want to empower them to do big, bold things. If one of our engineers sees that she can improve all of our code by adopting a smart, idiomatic code abstraction, why on earth would we force her to fix up the code in 10... or 20... or 50 repositories? At Cord, you can improve the entire stack with a single commit.

Is it harder? Sure. We have to think carefully about how to make everything work well and fast when it all lives in a single repository. And that investment pays a dividend every single time our of developers makes the client side change and the server side change in a single, atomic commit. Or when a junior engineer can do a cross-stack refactor in a single commit. If we needed to roll something back, we can do it just as easily. We don't have to think about asynchronous release processes or complex branching models.

It's a symptom of an operationally weak engineering team to impose boundaries and force dependencies on engineers. We work hard to break down those boundaries and dependencies. This makes the default workflow the efficient one. It means that people building products rarely or never have to think about the operations layer. They don't have to worry about synchronising their work across teams or services or repositories. It's hard work and it pays off in developer speed.

Build pipelines are a crucial piece of the software engineering puzzle. We want to know if our code works ASAP. If something is broken, the sooner we find out, the better. We've invested in the speed of our CI pipelines such that the time between commit and deployment is less than five minutes. This meant ripping out one CI toolchain and replacing it with another one. It's meant changing the order of operations in deployment multiple times. In another post, we'll describe the drastic step of ditching Webpack entirely because it's just too slow. Why did we do all of this? Because we want the development iteration loop to be as fast as humanly possible.

We take an all-access approach to our CI pipelines, too. If your commit breaks the build, you can see that. You can see the error and debug it. It's seems wild to have to make this point explicitly, but many great coders are stuck on teams where access to the CI tooling is kept behind lock and key of the operations teams or "senior" engineers. We don't believe in gatekeepers. We don't believe in protected enclaves or blessed, untouchable pipelines. We do believe in developer empowerment. We want every coder to be able to do their work and fix their shit without arbitrary hurdles to jump.

Let's have some real talk. The vast majority of software developers in the world never actually learn how to use Git. Many engineers will bristle indignantly at that statement, but a small subset will nod quietly to themselves and sigh and stare solemnly into the distance. Part of why we don't learn to use Git properly is because the Pull Request development style has become the standard for code review in the vast majority of tech companies. There are enough Stack Overflow snippets out there that most engineers can just copypasta their way through enough Git commands to get a pull request out.

There's a huge hidden cost to this. Pull Requests are actually *terrible* for code quality. Every merge commit is a little stopper on the productivity of the whole team. 99 out of 100 Git repositories are so wrecked by wildly branching and merging histories that senior engineers couldn't Git bisect to debug the codebase if they wanted to.

At Cord, we know that a clean, linear history in a repository is crucial. We believe every commit should build. Every commit should have a test plan. Every commit should be reviewed. 

 Now I hear you screaming, "But wait... this is a post about developer velocity?! That sounds painfully slow." Au contraire, my pear! There is a smarter way.

There's a much, much less-known working style for code review called 'patch-based development' or 'diff-based development' which includes a specific variant called the 

. Working this way means that instead of piling 10-1000 commits into a branch and then merging it all into the main branch, you can land each commit when it's ready to land. The commits are smaller and more coherent. They move faster through the CI pipeline. They get tested sooner. This means we actually significantly decrease the risk per-commit. When you can see the impact of each change minute by minute, you know immediately when/if something is broken. By contrast, Pull Requests encourage huge, messy merges that take all the risk of each commit and smash it together into one high-risk merge. That's no bueno, my... dang, what rhymes with bueno?

This does mean that our team are basically all Git badasses. We like engineers who want to be masters of their tools. Copypasta mercenaries aren't really our jam. So, for every engineer, we do a Git master class. We break things down all the way to objects and refs. We cover rebases and merges and the reflog. And the result is that we move way faster than most engineering teams while also keeping the quality super high.

You heard me. Now put your pitchforks down. This also doesn't require those torches. Look, it's not that K8s is bad. It's just cargoculted and implemented when it absolutely doesn't make sense. We've lost the plot a bit on this one, as an industry. An engineer shouldn't need a PhD in Kubernetes to run the whole stack. You should be able to run everything locally and not because you're running Minikube, two proxies, and a partridge in a pear tree.

When Cord scales to the point of hundreds of server instances and dozens of components in the system, we'll reassess. We just don't need much infra now and won't need it for a year or more. So, so many companies buy into the complexity of K8s as a technical achievement divorced from any real need for that complexity. Just like Agile, K8s seems to have spawned many acolytes and zealots. They come in with bold promises of isolation, templatisation, and -- most of all -- reliability. They're not wrong exactly. But they always dramatically underplay the primary cost: developer velocity. Once you've got your Megatron of docker containers and subnets and clusters, your application does run. And (probably) more reliably. But... what at what cost? 

Grimace from McDonalds and his daughter, young She-Hulk having a crucial conversation.

What did k8s cost you? Velocity.

Picture it -- you've K8s-ified your teams' whole stack, it's a glorious masterpiece. Like the inside of a Swiss watch. Then a troublesome product engineer comes along and says, 'Hey, how do I launch a new process to do real-time image encoding for this experimental feature we're building? We need production DB access.' You look at your Swiss watch. Its seamless edges. Its scratchless crystal face. And now this peasant dares to ask you to rip it open to bodge in 

 The bloody cheek!

The important point implied by the above is that once you've k8s-ified your stack, the average engineer is no longer capable of doing anything with the infrastructure. No room for launching a quick experimental cache layer. No easy way side-load data for analytics pipelines or *gasp* debugging production... . Too many security layers and subnets and permissions issues. To say nothing of the 50 layers of YAML config files they'd have to wade through. It works directly against your most investigative engineers. Instead, all the keys are held by the SRE or DevOps lead. This is a great place to be if you're an SRE lead. It's an absolutely terrible place to be if you're a good engineer trying to help a startup succeed. Importantly, there's usually a 10:1 or more ratio between product engineers and DevOps folks. So, what's the smart optimisation for the whole team? Hint: It's not a Swiss watch.

At Cord, we've avoided this pitfall. We may or may not ever take the K8s plunge. If we do, it will be for carefully weighed reasons. One of those key tradeoffs we'll weigh carefully is how our infrastructure choices impact individual engineer productivity. 

TL;DR: Developer velocity dictates company velocity so we optimise for it heavily

This post is about practical examples of optimisations we've made to enable maximum developer velocity. We do this because developers do the majority of the work in any tech company. If we make our work slow and clumsy, our company will be slow and clumsy. If we make the day-to-day workflow for developers super fast, our company will be super fast.

Btw, we're hiring. If you think you'd like to work at a company with these values, we'd love to talk. Shoot me an email: 

Our philosophy is simple: hire smart people, help them move very fast, and good things come as a result.

Enabling Developer Velocity

, you can annotate, @mention, and create tasks while using other SaaS tools, so your team can collaborate right where they are working. Connecting Cord and Slack unlocks the ability to work side-by-side in SaaS tools and Slack.

Slack is a communication hub for many teams. But in reality, you work in many other tools, too. As soon as you start working in another tool, whether it's Google Analytics, Webflow, or HubSpot, you have to send links and screenshots from the tools back into Slack to get your team up to speed. 

With Cord and Slack, comment in context in any tool and see all messages consolidated in Slack. You'll never send a link or screenshot again.

How does connecting Cord and Slack make workplace collaboration easier?

When you @mention or assign a task to someone in Cord, they'll get a Slack message and can jump right into to the tool you referenced to continue the conversation in context.

See Slack messages right on the page that they mention.

 If you send a coworker a link to a page on Slack, when they open that link, your conversation will automatically be there.

 Easily share Cord conversations to public channels or private conversations on Slack to get your team's input or share decisions.

What information does Cord get from Slack?

Cord uses your name and profile picture from Slack. No default avatars here!

See who's already using Cord or invite members of your Slack workspace.

We save the email you used to sign-up for Slack in case you need to recover your Cord password, because hey – it happens. We won't send you any marketing emails without your permission.

Get the Cord + Slack integration for free. 

More questions about Cord's Slack integration? Don’t hesitate to contact us at 

Add comments and annotate inside any tool you use at work, get updates in Slack

Cord + Slack = Collaborative magic ✨

This is the second part of a three part series about choosing a technology stack for a startup in the modern (at least as of the time of this writing) world. The 

 focussed on building a client side web app using modern tech. For this post, we'll dive into the server side tech choices.

This post is going to focus specifically on implementing a server and database for our use case. I'll touch briefly on things deeper in the stack than these components, but our devops choices will come through in part 3 of this series.

As we discussed before, we're building a tool that enhances other tools. We want to make any existing SaaS tool into a dynamic, multi-player tool that feels connected. To make the experience of working across many SaaS tools into a connected, coherent experience rather than a series of disconnected silos. We're building features that helps a team feel like they're working together, even if they have never even stepped foot in the same room together. Our application architecture demands a mixture of real-time and asynchronous features.

These constraints dictate many key pieces of our server-side product infrastructure. For one, we need authentication and sessions. We also need a way to communicate between the client and the server that works both synchronously and asynchronously. In particular, we need a server that can push updates to the client in real time.

The pivotal constraint for us is that we want a blisteringly fast pace of development for the product. Engineers working in our stack should be able to ship features as quickly as possible, experiment, throw things away, and redeploy changes in minutes. Our current continuous integration runs in about 3 minutes.

Why this need for speed? Well, for one, it just feels great to work in a high-speed tech stack. For two, we're still early in our product's evolution. Scaling our server-side infra is not a constraint yet. In fact, if we focussed on building an infinitely scalable server, we'd be wasting time. Right now, we're incrementally transitioning from early user tests to true product market fit. For our stage of life, things like maximum I/O parallelisation or complete database race condition safety just don't matter. What does matter is being able to iterate on product features, test them with real users, understand if they work or not, and then iterate again. So, our tech stack choices reflect the need for this flexibility over alternative choices (e.g. choosing a "high performance" language like Go or Rust; division of responsibilities like microservices; etc.).

One of the very first questions to answer when designing a server architecture is 'What language/run-time should the server be built in?' For us, this answer was trivially easy to reach -- NodeJS. While server-less architectures using AWS Lambda or Firebase are becoming quite trendy, they suffer from some really unfortunate deployment pathologies. Our philosophy is that it should be stupidly easy to know what is deployed -- from the server to the database. Because of this, we opted for a TypeScript monorepo which deploys atomically as a NodeJS server. Sure, this means we can't re-deploy individual functions at a time as we could if this were a Firebase app. Honestly though, the server-less trend seems like a really false economy in practice. With a monolith, we actually know what versions of the code are running at any given moment and we can trivially guarantee that all the pieces fit together properly all the way from the database schema to the client-side React UI components. For similar reasons, we did not build a microservices-based backend.

 as a backend or something built in Python using Flask or another hand-rolled server of some sort. Golang would give us obvious performance and concurrency benefits. Python has a fantastic ecosystem of pre-fab libraries. However, both of them offer one huge drawback: they aren't JavaScript. Try as we might, no one has yet killed the warty-but-loveable beast that is JavaScript. They came with their torches and pitchforks and ActionScript in the 2000s. They came with their big promises and DART in the 2010s. They now sing siren songs of WebAssembly in the 2020s. Yet, JavaScript remains. It is inevitable, like Thanos, but like... actually.

Inasmuch, since we're going to have to build a JavaScript app no matter what, the best we can do to make the overall development experience really, really good is build everything in JavaScript. Sorry Golang. Sorry Rust. Sorry Python. Sorry Haskell. So Erlang. Building an application in multiple languages quickly is much, much harder than building an application in one language quickly. We want our engineers to be able to deliver value across the entire tech stack deftly. We want to benefit from shared libraries and well-crafted language idioms whether they're building a complex client-side UI component or they're refactoring our server-sided data caching logic.

Using multiple languages creates a surprising number of barriers to contribution, the first and most obvious of which is that not everyone will know all the languages. On top of that, you instantly lose the ability for one engineer to make sweeping changes across the entire codebase with a single commit.

), these sorts of changes enable a small number of engineers to maintain massive amounts of code. If we'd chosen the industry standard practice of multiple repositories hosting multiple projects in different languages, such codemods would be impossible. They'd be relegated to at most one repository at a time and likely even worse than that because not all the repos would be in the same language. This leads to differing code styling standards, differing code review standards, differing CI pipelines, differing linters. And infinite -- I mean -- 

 about the right way to do things. Ain't nobody got time for that.

If you read Part 1 of this series, you won't be surprised to see TypeScript here. Most of the mixed blessings of TypeScript apply on the server-side as much as they did on the client, but for NodeJS application development, TypeScript offers a lot of value. Server-side development tends to be a bit less messy with respect to types than client-side. Part of the fiddly-ness of TypeScript in the browser is the DOM API itself. For server side development, you just don't have to fool with it, which is pretty great.

For data storage, we've opted for a tried and true, fast, stable, relational database. Server-less apps running against schema-free databases are all the rage because they make for ultrafast prototyping. If you're creating a build-and-forget backend for a hackathon, that stack choice is totally reasonable. However, if you need to maintain your data over a long period of time, schema-free databases like DynamoDB or MongoDB might trip you up unexpectedly. I can't tell you how many teams of engineers I've interacted with who don't properly migrate their schema-free database because "but it's schema-free, you don't have to know what fields you've set on every record!" That's great if you don't want to be troubled by keeping track of what you've stored...

However, if you want your data to match the code that is operating on it, you need to manage this relationship. There's nothing more difficult to debug than random bug reports that come from users in the wild because some poor user wrote some records to the database before you added some field or before you removed it and now when she loads the app, it crashes because you can't read some property of undefined. So, so many engineer lifetimes have been wasted debugging problems that boiled down to "

 the data is actually corrupt because they happened to hit the save button on the day before we added the spline-reticulation-rate field, so her app can never load without errors

". Yet, these kinds of bugs happen all the time in modern codebases specifically because the engineers don't operate with some discipline about their data. Ho hum. Not us.

But I agree... maintaining the schema of your data is a chore. That's why we use a migration framework which handles upgrading and downgrading our database for us. All our engineers have to do is make sure we've written correct SQL migrations to add and remove fields or tables and correctly transform the existing data. After that, our CI tooling handles the rest across our environments. Easy peasy, lemon squeezy.

Using an ORM in a large-scale, high-performance application is usually a fool's errand. Trying to get performance out of queries you don't have direct control over is a bit like trying to assemble a ship-in-a-bottle while wearing oven mitts. Fortunately for us, we're not a large-scale, high-performance application -- 

. For now, speed of development is the dominant factor in the Big-O equation of our company's success. We will almost certainly rip out Sequelize's ORM functionality over time, replacing it with carefully designed indexes and queries. For now though, we can add types, find, update, and delete stuff, and generally crack on with the work without stressing about every join clause or order by. I haven't written a single raw SQL query apart from a CREATE / ALTER / DROP TABLE query in months.

For the TypeScript savvy folks, you might be surprised to see Sequelize as our pick for ORM rather than 

, given that TypeORM is... well.. typed. We started with TypeORM and, within a month, hit so many issues and so many known-but-unfixed problems with TypeORM that we ended up ripping it out. As of the time of this writing, the TypeORM project has more than 2k open issues, which is actually staggering for a codebase with only 4k commits in it. Your mileage may vary, but TypeORM was not the ORM for us.

Also offers a really nice migration framework which we're likely to keep well into the future. I've worked previously with schema-free codebases (like Mongo) and codebases where the schema was maintained independently of the codebase. Having migrations tied to version control is an absurdly better world to be in. I'll never go back. Don't make me go back.

 of this series, you'll know I have mixed feelings about GraphQL. It's heavy. Like heeeeaaaavvvvy. You have to do a heap of work just to get the basic value of the abstraction. If the entities in your system don't have a lot of complex relationships, it's probably not for you. However, in our case, we have lots and lots of relationships in our schema. For instance, an organization has users. Also, users have messages. And messages can have mentions, which themselves can have users. In this sort of application, GraphQL begins to show its value peeking out over the mountaintops of extra work you need to do to get it up and running.

Why? Well, because a user is such a core type in our system, we get an economy of scale out of defining it 

 in GraphQL. Once we've done the heavy lifting of GraphQL-ifying the user type, we can use it over and over again everywhere that it comes up. This is because GraphQL makes entity composition trivial. The more types you have and the more relationships they share, the more valuable GraphQL becomes.

Let's compare that with a vanilla JSON API. If we started both versions of the server on the same day, Team Vanilla JSON API would easily race ahead at the start. Then, as the number of entities and number of relationships between entities grows, the Team Vanilla JSON API would start to become ponderous -- needing to reassemble the same queries and transformations again and again. Team GraphQL by contrast would get left on the starting line, toiling away trying to get their resolvers and mutations in order. However, as the race wore on, Team GraphQL would start to make huge leaps. Once the core types were defined and GraphQL-ified, Team GraphQL would basically never touch them and would instead only be building the new entities. They wouldn't have to worry about which fields of the sub-sub-queries to expose, because their earlier work would already account for that. Over the course of a year with lots of product iteration, Team GraphQL would have enabled way more product iteration, despite taking longer to lurch into motion.

A basic example would be the @-mentions in our chat. To render a message with an @-mention in the UI, you have to fetch: the message, the message sender's user data, the mention data, and the user data of the user that was mentioned. Since we have a well-defined User GraphQL entity, we don't have to do any extra work to the data required to display both the sender's user info and the mentioned users user info. They'll leverage the same GraphQL entity. To achieve this same value with a JSON API, we'd have to write custom server-side code to fetch the message, then fetch its users info, decide which fields to expose, then parse out the @-mention from the message, then fetch the mentioned user's info, then decide which of those fields to expose in the response, then assemble a JSON payload that includes all of this information. We'd also have to think about things like batching of these fetches if we're rendering a list of messages. And if we later on modified the User entity to have some new fields, we'd have to maintain all that custom JSON API endpoint code, too. What a pain. While GraphQL slows us down at first, its speeds us up shortly thereafter in a way that vanilla JSON APIs won't catch up with.

But let me be real with you -- the itch to just rip all this GraphQL out and replace it with a dead simple JSON API that stringifies data pulled straight from a raw SELECT * FROM ... statement is...

When working with a GraphQL backend, you feel the pain of all the overhead more acutely than working in the client. Again, this is a reasonable choice for us because we want to be able to iterate on the product as fast as possible. Whatever form our UIs take, the backend data types are quite stable. So, GraphQL unlocks a lot of value for present and future product development. If I had it to do all over again, I'd probably choose it again, begrudgingly. I really wish it was less heavy weight. Codegen can theoretically save you some of the pain of defining all these JS-to-GraphQL mappings, but then you also have GraphQL 

 codegen to worry about. There's no free lunch there.

In our application, real-time interactions are hugely important. We're building a product that makes people feel connected and present, even when they're not in the same room together. Surprising moments like "Oh, look, my teammate and I are on the same page at the same time" make the experience of working together-but-apart much more serendipitous and rewarding, like being in the office together. To enable that, we have loads of real-time, in-memory state that reflects the ephemera of day-to-day work. Without something real-time like WebSockets, we would struggle to keep entire teams in sync. If I add an annotation to a page in our shared SaaS tool, my teammate needs to know about it instantly. We can't wait for the teammate to reload the page whenever it might next happen.

 with Apollo means we have a really clean API for triggering data pushes down to all the currently connected clients. We also have the flexibility to put these pushes before, after, or concurrently with database transactions, depending on the desired user experience of the particular real-time feature. Lovely.

The dominant tradeoff of our server-side codebase is product development velocity over 

. We haven't chosen the most strongly typed language. TypeScript is a fairly good type checker, but it's far from the most opinionated type system. We certainly haven't chosen the most high-performance programming language. A Golang implementation of our backend would be between 10 and 100 times more efficient. However, with Golang, we'd also pay the bill of having two languages, two sets of frameworks, two sets of idioms, two sets of code review standards, two different development environments to maintain, etc.. Each one of these small frictions adds up to lost developer velocity.

We're a pre-product market fit company. We don't yet know what features will be the winning ones that shape the company's future. We don't yet know what the customer growth trajectory will look like. We also don't know how much we'll need to scale the engineering team to match the adoption we get once we launch. Success for us right now is definable as building out product features as fast as possible, testing them with users, and honing our product vision based on those outcomes. Anything that gets in the way of that iteration loop is a drag on our success.

One view of how to proceed would be to cobble together as many open source libraries as possible to build some vaguely product-shaped Frankenstein's monster just to test the product in front of users. Many, many companies take this approach. Some good, most of them... bad. You often see this approach led by a non-technical founder who is outsourcing development to an inexpensive, remote dev team. This overall approach is deeply unwise except in the case that you never need to scale the product or infrastructure. It's almost always the wrong tradeoff because it trades quick-and-dirty features today against your ability to grow with your success tomorrow. What happens if your product gains any traction at all? If you take this approach, the first thing that happens is that the application all falls apart and you have to scramble to find great developers to undo and rebuild all the work you already paid for once -- if you can get them at all. Most talented engineers will take two looks at a tech stack that is in such a poor state and keep looking for a better opportunity. What you end up with is contractors who are happy to saunter leisurely through your technical debt for a handsome day rate. That's penny-wise, pound-poor thinking.

Another view of how to proceed is to jump straight to a perfect implementation of the product and infrastructure on the first pass. If you're literally psychic or luckier than Harry Potter high as a kite on 

, you can make this tradeoff. It requires more time, more money, and more prescience than most startups have, certainly more than we do. For the mere mortals and muggles among us, building the "perfect" infrastructure is basically meaningless. Until you have real users using your product, you don't know what your product is. You don't know what usage patterns will create what pressures in your backend requiring optimisation. You don't know what killer features you'll discover when someone in a user interview casually opines that they "just wish the product did X" -- where X is some unbelievably brilliant insight about the user experience that your team never had. So, this approach is infeasible 

unwise, despite being extremely tempting to many tech teams, especially ones filled with PhD engineers or engineers from FAANG-esque companies who want to flex.

The wisest approach is a pragmatic compromise: move as fast as possible while building the application in a way that doesn't block you from changing and optimising it later. That's where we've landed. We have a really fast pace of product iteration and a super tight feedback loop between user testing and product development. Our stack is optimised exactly for this purpose. It hinges on the fact that a single engineer can make changes across the full stack in a single commit and see the outcome in production 3 minutes later. No waiting for code review from different 5 people, working with 5 different pull requests to 5 different repos, all written in different languages. This is the way.

Monolith versus microservices versus server-less

We chose a monolithic backend. This means everything is in-memory on the same machine until it can't be due to scalability constraints. Of course we will eventually have services that get sharded out from the monolith. However, we won't call them microservices. We'll call them services. And they won't be arbitrarily small function-calls-over-REST like many microservices tend to be in practice. Instead, they'll be whatever size they need to be to coherently solve the problem they need to solve. Everything that has similar performance and/or storage needs will get sharded out as a service.

As of the time of this writing, the most likely first candidate for this is a websocket termination service/loadbalancer that will act as a façade to our backend, allowing WebSocket connections to stay connected between service restarts.

What will the next service be? I don't know. Ask me again when we've had sufficient growth and user adoption to know what parts of our monolith end up dominating any of: CPU, memory, or network. When one piece of the application is behaving sufficiently differently from the rest of the application that they no longer make sense running in the same process, that's when we'll break it out.

Server-less architecture is a non-start for us. Because our application hinges on real-time, push-to-client interactions, there is definitely going to be a server. We could use some pre-baked WebSocket-as-a-service utility from an existing provider, but that's just paying someone else to handle complexity that our app definitely has. The lock-in that these sorts of tradeoffs create is very likely to kill our flexibility down the round. Right now, we can chose literally any path we like for how to tweak the server to handle larger scale traffic. If we were paying Google Cloud for the privilege of using their real-time infrastructure, we'd be building our application around that constraint. We've got a sufficiently talented engineering team to be able to manage that complexity ourselves and the tradeoff of getting to shape our application around our actual constraints rather than the arbitrary constraints imposed by a 3rd party service are absolutely worth it.

The MERN stack gets a lot of hype these days. In many coding bootcamps, it's taught as the de facto standard stack for modern development. Call us crazy, but we like doing far out things like 

. These arcane arts are not lost, though their power has been largely forgotten by recent generations of developers. We don't currently and likely never will use MongoDB in our stack. There is some potential for a caching layer like Redis to benefit us, but at the moment we can do everything flawlessly in Postgres.

I talked a lot about the downside of schema-free databases earlier in the post, but the same holds true here. We want to know exactly what our data looks like. We want to be able to shape it commit-by-commit with clearly defined migrations. Schema-free databases are easier at the start, but then tie your hands with respect to what features you can build later without writing your own data processing to create intermediate data representations and caches. And woe be unto you who writes some data to the DB in a different way that doesn't know to update the aggregations and lists stored elsewhere. No thanks. We'll take the burden of a relational datastore any day.

Part 2/3: Server Side Tech Picks

 in 2020 in the middle of a pandemic. Our mission is building absolutely killer real-time and asynchronous collaboration features that work with any existing SaaS tools.

This idea sounds simple enough, but when you dig into what’s required to deliver this, you start to see a lot of unique constraints starting to emerge, which lead us to some very specific technical choices. This post breaks down our choices for the client side of our tech stack.

 If you just want to know what tech stack we’re running here’s a quick run down. For more detail, see below. We’ve chosen 

We’re building tools that enhance tools. That in itself creates some super interesting constraints for how we have structure our tech stack. For instance, we can’t expect to have control of the page our code runs in. Our implementation has to play very nicely with our neighbours.

We have a heavy focus on both real-time and asynchronous features. On one extreme end, we want to be able to reflect up-to-the-second information like whether or not your teammate is typing a message. On the other end, we want to be able to reflect an entire conversation history in a tool quickly. This means we have to support short-term storage features, a lot like how WhatsApp works, 

 we have to support long-term storage features like complete-history retrieval, like Slack.

TypeScript is a mixed blessing everywhere you take it. It means that writing the code in the first place will be slower and more annoying. Learning all the finicky interactions between React and TypeScript is definitely a learning curve. Want to create an input event listener outside of your JSX markup? Cool, just learn the magical incantation e: React.ChangeEvent<HTMLInputElement>. Dead simple?! Right!!? Well, yes, once you’ve scratched your head and read docs for a while. Still, once you’ve paid the bill to create the TypeScript code correctly in the first place, it becomes indispensable for maintaining your codebase. I’ve now bounced between TypeScript and non-TypeScript codebases a few times and I miss TypeScript annotation every time.

Wow, no mystery about what properties this component supports!

The downside of the React + TypeScript combo is that you’ve pretty much guaranteed your compilation is going to feel sluggish. If you work locally on a reasonably modern laptop, it’s tolerable. Still, even after some optimisations, we’re seeing warm-recompile times in the low single-digits seconds. That’s just slow enough to feel sluggish. No whizbang tech is free. Come on 

Choosing TypeScript has huge implications for the rest of the stack, so I’ve listed it first. If you start off with TypeScript, you have to think about the TypeScript implications of every next technical choice. 

 will save you some of the time, but some projects are not TypeScript friendly and you’ll be incentivised not to choose them.

It may seem strange to include VS Code in the Tech Stack discussion, but the raw truth is that if you’re using TypeScript, you’re using VS Code. I grew up in the generation of developers who love to hate Microsoft. But I have to admit VS Code is the best IDE I’ve ever used. Ugh. I hate saying that. The ultra rich support for TypeScript is a killer feature. I still work in Vim/iTerm most of the time, but when I’m hacking React, I’m in VS Code (with Vim bindings!).

A screenshot of a text editor showing React code with TypeScript types.

React with TypeScript

React was the easiest of the choices for this project. Especially since Hooks have come out, React is just the clear, clear winner over everything else out there right now. In my last role, I ported an existing PolymerJS app into modern React and generally had an excellent experience with React. The tooling support is excellent. The React core team are beasts with incredible velocity. Couldn’t be happier with this choice.

Runners up are thing like AngularJS or VueJS, but honestly, it’s not a remotely close race.

A screenshot of the GraphiQL programming for building GraphQL queries.

Graphiql UI

GraphQL. Ooph. Where to start. Is it good? Yes. It’s really good. Especially with all the nice things you get from Apollo. Having pre-built hooks for queries and mutations is awesome. Being able to create queries that get you exactly what you need. Killer. When combined with TypeScript, this means that you get back typed, well structured data from your queries, which is pretty great. The downside is that GraphQL isn’t JavaScript or TypeScript. It’s GraphQL. Yet 

 language to support in a project that is, at least theoretically, JavaScript.

The GraphQL query UI is fantastic. Super useful for debugging and for composing exactly the right query. No Swagger or Postman UIs for our frontend engineers.

CSS in JS

Oh my goodness. Someone took CSS and found a good way to represent it in JavaScript? Shut up and take my admiration. This one is an easy yes. CSS-in-JS for life. Remember all the hype about CSS variables? Guess what’s better than CSS variables. Actual TypeScript variables!

Pro Tip™: Always alphabetise CSS declarations.

An oldie but a goodie. I’ve encountered the Metro bundler more recently in interacting with some ReactNative/Expo code and I’m curious how it compares performance-wise with Webpack. Still, I wanted something I knew and something that is reliable. The Webpack plugin ecosystem is extremely hit-or-miss, but the core tech is solid.

Still, when it slows down (which it will with all these bells and whistles and spinning rims), there is a huge body of well-documented Webpack optimisations to employ to reclaim that performance. I’m happy with Webpack for now, but I’d happily jump on something faster and simpler.

The biggest negative tradeoff of this whole setup is that it has so many damn languages in it. We have to have HTML, CSS, and JavaScript. It’s the web. Those are table stakes. React-JSS gives us the ability fold the CSS into the JS, which is awesome. So, that means there are only two languages, right? Wrong. We’re using TypeScript on top of JavaScript. And we’re using GraphQL in addition to that. Oh and don’t forget React’s JSX language. So, this means we have HTML, JSX, JavaScript, TypeScript, GraphQL all in a codebase that compiles into pure JavaScript. All this complexity isn’t free. It’s actually super expensive. Onboarding junior engineers or interns shows the cost. Our designer is a confident CSS hacker and he feels cautious around all this complexity. There is a lot of cognitive load to manage in understanding how the pieces fit together.

The flip side of this negative is that once it clicks, the codebase is pretty great. Having TypeScript underneath everything means that you can hover over just about any variable in VS Code and get a massive amount of context on what it is, where it comes from, what its types are. The constant sense of mystery you feel exploring a big JavaScript codebase isn’t something we feel. In fact, I can navigate this codebase faster than any JavaScript codebase I’ve ever worked in.

Another subtle tradeoff is that we’ve started from scratch. The upside of this choice is that we’re running the best-of-breed everything. Our codebase doesn’t suffer from any legacy ailments or poor technical choices (at least not that we know about yet!). No tech debt to pay off. Clean, linear commit history thanks to the 

. The downside is that we’ve had to fit all the pieces together ourselves. We didn’t start with any boilerplate project helper or 

. So that cost us some time. Still, this tradeoff seems easily worth it.