Simple Scheduling with Google Workflows

Background

As a small team, we’re always looking for reliable ways to maximise our current infrastructure. We (investa.co.uk) recently built a pipeline to fetch pricing data into our retail investment product. We needed a solution that would allow us to maintain our existing infrastructure without material changes to our engineering operations.

Our team relies heavily on Google Cloud Functions and Google Firestore to build our product. GCP provides a number of event triggers and bindings for Cloud Functions such as Cloud Pub/Sub and Firestore. For a simple pipeline, we often bind a Cloud Function to a scheduled CRON using Terraform. However, the one-minute minimum threshold was too long for the pricing pipeline, so we had to explore alternative options.

Existing solutions

There were a number of potential solutions that we had to evaluate as a team. Some of the more obvious ones included Cloud Composer (a managed instance of AirFlow), a containerised service such as Cloud Run or another streaming service such as Dataflow. We even considered using Cloud Tasks to schedule more frequent runs. 

Our approach

A less obvious solution, and one that we’ve found incredibly reliable, is Google Workflows. Google Workflows are simple, JSON or YAML-defined pipelines. They allow you to declare some basic control flow and data manipulation to orchestrate a pipeline. 

In our instance, we wanted to call our existing pricing update function every X seconds. To achieve this we defined a Workflow (which iterates continuously) and pauses for X seconds before triggering the next update. 

In addition to this loop, we used a cron to trigger the Workflow. This kicks off the initial run and can is also used as a failsafe against the Workflow failing.

Below is the Workflow for our pricing pipeline. The logic at the start allows additional cron-triggered runs to exit early. The remaining logic simply calls our pricing function, waits, and goes again.

The JSON is significantly more verbose. However, as we’re using terraform templates, we have found the parsing/output far more reliable than dealing with whitespace issues.

During execution GCP provides some additional debugging, such as consistent trace IDs (so long as you use the ones provided in the request headers) and a Workflow visualisation that looks like this:

Conclusion 

Our solution has only introduced one additional moving part (the Workflow itself) to our existing infrastructure. The Workflow is fully defined within JSON and Terraform. And we have so far seen zero operational overhead. 

In terms of documentation and tooling workflows do feel a little immature - but given the simplicity of our use case (faster iterations of Google Cloud Functions) it has been ideal.

For simple pipeline orchestration, we will continue to leverage Google Workflows. And we’ll continue to assess frameworks such as Airflow / Dagster as we scale.

Let us know if you’re using something similar or if this has been helpful.

Would you like to use modern serverless tooling such as Google Workflows? Our team is growing and we’re currently hiring both a Principal and a Senior Software Engineer. If you would like to know more about the role please reach out to ian@investa.co.uk or check out our cord profile.