How we publish user events for 100,000 customers

Posted by on September 15, 2020

The FreeAgent application is currently used by more than 100,000 companies. When users send their invoices, explain their bank transactions and do any other action to take care of their business, these actions are recorded automatically. We record them in different systems and for different purposes. One of the systems where we record actions like these is an event system and these records are called events. Events contain information about the action a user performed as well as information about who they are (e.g. their user ID, their company).

What is an event system?

In its simplest form, an event system consists of:

  • Publishers: objects or applications that create and publish events
  • Event Store(s): where events are stored temporarily
  • Subscribers: objects or applications that read events from a store and do any further actions needed

This simple diagram shows visually the connection between the 3 parts of an event system.  

Event System Overview

What is an event?

One of the best definitions of an event in computing is found in Martin Kleppmann’s book Designing Data-Intensive Applications:

An event is a small, self-contained, immutable object containing the details of something that happened at some point in time. An event usually contains a timestamp indicating when it happened.

The FreeAgent use case

The event system in FreeAgent looks similar to the general case described above, with the FreeAgent application itself being the only publisher at the time of writing. Our current system looks like this:

FreeAgent’s Event System

As already mentioned, the FreeAgent application is the publisher, while our events store is made of RabbitMQ queues. The only direct subscriber at the moment is a Rails application we call the Legacy Data Warehouse. From there the event records get pushed to Amazon Redshift, which is their ultimate destination. The current event system has served us well so far. However, moving forward we want something that is more consistent, maintainable and extendable. The issues we currently face in reaching these goals are:

  • We have more than one event type and serialisation format: 
    • a simple engagement event type recording the action as a string and some generic context (e.g. the user, the company), serialised with JSON
    • two event types recording the action with additional data and the explicit context needed for each event, serialised using Apache Avro
  • We do not have a single, consistent way of publishing events. This means we have a few different functions that can produce the same result and they are used inconsistently through our code base

Recently, we started refactoring our existing event system in order to fix all the issues mentioned above. This is a substantial task that will keep us busy for the next couple of months! We started the refactoring from the publishing. The reason behind this decision was to have an impact on the engineering team sooner, by creating an event system that looks more consistent and easier to use. 

Event Publishing

To begin with we started reworking the event messages we send, and their structure. We decided to use two objects to capture all the information needed by the different types: an Event object and a Context object.

Event object

An Event object contains information about what happened and when. The fields we chose are inspired by CloudEvents. Each event contains:

Event object
  • a name, which describes in a phrase what happened
  • a unique ID
  • a timestamp, which describes when the action happened
  • (optionally) any additional information relevant to what happened

Context object

A Context object contains information about the circumstances under which something happened. This can contain:

  • Information about the user and company, describing who did the action
  • Information about the platform, describing where the action happened

In the case of FreeAgent’s event system, the context does not have to be present for all the events we publish. 

Context object

Here is an example of how the two objects look in the case of an ‘invoice created’ event:

Event and Context object example

Event Payload

The Event and Context objects are combined to create the event payload, which is the message we are publishing. We use our internal formatters to make sure we will provide a serialised message that can be de-serialised from the Legacy Data Warehouse application. We currently have more than one formatters to be able to serialise the different types of events we have, but at the end of the full event system refactoring we will have only one, using JSON format. 

Using the example from above, the event payload of the ‘invoice created’ event will be:

{
    "event_name": "invoice created",
    "event_id": "f8d5f236",
    "company_id": 1234,
    "user_id": 12345,
    "event_timestamp": "15909696000000",
    "data_source": "API",
    "data": {
        foreign_currency: false,
        total _value: “100.00”,
    }
}

This is the final payload published on our event queues.

Subscribers

Our only subscriber at the moment is the Legacy Data Warehouse application. This application subscribes to the event queues and listens for new event messages. When new messages arrive they are consumed, which in our case means de-serialising the content of the messages and storing them in a database. These events are periodically uploaded to Amazon Redshift by the Legacy Data Warehouse application.

In the future we want to add more subscribers. The events we are dealing with at the moment are engagement events, tracking usage of different features. In the next couple of months we plan to extend our event system to include the FreeAgent application itself as another subscriber as well as other applications external to FreeAgent.

Best Event Publishing Practices

Publishing and consuming events is a very important task for FreeAgent. These events are used to build business reports, test new features and create automated responses based on user actions or system measurements. All of the engineering teams in some way are publishing or consuming events to do the above.

For this reason, while refactoring the publishing part of the event system, we compiled a set of rules to ensure best practices are followed when working with events. This guarantees correct usage of the system, consistency and maintainability.

Here is the list of rules that make our best practices for publishing events:

When talking about the event system or writing code, make sure a common and ubiquitous language is used

This is an important principle of domain-driven design. In cases like this where there are a lot of ways to describe similar things, we decided to create a glossary to ensure a common language is used.

The event name should be a brief description of the user action

In FreeAgent, in order to create consistent event names, we decided to follow the naming convention ‘<noun> <verb>’. Desirable event names are ‘invoice created’, ‘user logged in’, ’email sent’, and we discourage using names like ‘enable bank feed’.

Multiple events should not be published for the same action

When there are any specific details about the action, they should be placed in the data field of the Event object. For example, when an invoice is created in a currency other than the company’s native currency, we should publish one ‘invoice created’ event with ‘data: { foreign_currency: true }’ rather than 2 events for ‘invoice created’ and ‘foreign currency invoice created’. This makes events more future proof and keeps the consistency of what actually happened.

Event publishing should be tested at the level the event is being published

Testing event publication in feature tests is only desirable if the consumed event produces something in the UI. For maintainability reasons, we prefer testing the publishing of the events alongside data validity in integration tests, for example while testing a controller action for a controller that publishes events.

In the coming months when the refactored event system will be used more by the engineering team, this list of rules may be adjusted or extended to fit the team’s needs.

Conclusion

We hope you now have a better understanding of what an event system is and insight into how we use it at FreeAgent. The project of refactoring and extending the FreeAgent event system is still underway and we are sure that we will continue learning and making adjustments as the project progresses.

Leave a reply

Your email address will not be published.