The FreeAgent application is currently used by more than 100,000 companies. When users send their invoices, explain their bank transactions and do any other action to take care of their business, these actions are recorded automatically. We record them in different systems and for different purposes. One of the systems where we record actions like these is an event system and these records are called events. Events contain information about the action a user performed as well as information about who they are (e.g. their user ID, their company).
What is an event system?
In its simplest form, an event system consists of:
- Publishers: objects or applications that create and publish events
- Event Store(s): where events are stored temporarily
- Subscribers: objects or applications that read events from a store and do any further actions needed
This simple diagram shows visually the connection between the 3 parts of an event system.
What is an event?
One of the best definitions of an event in computing is found in Martin Kleppmann’s book Designing Data-Intensive Applications:
An event is a small, self-contained, immutable object containing the details of something that happened at some point in time. An event usually contains a timestamp indicating when it happened.
The FreeAgent use case
The event system in FreeAgent looks similar to the general case described above, with the FreeAgent application itself being the only publisher at the time of writing. Our current system looks like this:
As already mentioned, the FreeAgent application is the publisher, while our events store is made of RabbitMQ queues. The only direct subscriber at the moment is a Rails application we call the Legacy Data Warehouse. From there the event records get pushed to Amazon Redshift, which is their ultimate destination. The current event system has served us well so far. However, moving forward we want something that is more consistent, maintainable and extendable. The issues we currently face in reaching these goals are:
- We have more than one event type and serialisation format:
- a simple engagement event type recording the action as a string and some generic context (e.g. the user, the company), serialised with JSON
- two event types recording the action with additional data and the explicit context needed for each event, serialised using Apache Avro
- We do not have a single, consistent way of publishing events. This means we have a few different functions that can produce the same result and they are used inconsistently through our code base
Recently, we started refactoring our existing event system in order to fix all the issues mentioned above. This is a substantial task that will keep us busy for the next couple of months! We started the refactoring from the publishing. The reason behind this decision was to have an impact on the engineering team sooner, by creating an event system that looks more consistent and easier to use.
Event Publishing
To begin with we started reworking the event messages we send, and their structure. We decided to use two objects to capture all the information needed by the different types: an Event object and a Context object.
Event object
An Event object contains information about what happened and when. The fields we chose are inspired by CloudEvents. Each event contains:
- a name, which describes in a phrase what happened
- a unique ID
- a timestamp, which describes when the action happened
- (optionally) any additional information relevant to what happened
Context object
A Context object contains information about the circumstances under which something happened. This can contain:
- Information about the user and company, describing who did the action
- Information about the platform, describing where the action happened
In the case of FreeAgent’s event system, the context does not have to be present for all the events we publish.
Here is an example of how the two objects look in the case of an ‘invoice created’ event:
Event Payload
The Event and Context objects are combined to create the event payload, which is the message we are publishing. We use our internal formatters to make sure we will provide a serialised message that can be de-serialised from the Legacy Data Warehouse application. We currently have more than one formatters to be able to serialise the different types of events we have, but at the end of the full event system refactoring we will have only one, using JSON format.
Using the example from above, the event payload of the ‘invoice created’ event will be:
{
"event_name": "invoice created",
"event_id": "f8d5f236",
"company_id": 1234,
"user_id": 12345,
"event_timestamp": "15909696000000",
"data_source": "API",
"data": {
foreign_currency: false,
total _value: “100.00”,
}
}
This is the final payload published on our event queues.
Subscribers
Our only subscriber at the moment is the Legacy Data Warehouse application. This application subscribes to the event queues and listens for new event messages. When new messages arrive they are consumed, which in our case means de-serialising the content of the messages and storing them in a database. These events are periodically uploaded to Amazon Redshift by the Legacy Data Warehouse application.
In the future we want to add more subscribers. The events we are dealing with at the moment are engagement events, tracking usage of different features. In the next couple of months we plan to extend our event system to include the FreeAgent application itself as another subscriber as well as other applications external to FreeAgent.
Best Event Publishing Practices
Publishing and consuming events is a very important task for FreeAgent. These events are used to build business reports, test new features and create automated responses based on user actions or system measurements. All of the engineering teams in some way are publishing or consuming events to do the above.
For this reason, while refactoring the publishing part of the event system, we compiled a set of rules to ensure best practices are followed when working with events. This guarantees correct usage of the system, consistency and maintainability.
Here is the list of rules that make our best practices for publishing events:
When talking about the event system or writing code, make sure a common and ubiquitous language is used
This is an important principle of domain-driven design. In cases like this where there are a lot of ways to describe similar things, we decided to create a glossary to ensure a common language is used.
The event name should be a brief description of the user action
In FreeAgent, in order to create consistent event names, we decided to follow the naming convention ‘<noun> <verb>’. Desirable event names are ‘invoice created’, ‘user logged in’, ’email sent’, and we discourage using names like ‘enable bank feed’.
Multiple events should not be published for the same action
When there are any specific details about the action, they should be placed in the data field of the Event object. For example, when an invoice is created in a currency other than the company’s native currency, we should publish one ‘invoice created’ event with ‘data: { foreign_currency: true }’ rather than 2 events for ‘invoice created’ and ‘foreign currency invoice created’. This makes events more future proof and keeps the consistency of what actually happened.
Event publishing should be tested at the level the event is being published
Testing event publication in feature tests is only desirable if the consumed event produces something in the UI. For maintainability reasons, we prefer testing the publishing of the events alongside data validity in integration tests, for example while testing a controller action for a controller that publishes events.
In the coming months when the refactored event system will be used more by the engineering team, this list of rules may be adjusted or extended to fit the team’s needs.
Conclusion
We hope you now have a better understanding of what an event system is and insight into how we use it at FreeAgent. The project of refactoring and extending the FreeAgent event system is still underway and we are sure that we will continue learning and making adjustments as the project progresses.