A brief introduction to ‘the cloud’ and managing infrastructure with code

Posted by on August 31, 2022

Over the last decade the ‘cloud’ has become increasingly prevalent . A cloud based system allows a  company to flexibly buy servers, storage, networking and various other services that are hosted externally rather than on-site, typically with a programmatic interface to allow large-scale use. 

According to a 2019 report, 94% of companies were utilising the cloud in one way or another. The market for cloud providers was valued at $200 billion by the end of Q2 2022, with expectations for it to increase drastically over the next decade.

There are a number of different cloud providers currently in operation, each constantly creating new innovations to standout from the rest. The main players are currently AWS, GCP and Microsoft Azure. These three companies alone capture 65% of the available market. While each one provides similar services, they all have very different ways of deploying and managing their resources.

What are the advantages of cloud systems?

Zero Wires Zero Worries

One of the main advantages for companies that run the majority of their services on the cloud is the reduced upkeep of the hardware for their servers, storage and networks. Instead, an office only has to worry about its own power and internet connections.

Autoscaling

Another benefit of using cloud hosting is the ability to auto-scale. Companies can experience various situations where a larger than usual volume of users wants to access their online resources. This can be due to DoS (denial of service) attacks or simply a large influx of customers from a new marketing campaign. 

Cloud hosting enables providers to reduce the impact of these periods of high demand automatically by deploying more resources to maintain the normal functioning of their services.

Managing infrastructure effectively with code

Using a cloud provider’s UI to create infrastructure by hand is simple but it introduces risks. It’s easy to overlook how something should be configured and this method makes auditing difficult. In addition, you can end up introducing configuration drift over time which makes it harder to link different resources together. It’s also time-consuming to create many instances of a resource.

We can mitigate these issues by managing infrastructure as code.

What are the benefits of using Terraform to manage infrastructure?

Terraform is a declarative programming language that describes infrastructure. Unlike most standard languages, developers don’t have to describe how a resource will be deployed. Instead, they state the resource desired outcome in code and Terraform will work out what steps to run.

Resources to deploy are defined in a folder of files with the .tf extension. Terraform uses this configuration to create a state file when the command terraform apply is run, which is used to directly track the resources that have been deployed. This enables commands such as terraform destroy to be used to destroy the relevant resources that are being managed by Terraform.

Below is a part of a Terraform configuration that can be used to create a simple EC2 instance (virtual server).

Simple AWS Instance using Terraform

This is a very simple example of what an EC2 instance will look like in Terraform. In practice, there will usually be a lot more configuration required to enable it to do very specific tasks and communicate with other resources.. The extent of the configuration capability can be found on Terraform’s website here. Terraform’s documentation is generally very comprehensive, making it very useful when developing code for it.

What if you want to change configurations once resources have been deployed with terraform? Terraform has made this extremely simple whereby you can alter the terraform files you have written. Once changes are made to the files, terraform apply will be run again and these changes will be made in your cloud service.

What other tools does Terraform have?

Terraform has a huge ecosystem of developers trying to improve and innovate the language and concepts behind it. There’s a variety of projects both open source and closed source that can be utilised to improve IaC (Infrastructure as Code) workflow efficiency.

Terragrunt by Gruntwork

Terragrunt is a thin wrapper that helps prevent the repetition of Terraform code. 

Creating resources using Terraform requires various bits of information about your AWS account in order to run. Using Terraform alone means developers have to state this backend information every time they want to apply a configuration.

Terragrunt enables users to deploy resources from a terragrunt.hcl file. This file contains blocks of code that allow you to pull in configuration from other places e.g. enabling the backend configurations to be stated in a single place and re-used.

Below is an example of what a terragrunt.hcl file will look like.

Basic terragrunt.hcl file

The Terraform block points to the location of a module that contains Terraform code.

The ‘include’ block tells terragrunt to look through parent folders to find any other configuration. Typically this will include at least the backend configuration in a shared root configuration, which will save developers a lot of time.

The ‘inputs’ block can be used to alter any of the predefined configuration using the variables within the module. It’s the Terragrunt equivalent of Terraform’s tfvars files.

You would now run terragrunt apply to deploy the resources, instead of using Terraform directly. This will pull in all the Terraform code and insert the extra configuration provided by Terragrunt, and then run Terraform for you.

Terragrunt’s modules

In addition to Terragrunt augmenting Terraform usage, Gruntwork, the company behind the tool, has a huge number of Terraform modules that provide configuration for a wide range of best-practice cloud services. These can be deployed by inputting a git ssh link to the source section at the top of the terragrunt.hcl file. These modules can be used to create VPCs or load balancers, for example, which can then be linked together in your own code.

Account cleaner

Tools such as cloud-nuke and aws-nuke are becoming increasingly popular to implement within organisations. These tools will ‘nuke’ your account, meaning they will destroy all resources that the tool is configured to destroy. 

When to use the nuke function

We mentioned the terraform destroy command above, which can be used for destroying resources managed by Terraform. However, this can’t necessarily remove everything you might want to destroy. 

For instance, you might create a resource with Terraform that then creates another resource, or someone might use the console to add things to your system. These resources won’t be stored in the state file for Terraform to destroy. As resources in the cloud cost money to run, you’ll want to ensure they are tidied up. These tools will iterate over all the resources in an account and then delete them.

How to use the nuke function

The tool is configured using a configuration file such as nuke.yml. An example of an aws-nuke configuration file is shown below.

Example configuration file for aws-nuke tool

As you can see there is a large amount of customisation available in this tool. This example is for aws-nuke to find every eu-west-1 and global resource in AWS account 565872034739, excluding the named policy and any S3 objects from deletion. It will be impossible to nuke anything in account 467527364293. 

Ensuring that careful testing is done when creating the configuration file is extremely important. Poor configuration can lead to entire production accounts being destroyed, and can have severe effects on the organisation.

Summary

Terraform and the tools surrounding it ensure developers are able to produce resources quickly and follow best practices. There can be a bit of a learning curve when getting started with these tools but, once your code is written once, it’ll be as simple as linking your module and providing some input variables for terraform apply to deploy your resources. 

At FreeAgent, infrastructure is entirely managed following the IaC principles. Resources are never deployed in production accounts without Terraform code, and the tools outlined above are used for development, testing and deployment for reliable infrastructure.

What is a tech spec and why do I need one?

Posted by on August 26, 2022

For the past three months, I’ve been a Software Engineering Intern in the Connections & Add-ons team here at FreeAgent. We look after a number of features that allow FreeAgent to work with third-party services: if a business owner wants to get paid faster, they can add a PayPal, Stripe, GoCardless or Tyl payment link to their invoices; if they need to get their business insured, they can sign up for a Hiscox insurance policy directly through FreeAgent; and if they’re looking for an accountant to work with, they can sign up with one of our CoPilot practices.

Recently, I’ve been working on writing a tech spec for the work that we’re planning to do in the next few weeks. We want to give the Connections & Add-ons page a rejig, a lick of paint, and then introduce PayPal as an option to sit alongside the other payment providers. Currently, the page looks like this…

…but the new update is coming soon.

As mentioned, I’ve been working on a tech spec for some of this work related to moving PayPal from one area of the app to our newly designed Connections & Add-ons page. I thought I’d give you an insight into what I’ve learned whilst writing my first tech spec as a FreeAgent intern. So, talking of tech specs…

What is a tech spec?

A tech spec – short for ‘technical specification’ – is a document that outlines a problem to be solved, why it needs to be solved, the requirements of the solution, and the proposed solution to the problem – essentially it’s the What, Why and How of any software engineering related problem.

As the name suggests, a tech spec can be quite technical and it’s usually written by an engineer. Once written, it can then be shared with the wider team (fellow engineers, product managers and designers), who will discuss solutions to specific issues, answer any open questions and offer suggestions around the proposed implementation.

Another important aspect of a tech spec is that it outlines any other stakeholders involved in the project. For example, a stakeholder could be another team within FreeAgent or it could be an external third party.

Why do I need a tech spec?

When starting a new project or piece of work, it is very tempting to jump straight in and start writing code (I’m definitely guilty of this!). However, the planning stage of any software development project is just as important as the actual coding stage.

Most software projects follow a similar pattern:

💡 Discovery – what is the problem that we are trying to solve?

 ✏️ Design – how might a solution to this problem look?

📄 Technical specification – what do we need to consider and how are we actually going to solve it? How are we going to break this work into manageable chunks?

🔎 Refine – have we covered everything? Are there still any open questions?

🏗️ Build – time to write some code…

✅ Deploy – time to share your new features or changes with users.

Notice that the tech spec stage is after Design but before Build. If we were using a very linear software development process (for example, the Waterfall model) and we got our design and planning stage wrong, it could be time-consuming and expensive to make changes during the Build stage. If you instead use the Agile methodologies, it can be a lot easier to make last minute changes, however, it’s always best to have a solid plan of action and anticipate any blockers from the start – this is exactly what a tech spec is for.

Another benefit of a well written tech spec is that it keeps everyone in the team on the same page. My team, like most teams at FreeAgent, isn’t solely made up of engineers – we have a product manager and a designer too. When embarking on a new piece of work, there are a lot of engineering, product and design considerations that need to be taken into account and it can be useful to summarise these in one document whilst thinking about the implementation.

How do I write a tech spec?

You can choose to write your tech spec in a few different ways but most tech specs follow a similar format. In the Connections & Add-ons team, we have a good template so I decided to closely follow that, therefore, my tech spec was made up of the following sections:

Summary and motivation

What are you doing and why are you doing it?

Even if the reader isn’t familiar with the project, they should be able to read this section and be left with a good understanding of what you are attempting to achieve. If possible, try to make this section readable to a wide range of audiences (i.e. not just fellow engineers).

In my case, this started with something like: “We want to introduce a PayPal card on the Connections & Add-ons page so that users are able to see and compare each payment provider side by side.”

Considerations

Is there anything that you need to think about before making any changes? Does the project impact other areas of the codebase?

If you’re working on a fairly large codebase, like FreeAgent’s, it’s likely that your work will touch a number of other areas within the codebase. If this is the case, you should outline what impacts your work will have on those other areas and what you plan to change. Also, if there is some technical debt that needs to be cleaned up before you begin your work, this is a good place to talk about that.

As I mentioned before, your tech spec will likely have a list of stakeholders. If your work affects another area of the codebase, then the team that usually looks after that area should become a stakeholder in your project.

For my tech spec, this went something like: “As PayPal currently exists within the ‘Banking’ area of FreeAgent, we will have to ensure that users are redirected back to the correct section on the Connections & Add-ons page once PayPal has been successfully set up.”

Scope

What will and what won’t you be changing? 

When you begin coding, it’s easy to get carried away and build or change more features that you were initially planning to do – this is known as scope creep. In this section of your tech spec, you should set some boundaries so you don’t end up with scope creep and you manage to finish the piece of work on time.

In my tech spec, this went something like: “We will be keeping the current code that handles the connection to the PayPal API; however, we will be moving the PayPal payment provider option from the Banking area to the Connections & Add-ons page.”

Related work

Is this piece of work related to something else? Is it a small part of a bigger project? Is there some future work planned that is based on this work?

If you’re working on a standalone piece of work, then you could probably skip this section. However, if your work is part of, or closely related to, another piece of work or project, then you should give a brief summary of that and how this piece of work fits into the overall goal of the wider project.

In my case, this went something like: “We want to bring PayPal to the Connections & Add-ons page so that users can easily compare between all of our available payment methods. This piece of work is part of a wider refactor and redesign of the Connections & Add-ons page to ensure that it is easy to navigate the page and users can compare all of FreeAgent’s available integrations.”

Implementation

How exactly do you plan to build this feature?

If you love to talk about code, then this is your time to shine – the more detail the better. In this section you should go into great detail about what changes you’re going to make to the codebase and how these will help solve the problem.

For my tech spec, this section was quite large. It covered some topics including:

  • Creating a new controller (as we use Ruby on Rails/MVC at FreeAgent) and the list of methods that the controller will have and what each method will do
  • Making changes to the app’s routing so that requests are sent to the newly created controller
  • Creating the new HTML views and making changes to the existing view on the Connections & Add-ons page
  • Writing new RSpec tests
  • Creating a new Knowledge Base article with screenshots and updating any existing ones

What’s next?

Now that you’ve got a solid plan of action, you’re ready to start building! During the build stage you can regularly refer back to your tech spec to check that you are on track to implementing everything that you initially set out to do.

Whilst the PayPal work is still in progress, the team and I have regularly checked our progress against the tech spec to keep us all on track. The newly designed Connections & Add-ons page and PayPal integration will soon be live on FreeAgent!

The 4 SQL queries you need to debug Redshift performance

Posted by on August 25, 2022

Redshift is Amazon’s data warehouse product available on AWS. To help users understand the contents and performance of their Redshift cluster, users are provided with some tables and views, including:

This blog focuses on the Redshift-specific objects, and the queries that I often come back to when I need to look “under the hood” to try and understand and improve performance issues.

Understanding the Redshift-specific system objects

All Redshift system objects have prefixes on their name which follow a naming convention:

  • First, they have S for “system object”
  • Then they have either T for “table”, or V for “view”, depending on the object type
  • Finally, they have either L_ for “logs”, or V_ for, erm, “snapshots” (I haven’t been able to find anything which explains why this isV_ and not S_!)

That gives you object names like STV_INFLIGHT which is a system table with a (v)snapshot of the current queries running on the cluster. Simple! Now we understand that, let’s look at the SQL itself.

Queries

This SQL snippet tells you what’s actively running on the cluster (you need to be a superuser to see all queries running):

SELECT
  userid,
  query
  pid
  starttime
  text
FROM stv_inflight;

The text field will only store the first 200 characters of the queries running, but the real benefit here is that you can identify long-running queries. If you have a query you need to kill for any reason, you note the pid from the results, and run pg_terminate_backend(pid) to stop it running.

If you need the full text of the query, you can make a note of the query from here, and plug it into this SQL:

SELECT 
  q.query,
  q.starttime, 
  t.text,
  t.sequence 
FROM stl_query q
INNER JOIN stl_querytext t ON q.query = t.query
WHERE q.query = 1234
ORDER BY sequence;

Locks

If you are seeing an unusually long-running operation, it may be in a deadlocked transaction, which is visible in the STV_LOCKS table. Again, you can use the lock_owner_pid of the relevant sessions along with pg_terminate_backend(pid) to kill the offending session (start with the oldest lock first):

SELECT 
  table_id, 
  last_update, 
  last_commit, 
  lock_owner_pid, 
  lock_status 
FROM stv_locks
ORDER BY last_update;

Query/Table Performance

If you are digging around the system tables, you’re probably unhappy with the performance of something. There’s a series of objects which are aimed at providing useful information.

SVV_TABLE_INFO

SVV_TABLE_INFO provides information about state of your tables, which can used to troubleshoot poor-performance. I use this view as a guide for when to VACUUM a table. I tend to just select all columns from the view:

SELECT *
FROM svv_table_info
ORDER BY database, schema, table;

Here are some of the columns in the results I look at to see if there are any “low hanging fruit” with respect to improving performance:

  • empty shows how many blocks are waiting to be freed by a vacuum.
  • unsorted shows the percent of the table that is unsorted. The cluster will need to scan this entire section for every query. If this creeps up too high you need to vacuum to re-sort the table.
  • skey_sortkey1 shows the ratio of the size of the first column of the sortkey to the size of the largest non-sortkey column. You can use this value to evaluate the effectiveness of the sortkey.
  • skew_rows shows the ratio of rows from most on a slice to least on a slice. Use it to evaluate distkey.
  • pct_used shows the amount of disk that the one table occupies. Use it to review if a table, or portion of it, could be stored in S3 or elsewhere. This might be the case for less frequently accessed data, or if you have “backup” tables.

STL_ALERT_EVENT_LOG

The STL_ALERT_EVENT_LOG table is important for optimising queries. When the cluster executes a query, it will record issues found by the query planner into STL_ALERT_EVENT_LOG, along with suggested fixes.

Some problems can be fixed by running ANALYZE or VACUUM, while others might require rewriting the query or how you store your data. Amazon themselves can advise you there, with their table usage notes.

Again, I tend to just select all the columns from the table — the solution column will tell you in plain language what will improve performance:

SELECT *
FROM STL_ALERT_EVENT_LOG;

I hope you find these queries useful. Feel free to connect with me on LinkedIn, and leave your comments and suggestions below.

So you’re going to be an intern

Posted by on

After months of nerves, interviews and one too many LeetCode questions, you’ve done it. You’ve landed an internship, you’re free from the dread of a long, empty summer and your brain is filled with vague notions of what your internship might be like.

I’m here to gather up your brain clay and sculpt it into solid ideas of what things will be like. Before we start, it needs to be said that your internship experience will differ from mine (even if you come to FreeAgent), but don’t click away. From discussing with friends, reading online, and my own experience there are commonalities between all software engineering internships. After all, you’re working as a software engineer, albeit with the stabilisers on.

The things you learn in your internship, by their very nature, are fundamental. Reviewing code, working in a scrum environment, deploying to production, technical writing and using Git are all skills you will use if you pursue a career in software engineering.

So let’s dive in…

You’re going to be useless for a little while

Your first couple of days or even your first week will be spent onboarding. This is true of almost all companies that will be offering internships (unless you’re working at a small start-up). You’re going to have to get used to a lot of different systems. On a typical day of writing code, I use a couple of systems without really thinking about them. The first is FreeAgent’s Devkit, which is a system that allows us to develop our application locally. By running Devkit I can launch a version of our website locally. I can make up crazy data and no one will know! Having local development tools is extremely important because you can change the code and see how it affects the app on your own computer like it’s your own personal playground.

However, they can be a pain to set up. Do not feel stupid when setting them up. Even the guide you might be given (or god forbid, some terse documentation) won’t be 100% clear. You’re going to encounter bugs that aren’t in the guide, and you’ll have to sift through extra documentation or ask for help.

Overall you’ll spend a lot of the first week setting things up. It can be frustrating, but once it’s done, it’s done and you’ll never have to look at the stuff again and will be off writing code. Someone’s probably going to have to help you with that…

You’ll probably have a buddy 

Yes, a buddy (possibly a friend). Most companies will assign you a buddy. This person is usually a senior engineer who has either volunteered or been coerced into helping you get going. Jokes aside, you’ll hopefully find that your buddy is extremely knowledgeable, affable, and infinitely patient – or at least mine was (thanks Chris!). Your buddy will be the one-stop shop for stuff you can’t google.

Things like: “why doesn’t this piece of company-specific software work?”, “sorry but what exactly is a ….?”, and of course, “I think I messed up a rebase”. Don’t worry if you don’t know what a rebase is yet. We’ll get to those later and – trust me – you’ll become familiar with them. 

At the beginning you may feel extremely clingy. Don’t worry, this is normal. You’re being inundated with lots and lots of information and are spending hours each day setting up company-specific applications and learning about other applications you’ve never really used before. At this stage I think it’s helpful to set up a few rules about asking for help. You should have a certain amount of “struggle time” before you ask for help. But when it comes to setting things up and getting the lowdown on new technologies there’s no need to bash your head against a wall, googling for hours when your buddy could tell you how something works in five minutes. Similarly, if you think you can probably google it, try that first.

Your buddy’s ultimate goal is to get you writing code, I suspect that’s what you want too. Brilliant – let’s get to it.

Writing your first line of code

You’re finally here. After several days of setup, two broken mice, and your buddy muting you on Slack, you’re ready to write your first lines of code. There’s just one thing in your way – the codebase. At first the codebase is an indistinct endless sea. White waves roll violently, shaking your boat and you’re a little bit daunted. Yet your buddy stands tall, manning the sail and staring into the distance like they can see something you can’t. They can.

Chances are that the codebase you’ll be working on is huge. FreeAgent in all its various forms comprises around a million lines, let’s just say that it’s going to be a lot. Hundreds of times larger than anything you’ve ever worked on before. I’m sure this will be true of whatever company you’re working at. 

Your first change will be something small. Mine was updating the text on a page. But that took me longer than expected. The first hurdle was finding the corresponding code file. The problem was twofold; one, I barely knew ruby, let alone rails. Two, I had no idea how to navigate the codebase. Both of these things come with time.

However, you can learn to navigate the codebase much faster if you take some time to get to grips with it. I’d suggest picking an area of the application at random and trying to find the corresponding file. You could do this by searching for the filename, or searching for text that is on the page. Code editor tools are your friends. Once you’ve found the file, and made the change, things are far from over. Unlike most university or personal projects, solving the problem isn’t the end of it, it’s only the beginning…

You’re going to need to test

You’ve done the work but there’s so much more to do. Universities rightly foist testing onto their students and turn a blind eye when the unit-testing gang claims another victim, but they never really go into why testing is so important. “Yeah yeah yeah, to test my code for things that I won’t notice,” you regurgitate, as you roll your eyes. The problem is that we’re never really given scenarios where testing is truly useful. Projects that involve testing are often contrived or don’t lend themselves to testing. It’s no surprise that being given a hammer when you don’t actually have a nail makes you question the need for a hammer. However, internships involve nails, lots of nails.

During my internship I’ve gone from seeing testing as a necessary (but often superfluous) evil to the thread that holds my sanity together. It boils down to this. Usually when working on a smaller project, you’re the one making all the changes. Even if you aren’t, the codebase is small enough that if someone else makes a change you can see it quickly through your normal use of the application, or it affects what you’re working on. Neither of these things are true when you’re working at scale. Someone else’s work probably won’t affect the area of the application you’re working on, and you certainly won’t find all the new bugs they’ve introduced by clicking through every possible path in the application. 

As I see it, tests are there for two reasons; to check what you’ve done is right and to make sure that if someone breaks your code they’ll know about it. In web development the former often seems redundant, that is until things become complex enough. Time and time again I am caught out by my own tests. You don’t necessarily need to follow test driven-development, but you will find that thinking about what could go wrong helps you solidify your mental model of the code and what it ought to do.

So write your tests. It may feel like pulling teeth when making small changes but tests will soon become your greatest allies, helping you stay on top of everything. In fact, if you’re good enough at testing you won’t need to look at the application. This was one of the most surprising aspects of my internship. It seems natural to check that all your changes work as expected in the application, and this is done, but not by you. We’ll get into this later when we talk about publishing your code. In theory your goal should be that your tests are so comprehensive that you never need to look at the application itself. So, you’ve written your magnificent code and changed that text. Your tests are looking great. Surely someone is ready to present you with a medal now? No, it’s time for some feedback.

Looks good to me

The next step is to have your code reviewed. At this stage the senior engineers descend from their hallowed environs to assess your efforts. They’ll take out their heavenly weighing devices and prod your code with their tridents until it falls apart. This is good, even though it may be embarrassing. When you’re starting out, even on the most basic changes there will be things that even experienced engineers would miss, such as company code style decisions, new language idiosyncrasies or code structure.

You’ll have to take it on the chin, code reviews don’t just stop you polluting the codebase, they are also extremely effective learning opportunities. As you progress you’ll be working on larger, more complicated changes, and getting even more feedback. Don’t be alarmed if there are 15 comments on a change and three rounds of changes – these things happen. Sometimes the feedback won’t quite make sense, or even be incorrect. It’s okay to ask clarifying questions! To figure that out you had to know what you were doing well enough to explain and justify your decisions; an extremely important ability to have.

Git good

Git is a version control system that you’ve probably encountered before. It allows multiple developers to work on several different parts of a codebase at once without catastrophe. You’ve probably used basic Git commands before: push, pull, perhaps even merge. But there’s a whole new world out there. The one you’ll use the most is rebase. Rebasing allows you to squash together commits, change which commit your branch is based on, edit commit messages etc. It’s essentially a big Git sledgehammer; extremely powerful if you know what you’re doing, disastrous if you don’t.

For example, say you have 10 commits for a disgustingly minor change, you could rebase and squash all these commits together into a single commit. Your commit history probably isn’t something you’ve thought too much about before, but when working on a commercial codebase it’s important that you don’t pollute the codebase with a thousand commits, and that your commits have descriptive messages. A personal favourite commit message: “Stops evil behaviour”. Git rebasing can be dangerous though. It feels more like bomb disposal than managing how your code is stored. You have the power to delete commits. You can also stick your branch onto the wrong branch and create a nightmare of merge conflicts, so be careful. Once your code is up on GitHub as one nice commit (with the several hundred others needed to make the requested changes bulging underneath it) you have only one step left…

Manual testing and malaise 

Once your code has been reviewed and passed all other automated tests, someone has to take a look at how it performs in the app. For my first deployment, I did this myself without thinking. It was only when I was told to write the ‘PPT steps’ ( pre-production testing) that I realised that I’d done someone’s work for them. It’s best to have someone else look at how your changes affect the app to reduce errors. They haven’t been working on the problem and they’ll probably be able to spot things you missed out.

Furthermore, they’ll be following your instructions to the letter. If they can’t, then how you’ve implemented the functionality is probably incorrect, or you don’t understand the changes well enough to describe their functionality! It’s unusual to find too many errors at this stage, but it’s quite common to be told that your changes don’t quite match the design specification or that things aren’t reloading how they should be. 

Fin

That’s it, you’ve done it and you’re ready to deploy your code. It was quite the journey and will probably be a bit underwhelming and tedious for your first deployment. It gets better. In this post I’ve tried to give a brief overview of the parts of my internship that I didn’t find immediately obvious, but that are likely to be common to most software engineering internships. 

An internship is like a three-month interview; for you and for the company. Naturally, you want to produce your best work and appear in the best light. Don’t let this burden you. Ask questions and be open. If you truly weren’t competent you’d have been weeded out in the application process or someone would be letting you know by now. Take time to consider the company. Would you want to work there if they blessed you with a grad offer? Why? As your first experience in the industry an internship can let you know what you do and don’t like. Everyone’s internship is different, and every company is different. You should think about what the company does well and what it doesn’t. I know this is difficult to do when you’re forever knee-deep in a code swamp. Every so often you should come back onto solid land and assess. 

In the end you should just try and enjoy it. You’ll only ever be completely green once, and no one expects you to know everything. Make use of that, you’re being moulded by your experiences whether you like it or not. So go out and seek good ones, spend days debugging and hours failing. That’s where you’ll learn the most.

CSS Day conference, June 2022

Posted by on August 24, 2022

In early June 2022 we (Laurence and Ryan from the Design System team) visited Amsterdam for CSS Day. Despite the name this is a two-day event which, while CSS-focused, covers a variety of interesting front-end and user-centric ground.

And what a fantastic event! The venue, Zuiderkerk, is magnificent and the hospitality, professionalism and production values made for a special experience.

The stellar cast of speakers included CSS spec implementers, specialists from Google Chrome and Figma, Design System leads, W3C Technical Directors and others working at the cutting edge. There was a mix of seasoned pros alongside relative newcomers. And the audience too was very knowledgeable. This definitely felt like a conference by CSS nerds for CSS nerds!

CSS is currently moving at a startling pace. New and built-for-purpose responsive layout technologies have changed the game. Custom properties have ushered in API-like capabilities for theming components. Ideas previously written off as impossible – such as Container Queries and the parent selector – have been specified and shipped in browsers. And much of this is because we’re in a new era of collaboration between browsers.

This makes it a really exciting time to be writing CSS. Furthermore, we can influence its direction. A key takeaway from the event was just how close we as “normal developers” are to the creation of the HTML, CSS and JavaScript standards. Numerous speakers, including prolific spec-writer Tab Atkins and the CSS Working Group’s Rachel Andrew, were at pains to point out how organisations like the W3C need our feedback, issues, bug reports and pull requests on their proposals in order to shape features and specs. This access feels very empowering and only makes our job more exciting.

Our highlights

Jeremy Keith: In and out of style

Jeremy Keith kicked things off with a fascinating and thought-provoking talk, setting us up nicely for the rest of the conference.  We were taken on a trip from the very early days of the web to the present day, understanding how we got here through the concepts of path dependence, agreement and a shared context. Just as was done then, we are creating the future right now with the decisions we make today.

On our trip back through the years, we were reminded of all the “hacks” we’d have to employ to achieve what can be done with pure CSS today – border radiuses, gradients, custom fonts etc. He reflected on how third-party libraries like jQuery and Sass paved the way for vanilla JS and CSS features. There is a train of thought around material honesty and how we should abstain from non-native solutions, but as we know, people are going to do it anyway. Indeed, maybe this isn’t a bad thing – hacks can push the web forward. Reflecting on how we may take this approach into FreeAgent, workarounds could be implemented if accessibility and usability are sound, with performance and maintainability being weighed against native alternatives. Jeremy suggests we get behind Open UI – an initiative aimed at providing the customisable, extensible HTML elements that so many of us want but which are not natively available.

Check out Jeremy’s presentation

Michelle Barker: Creative CSS Layout

Michelle’s talk leaned into bleeding-edge CSS such as Container Queries, the aspect-ratio property and the :has pseudo-class and how to combine these to create context-sensitive, adaptive components. But just as interesting was the audience question asking how we should communicate all these amazing capabilities to our designer colleagues. Michelle couldn’t answer it alone and it highlights that we, as an industry, must find ways to build bridges and share knowledge between designers and front-end developers. CSS (and HTML) in 2022 can do so much yet in many cases our interactions with designers remain waterfall. The modern capabilities of the web should inform our designs; our conversations and collaborations should be push and pull.

Check out Michelle’s presentation

Bramus Van Damme: The CSS Cascade, a Deep Dive

Bramus taught us everything we wanted to know about the cascade but were afraid to ask! He also introduced Cascade Layers which may just be CSS’s next game-changer. They’ll allow us to segment our styles, work with the cascade rather than against it and avoid the specificity issues that have plagued our large-scale applications.

Check out Bramus’s presentation

Marcin Wichary: I pressed ⌘B. You wouldn’t believe what happened next

Marcin’s talk isn’t one we had marked as a potential show-stealer, but it turned out to be both very insightful and seriously entertaining. Like Jeremy Keith, Marcin took us on a history lesson, this time of keyboards and typography and the complexities at the intersection of both. Different fonts, weights, naming conventions, font origins, keyboards, commands, languages,  sup and sub font support… even with all this, as Marcin says, sometimes “stuff just breaks that we cannot foresee”.

Check out Marcin’s presentation

Summing up

We came away from CSS Day having had two great days and bursting with ideas. But aside from all the brilliant tips and tricks, this served as a more general reminder that the web – at its best – is inspiring, evolving, inclusive and in our hands.

Many thanks to FreeAgent for the opportunity to travel and attend. We really appreciated the change of scenery, the investment in our skills and – given the events of recent years – the chance to once again attend a first-class conference in person.

Laurence Hughes and Ryan Deegan,
Design System Team

PS: FreeAgent is hiring for a front-end developer on our Design System team. Come join us!

Engineering lessons from my SaaS startup

Posted by on August 9, 2022

I’m now almost two months into my software engineering summer internship at FreeAgent. When I was 18, I chose to delay the start of my university course for 18 months to have a go at building a ‘Software as a Service’ (or ‘SaaS’) startup with two others.

Our software, ‘trackio’, was a bring-your-own data platform that allowed sales teams to collect, connect and reconcile their disjointed data sources. It was essentially a business intelligence tool. Think of fancy dashboards and shiny reports you can use to make decisions.

Ultimately, the startup was closed for a multitude of reasons; the pandemic, for one, didn’t help. We did have some success, however. Getting trackio into a number of Vodafone franchise stores and becoming an AWS activate startup were among the highlights. 

Today I’m here to talk about some of the engineering lessons we learned at trackio as a result of our first real attempt at software engineering beyond WordPress and OpenCart.

Early version of our WebView mobile app

The feature factory

A few months into the project as the business grew and the codebase became more complex, we started to receive lots of feature requests. We would add whatever our customers requested without giving it much thought, simply to keep them happy. In doing so, we unwittingly became what is known as a ‘feature factory’. 

The product development expert, John Cutler, has written some excellent resources that can help you identify whether you’re working in a feature factory. He says that one of the biggest red flags is when you are solving real problems for real users but your product or service doesn’t translate well beyond that niche use case. This became apparent at trackio when we started pitching to venture capitalists who could see that what we’d built for Vodafone franchises worked well for their sales teams but wouldn’t provide much utility in a different industry.

The main driver of our feature factory was chasing upfront revenue. To close deals we incurred technical debt by trying to implement as many features as we could, often having to pay for it in the future. Another contributing factor was a lack of retrospectives to determine how well the shipped features were received and whether they made sense for the product roadmap and overall vision. 

During my internship here at FreeAgent, I’ve enjoyed our weekly retrospectives. As a team, we’re constantly discussing the work we’ve been doing and if the impact was what we expected. There’s no shame in admitting we anticipated something incorrectly.

“Does this feel right?”

I often ask myself “does this feel right?” when I’m working on something. When my trackio teammates and I first started out we were novices at best, and we only really had Google and books to guide us. It’s great being in an internship and having a team of more experienced people to bounce ideas off and learn from. 

For me, instinctively asking yourself “does this feel right?” is one of the best indicators that you are improving as a developer. It’s the seed that often grows new knowledge. It was that very question that made us start using git for version control instead of Dropbox at trackio.

Don’t make me think

Early on in our product development, we didn’t think enough about who our users were or how they were going to use our product. With the help of user stories, we were able to start diving deep into features from the perspective of the people who would be using them.

With this, we learned our database design wasn’t going to be able to handle users editing key performance indicators (KPIs) while the figures were being calculated. We needed a first-in-first-out queue to manage concurrency and consistency.

We also learned that most of our data accuracy problems came from notification fatigue because users didn’t have time to read information modals, so we needed a user experience that didn’t require explanation.

Users don’t want to think when they’re using your software. It’s your task to build them something where they can accomplish their intended tasks as easily and directly as possible. Shoutout the author of Don’t Make Me Think and UX specialist Steve Krug. 

Iterate quickly and validate often

When we first started exploring a mobile version of our platform, we were already spread thin. The idea of standing up two native iOS and Android mobile apps using completely different languages was a daunting and quite frankly impossible job for two people. Given our hands were full with a web platform, point of sale system, and everything else startup related.

Instead, we opted for WebView containers. WebView is a way to use web technologies in a native-like app experience. Therefore, we could use the already existing mobile version of our web app and deliver it as if it was a native mobile app. WebView has its share of drawbacks which makes it far from ideal in the long term. In particular, you’ll often find yourself re-inventing the wheel and working on things that come out of the box with native app development.

But what WebView is great for is rapid development and quick iteration. In less than two weeks we had both an iOS and Android app released that was being used and gathering feedback. Going mobile gave us engagement like never before and made us completely rethink our product direction.

When you’re building a startup the faster you get things into the hands of users the better. The product will go through many iterations and nothing is set in stone. By the time we started planning a migration to native, we now knew what our mobile apps needed to do and we were improving on something already validated by our users. Without WebView we either wouldn’t have released anything. Or, it would have taken months to release native apps that were inevitably going to change based off user feedback.

Every SaaS needs its support

Support is the backbone that pushes a SaaS business forward. Once you find product market fit and have paying customers, the last thing you want is to lose those hard-earned customers to churn. I learned that it’s vital to listen to what your customers are saying and to fix the problems they’re facing as quickly as possible. But when customers have feature requests, remember the perils of the feature factory and don’t feel the need to implement everything a customer wants just to please them. When you’re an engineer, you know your product better than anyone else, giving you foresight that others might lack.

Unfortunately, support was something of an afterthought for us at trackio. It took up a lot of our precious time and distracted us from getting back to building a great product, purely because we didn’t have the systems in place to receive and manage customer issues.

My internship at FreeAgent is with the support engineering (SE) team. At trackio, SE was something that we didn’t know existed, but we definitely needed it! With a customer support ticketing software like Zendesk you can handle the customer-facing side of support efficiently, while escalating engineering problems to a support engineer. It’s what keeps the development of the product moving, while improving the experience for current users.

There’s no glamour in burnout

Thinking back to why I started programming, I wanted to build cool things. Programming felt like the superpower that enabled creation. It’s fun working on personal projects and learning new technologies. When you’re in a startup, you inevitably have a lot of roles and responsibilities at the beginning, including some that you don’t necessarily enjoy. Until you can scale a team, it’s just you and (if you have any) your co-founders.

If you were to ask me what the biggest startup killer is, ‘founder burnout’ is what comes to mind. Nothing will destroy your dreams faster and if it goes unchecked it will likely be the thing that makes you finally throw in the towel.

Engineering is a mentally intensive role. Prioritising exercise and diet can go a long way to help ensure you’re mentally ready to face the challenges that lie ahead. The times we were crushing our development at trackio wasn’t during the 20-hour work days but when we spent the greatest amount of time planning our backlog, product roadmap and constantly evaluating our in-progress work. Startups are marathons and the longer you stay in the race the greater your odds of survival.

Final thoughts

Closing a company in the tech world is almost like a right of passage. Most don’t make it and of those that do, it’s usually not the founder’s first attempt. 

Once it’s over, you might try to blame external circumstances, timing, or bad luck. But it’s a bit like failing your driving test. At the time, you might think that the world is against you. However, once you finally get back onto the road you realise you needed that first failure under your belt to guide you to later success.

Engineering is the art of problem-solving, and how you get to your final destination is your choice. Enjoy the ride and all the twists and turns it provides.

How to boost your confidence as a beginner software engineer

Posted by on August 8, 2022

Every engineer has questioned at least once in their life whether they are skilled enough for their chosen field. I experienced these doubts when I started my computer science degree at university, and despite the fact that I got through the first three years successfully, I still wasn’t fully confident in my abilities when I started my internship at FreeAgent. 

However, this is entirely normal. Every engineer faces challenges on a daily basis, no matter how many years of experience they have. In this blog, I’ll share some methods that helped me overcome the fear of not being skilled enough.

Practice makes perfect

One of the main ways to gain more confidence is to practise. Yes, it’s that simple… or maybe it isn’t! 

At university, you should be able to learn the basics of a programming language and practise them while developing different projects. The problem is that the number of projects that students typically participate in is simply not enough to explore the different ways of implementation, and sometimes there might be a more efficient solution. 

Another thing to keep in mind is that people are different and some of them need more practice. So don’t compare yourself to others, because everyone learns at a different pace. 

It’s important to develop your thinking along with your programming skills, which you can do by completing coding challenges at free websites such as LeetCode and FreeCodeCamp. These challenges are a good way to prepare for job or internship interviews, as most companies use the same format to examine their candidates. They will definitely boost your confidence and help you to remain calmer in these kinds of situations.

There are also really useful YouTube tutorials for different programming languages where people show step-by-step how to implement certain functionalities or even whole projects. These can be a great way to maintain your skills and learn something new.

The more experience the better

As a software engineer, the best thing you can do is solve as many problems as you can. That way you’ll be more confident when dealing with a similar situation in the future because you will have already experienced it. 

Internships are a great opportunity to gain this extra experience and put into practice the knowledge you gained at university. If you’re a student, I’d encourage you to find one, as this is the perfect opportunity to learn something new and actually be paid for it. 

You’ll have the chance to work with people who are more experienced in the computer science field, each of whom has a different way of thinking. This will really benefit you, as they’ll suggest different unique solutions that will be applicable not only to the current project but also to future ones. 

People will be happy to guide you, so don’t be afraid to ask questions when there’s something you’re unsure about or you don’t understand. This doesn’t mean that you’re not smart enough – on the contrary, it shows that you’re being involved in your task, paying attention to the details and trying to learn as much as you can.

Give yourself time to do the necessary research

The first thing that came to my mind when I started my internship at FreeAgent was: “Am I going to be able to finish this project on time?” 

I spent the first few days copying and modifying some existing code, which was my first mistake. I didn’t give myself enough time to analyse the code and its structure and to understand the purpose of the existing classes. 

What’s more, this was a completely new programming language to me, which made the coding even harder. I barely made any progress because I didn’t understand it, which in turn affected my confidence. 

The first step to achieving a better understanding of a piece of code is to take the time to learn the basics of the programming language. Although programming languages tend to follow the same logic, they have some specificities in their syntax. 

The next step is to read about the architectural pattern being used, because this will give you a rough idea of the functionality that each structure has. And last but not least, if you’re in a company, ask a team member to explain the main parts of the code and how they work. I guarantee this way you’ll fill in most of the missing pieces in your head. 

Just keep in mind that there will always be something that doesn’t make sense when you start a new project, so be prepared for that and accept it. No one expects you to know everything from the beginning and you don’t have to figure out everything alone.

Take a break 

Some of us, including me, tend to overcomplicate a given problem and spend a lot of time on it. This happens sometimes even to the best software engineers.

Most of the time, the reason is that after hours spent in front of the computer, our eyes are getting tired and we can’t even see the obvious and simple solutions. This leads to frustration, which impacts our concentration and thinking. 

We start blaming ourselves and become less productive, but in reality, we just need to rest. In these situations, the best thing to do is to take a break from the monitor or move on to a different task. Once your brain is refreshed, you’ll find the solution more quickly.

Don’t copy solutions without understanding them

Speaking of fixing bugs, let’s not forget that Google is our best friend – or more specifically Stack Overflow, where we can find solutions to thousands of common problems. 

For a beginner engineer, sometimes it might be hard to understand the proposed solution or apply it to the code. In both cases, don’t hesitate to ask someone from your team to explain the code to you and help you modify it for your project. 

What’s more, the person might suggest even a better and more efficient solution. Try to avoid using a code that you don’t understand, because you won’t be able to work with it nor modify it if needed, and you probably won’t be able to solve the same problem in the future.

Play with the code

To me, programming is like maths. Even if you have a lot of ideas for a solution, you need to implement them in order to see which one will actually work. 

Don’t be afraid to play around with the code, because you won’t find the solution by staring at it – and sometimes what appears to be the most absurd solution can actually be the answer. Also, testing different implementations means making mistakes, which is not a bad thing. This is the best way to learn and get a better understanding of the flow of the code. 

Don’t ignore the ideas that don’t appear to be good enough, and don’t try to implement the perfect solution from the first try – you can always improve your code later. The important thing is to make a start and write a few base lines on which you will continue your work. And remember that you can always ask for help if you can’t spot an error or you don’t understand why a solution that seems right to you doesn’t actually work. 

Conclusion

Being a software engineer is not an easy task, considering that you need to be up to date with the latest technologies – which are evolving each day – and that you have to maintain and further develop your coding skills at the same time. 

So don’t lose confidence in yourself. It takes bravery to start this journey, which will be full of ups and downs. If you’re really involved in it and make the effort, you will definitely achieve results. Just give yourself time.

An introduction to object-oriented programming

Posted by on August 3, 2022

Object-oriented programming, or OOP, is a programming paradigm centred around the concept of objects. This article will aim to provide a basic understanding of objects, classes, OOP principles and various other concepts within the paradigm. Code examples will be mediated in the Ruby programming language, which is an example of an object-oriented language. In Ruby, any value is an object, even data types like String and Integer, which are often represented as primitive types in other object oriented languages.

Objects and classes

As mentioned previously, any value in Ruby is an object. Objects in Ruby contain two primary properties: states and methods. States are the data values which are stored inside an object. Methods are the functions which are associated with an object. Objects are built based on classes, which can be viewed as the ‘blueprint’ for an object. Classes contain the definition of what an object of that class will consist of and what procedures can be performed on the object.  

The following example shows the object ‘obj’ being initialised and then the class method being invoked on it. The class method can be used on any object in Ruby to show which class an object is made from. We can see that the object ‘obj’ is an instance of the class String.   

Class method invoked on an object of class String

Classes will often contain a constructor, which is a class method that will be invoked whenever an object of that class is created. Constructors are used to initialise variables whenever an object of the class is created. These variables define the state of the object. Constructors can be passed arguments, which will initialise the variables with relevant values. For example, an object could have a constructor that takes an argument name and assigns the value it is passed to the variable @name when an instance of that object is created. Variables defined in this way are known as instance variables.  

Methods define the behaviour of an object. They can be viewed as actions that an object is able to perform. Methods in Ruby are very similar to methods in other languages. Methods can have a level of access control which varies between public, private and protected. If a method is defined within a class, then it will be public by default, assuming it is not defined under a private or protected keyword. A public method can be called from anywhere else in the code via invoking that method on an object of the class. 

A private method is a method defined under the private keyword. These methods are for internal usage and cannot be called directly from outside the class, however they can be called from within the class’s public methods. An example of where this behaviour may be useful is in a large codebase where a programmer changes the name of a public method. That change will produce errors at every point in the program where that method is invoked on an object, which will lead to a tedious tidy up. Instead if this method was private and only called within the class’s public methods, the only changes required would be within its definition and wherever it is called in the class. 

A common example to illustrate some of these concepts is a Car class. 

Car class definition

The example above shows the definition of a Car class with a constructor which will set the variables to whatever values are passed in as parameters. The class also contains some public and private methods. An object of class Car, audi, is then created and the print_informations method invoked on it. When run, this returns the following: 

Output of running print_information on the audi object

As mentioned previously, private methods cannot be called from outside the class. When attempting to invoke the private information method on the audi object, the result is a NoMethodError.

NoMethodError from invoking a private method outside the class

Four pillars

Four of the most important OOP principles are known as the ‘four pillars’. These are abstraction, encapsulation, inheritance and polymorphism. In this section I will give a brief and basic introduction to each of these concepts. 

Abstraction

Abstraction can be viewed as a method of handling complexity within a program by hiding unnecessary details and showing only relevant data, which increases efficiency and reduces complexity. Objects are a perfect example of this. The methods that are invokable on the object and its relevant parameters are available, but the actual implementation is abstracted within the object’s definition. 

Encapsulation

Encapsulation is the process of bundling related data and methods together into one overall unit. This process, also known as information hiding, conceals segments of functionality from the rest of the code base, so that it cannot be altered unintentionally or viewed from the outside. Encapsulation can be seen as the boundaries within your code. An example of encapsulation in practice are classes.

Inheritance

Inheritance allows characteristics of one class to be inherited into another. In an inheritance scenario, there is a ‘subclass’ which will inherit characteristics from the ‘superclass’. This concept has many applications and is key to providing reusability within programs. For example, if a subclass of the Car class was created, it would then have access to characteristics such as the print_information method.   

Subclass of car

When run, this returns the following:

Output of running start_engine on object of the Toyota subclass

Polymorphism

Polymorphism is composed of two words, poly and morph, which mean many and forms respectively. In an OOP context, this can be viewed as the ability to execute the same method using different objects. In Ruby, one way of achieving this is via inheritance. If two subclasses of the previously discussed Car class were created with slightly differing definitions of the start_engine method, such as ‘Toyota engine has started’ and ‘Audi engine has started’, creating an object for each sub class and invoking the start_engine method would be an example of polymorphism, as the method would return different results. This type of polymorphism is known as subtyping, or subtype polymorphism. 

Example of subtype polymorphism

When run, this returns the following:

Output of invoking start_engine method on different objects

Why OOP

There are many advantages to the object-oriented approach to programming. In this section I will discuss only a few of the reasons why this paradigm is so effective for creating well designed and effective programs. 

In general, object-oriented programs have a high level of readability compared to other approaches. The use of objects makes reading the program’s source code easier at a glance, and it is easy to find relevant code for any object via its class file. This also makes modification and maintenance of the program easier.  

OOP allows for the easy reuse of existing code, which helps in eliminating a lot of redundant or repeated code. In the situation where a programmer is defining a new class and realises that there is an existing class with some of the functionality required, they can easily allow the new class to inherit that functionality from the existing class. This drastically increases reusability and avoids repetitive code. 

Encapsulation provides modularity. As objects are self contained the troubleshooting process is easier. Encapsulation also increases overall security, as it prevents data from being unintentionally altered or viewed from the outside. Polymorphism increases flexibility, allowing the reuse of methods which saves time.

Conclusion

Object-oriented programming is an excellent paradigm for producing readable and effective code that can be easily understood and maintained. The use of objects and classes allows for complex concepts to be represented as simple structures that maintain a high level of readability, as well as providing a multitude of other advantages such as reusability and easiness of debugging. These, among many other benefits, are provided via following the various OOP methodologies such as the four pillars.  

We're totally hiring!

See our current engineering vacancies