Lots of our infrastructure is managed by Puppet, the open source configuration management tool that uses its own domain specific language (DSL) to describe how services, files and systems should look.
The DSL was inspired by Nagios configuration files, so you’ll find lots of nested blocks under curly braces, indentation, groups of parameters assigned with => separators, all of which can begin to look messy in a large manifest, as Puppet calls them. Keeping the syntax and style clean and consistent with the style guide helps team members read and understand these manifests quickly, so it’s vital to keep a large repository navigable.
The community has developed some great tools that help you keep your code correct and the style consistent and readable.
puppet-lint with its own DSL parser is maintained by Tim Sharpe and also has a community of plugins to customise it
puppetlabs_spec_helper integrates these tools into your project, making them easy to run and use
Keeping lint-free with no effort
Vox keeping a watchful eye out for syntax errors
Our repo receives a few contributions from outside the ops team at FreeAgent, such as our corporate IT team maintaining VPN access, and engineers in other teams adding and changing configuration files or entire services. So when it came to adding puppet-lint recently, I wanted to minimise the effect of failing lint tests on other contributors and keep it easy to accept changes while enforcing a consistent style.
While someeditorsmake it easy to run linters for you, not everybody inside and outside of our team has this set up — this meant having Jenkins, our test/continuous integration server, also help out by automatically fixing issues where it could.
The idea here is to have Jenkins run puppet-lint in fix mode to correct style issues in manifests, then push any fixes it can to the user’s branch — which is possible because our branches are in a shared Git repository. This causes a little feedback loop that results in the branch passing all of the lint tests, or for any unfixable issues to be left to the user. They can then check the output of Jenkins to see the remaining errors, run git pull, fix the issues and push the last changes.
Feedback loop where Jenkins checks for lint issues, fixes what it can, and triggers another build if any issues were fixed
De-linting pipeline with Jenkins
This new step is added into the Jenkins build pipeline, configured in the Jenkinsfile for the project, and looks like this:
/* Utility to run a command and return stdout */
def capture(String script) {
return sh(returnStdout: true, script: script).trim()
}
stage('Lint check') {
try {
sh('chruby system && bundle exec rake lint')
} catch(err) {
if (env.BRANCH_NAME == 'production') {
echo 'Unable to help on production branch, please open a PR!'
throw err
}
/*
* Attempt to autofix any lint issues and push another commit.
* If it works, Jenkins will re-test the new commit and then pass.
*/
echo 'Attempting to run auto-fixer..'
fixedOutput = capture('chruby system && find . ! -path ./.bundle/\ -name \.pp -exec bundle exec puppet-lint --fix {} + || true')
/* If issues were fixed, push the changes */
if (capture('git diff-index HEAD --') != '') {
echo 'Found some automatic fixes, preparing to commit them'
commitMsg = fixedOutput.split("n").findAll {
it.contains('FIXED:')
}.inject("Fix lint issuesnn") {
result, line -> result + " " + line + "n"
}
sh('git config user.email jenkins@example.com')
sh('git commit -a -m "' + commitMsg + '"')
sh('git show -s | cat')
sh('git push origin HEAD:' + env.BRANCH_NAME)
}
throw err
}
}
Now, each time a branch is tested by Jenkins, it can run through these steps:
Check if the lint checks are passing
If not, try to run puppet-lint with the fix mode enabled
If it was able to fix issues, construct a new git commit message with the output of the command, commits and pushes
It raises an error as the branch (as it was) failed the lint test
The new commit on the branch triggers the test to run again with the fixes, and it hopefully passes
And the end result looks like this, with our butler doing the dull work!
Two commits in GitHub, one from a developer with failing tests, then a second commit from Jenkins listing the lint issues it fixed with tests passing
This is just one of the ways we aim to make the development process easier and smoother for engineers in FreeAgent. If this sort of thing interests you, look out for jobs in our Development Platform team who come up with tools to power our growing engineering department.
As of Monday, 24th of September 2018 FreeAgent is now running on the latest version of Rails. We were inspired by Eileen’s blog post about how GitHub upgraded from Rails 3.2 to 5.2 and we wanted to share with you the challenges we faced and how we managed to overcome them. We faced similar challenges to GitHub and we believe it is worth reiterating them to highlight their significance.
Motivation
Considering that each upgrade requires considerable efforts that span across multiple teams, some people might ask “is it even worth upgrading?”. That is an excellent question and we would like to answer with a list of key points that drove our decision:
Security Patches — Security is taken very seriously at FreeAgent. We aim to apply security fixes as soon as they become available and have been tested thoroughly. Running an older version of Rails can mean that it may not be supported to receive these fixes.
Performance Improvements — The community constantly pushes the limits to improve the performance of libraries and frameworks. Rails is a great example where a new version can provide a significant performance boost to your application.
New Features — Each major and minor version of Rails usually comes with a host of new features. Just recently Rails added the ability to manage file attachments via ActiveStorage which is an excellent feature that allows us to migrate away from the deprecated solution which is paperclip.
Ruby Language Improvements — Major version changes tend to support newer versions of the Ruby language. This allows us to benefit from performance improvements as well as new language features.
Ecosystem (gems) — Ruby’s ecosystem is simply incredible, providing thousands of individual libraries that can aid you in your day to day job. Whilst library maintainers do an amazing job at supporting multiple versions of Rails, there is a great chance that not all features and fixes will be available to you if you run an outdated version of Rails.
Upgrade Process
Preparation
Due to the structure of our development teams, changes that impact the entire codebase usually fall within the remit of our Core Services team. We think of the Core Services team as a team that provides the glue between the other units within our application.
We usually start by getting a good understanding of the changes that were introduced by a new version. A good place to start are the release notes for the individual versions. It turned out to be beneficial to also check the changelogs for the individual gems. Changelogs provide a more detailed list of changes. Keep in mind that incremental version changes are essential as skipping one version can mean that you miss valuable deprecation warnings that just result in errors.
Upgrading
At this point we are trying to address as many deprecation warnings as possible. A benefit is that the commit introducing the upgrade only includes changes that are actually related to the upgrade. Furthermore, addressing deprecation warnings can be distributed to the entire development team. We added a set of tests that would prevent developers from reintroducing deprecation warnings into the system. This is important as a large codebase can have several of these deprecation warnings that need addressing. This also results in smaller changes to the application which can be tested in isolation.
Any changes that can be addressed prior to the upgrade will be addressed at this stage as well. This could include new framework defaults that don’t manifest as deprecation warnings but could be implemented prior to the version change. The aim should be to be as compatible as possible with the version you are targeting.
Once we are confident that we addressed all things that can be done prior to the actual upgrade we would continue with the upgrade process by creating a new branch that contains changes that are exclusively related to the upgrade and can’t be addressed upfront. In an ideal world that would just be a change to your dependencies. An initial run against your CI tool will normally reveal the areas that you need to focus on. Over the next weeks we would fix any issues that were flagged by our CI tool and periodically integrate changes that went into our master branch.
Finally, we perform a number of manual tests to ensure that the application is still operating as expected. This can take several days to complete. Any issues that are found during these tests need to be addressed and some subsequent tests need to carried out to validate the fixes.
Rolling out the changes
Once we are happy with the changes, we start to communicate our intentions to roll out the upgrade with the other teams. This includes our Support team to make them aware of potential requests from our customers that need to be fed back to the Engineering team. We usually make several announcements via Discuss (our internal Discourse-powered discussion forum), Slack and Email.
On the day of the upgrade we implement a merge freeze that prevents other changes from interfering with the upgrade. The merge freeze is expected to last an entire day under normal circumstances. We keep our engineering teams up to date and lift the merge freeze earlier if everything runs smoothly.
A number of tools support us to validate the health of our applications. We use NewRelic, HoneyBadger and some custom Grafana dashboards to see the impact of any deployments.
At this stage it is important to monitor the application and to decide how to react in the event of an issue. Based on the severity of the issue we consider to roll back to a previous version. However, due to the increased awareness of the change, we have engineers, support teams and managers ready to discuss and address any issues. We aim to move forward and unless an issue is very severe we will not rollback the version upgrade.
Challenges
Large codebase — FreeAgent’s codebase is relatively large in size. There are over 300 controllers and nearly 1000 models to look after. Upgrades to the underlying web framework can therefore affect many areas of the application.
Old codebase — This can be considered relative but the origins of our codebase are over 12 years old, and the app has been upgraded many times starting from Rails 1.1.2 all the way up to Rails 5.2 in incremental steps.
Many contributors — We have many engineers working on our (majestic) monolithic Rails application. Having long-running branches to upgrade Rails and merge freezes are not ideal as we would like to continue to add more value for our customers and improve our application. It is challenging to keep branches up to date and to continuously integrate changes (and fixing conflicts).
Code Coverage — Adequate code coverage will help you identify any problems that could come from an upgrade early on in the process. On the contrary the lack of test coverage can have a detrimental impact when rolling out the upgrade.
Accumulated some technical debt — There is no codebase out there that is clear of technical debt but it is important to understand that the higher your technical debt is at this stage the more difficult it is to perform large scale upgrades like these.
Use of private APIs — Sometimes it is necessary to use a private API to solve a particular problem. It’s important to keep in mind that these APIs are private for a reason and therefore subject to change. We identified a number of these places where we were using private APIs and it caused us problems.
Lessons Learned
Rails upgrades are a difficult undertaking and our process is by no means perfect. We strive to continuously review and improve this process. Below is a list of lessons that we learned during the updates we have performed since Rails 1.1.2:
The use of private APIs is harmful — The use of private APIs causes more problems than it solves. What seems like a great idea at the time will come back and cause you problems further down the line. Review what options you have and see if you can provide a solution that doesn’t involve the use of private APIs.
Incremental updates are essential — We considered going straight from Rails 5.0 to 5.2 but quickly changed our minds on this. Even though only two minor versions apart, the changes were significant enough to cause a lot of work. Skipping a minor version not only removes valuable deprecation warnings it also almost always will result in a longer upgrade process.
Long-running branches are not ideal — Long running branches are not ideal and this is an area where we would like to improve. We already do a good job at integrating as many upfront changes as possible but limited testing resources can still sometimes mean that a branch runs longer than expected. For larger upgrades we have some strategies to avoid long running branches, which we plan to discuss in another post. GitHub have shown an alternative approach to a long running branch that we will explore.
Code coverage is crucial — Code coverage is your best friend to understand what is working and what is not when making large scale application changes like upgrading Rails. We identified some areas where we lacked code coverage and have since mitigated some risks by adding some additional tests.
Conclusion
Rails upgrades are challenging and take significant development efforts to perform but we believe it is worth it. We hope you found our findings useful and we could help you in making your next upgrade easier.
It just seems like yesterday when I was settling in at FreeAgent and writing a blog about my first week as a FreeAgent data science intern. Yet here I am, having finished my internship and remembering all of the good times I had. Outside of my internship at FreeAgent, I am a PhD student specialising in veterinary biology, so the last 13 weeks have been a steep learning curve into the domain of an accountancy software company. This blog reflects upon this experience and all of the things that I have learnt whilst being fully embraced as a member of the FreeAgent team.
Finding my feet at FreeAgent
After being generally impressed with the company ethos in my first week, I soon realised that this wasn’t the type of internship where I would be running errands and getting lumbered with time-consuming yet low-skilled tasks. On the contrary, I was treated like every other member of staff and was given clear and achievable goals for my project, personal responsibilities and the tools I needed to succeed.
By the end of my second week, I had already given my first ‘Town Hall’ talk: a company-wide meeting that all departments attend. In fact, despite my nerves, during my internship I got the opportunity to speak in front of the entire company 5 times and many more times within specific departments. Coming from an academic background, I wasn’t used to preparing talks at short notice that were suitable for such a diverse audience but my talks were always met with enthusiasm and questions, which has hugely boosted my confidence at public speaking. As an added bonus, my penchant for gifs and bad puns went down a treat: it is always so much easier to talk to an audience with a sense of humor!
I was impressed with the ease at which meetings with people at different positions within the company were set up: complete strangers were willing to listen and give feedback on what I had to say. Remote workers were never excluded from meetings and felt as easy to contact as any other person physically in the office, which gave me a new perspective about what it means to work from home. Everyone was extremely open and easy to converse with on a professional level, which was complemented by the company’s adoption of cloud-based software, making it easy to share work and notes with colleagues. I also had the opportunity to ‘shadow’ other departments, such as the sales and support teams, which not only allowed me to observe job roles that I had no experience in but also gave me a broader view of the company as a whole and helped me to establish my place in the team.
Aside from feeling like I was accepted in the whole FreeAgent team, I was also made to feel fully supported within my specific team: data science. Every morning, we had ‘stand up’ meetings (a concept of the Agile method for software development) where everyone explained their goals for the current day. Despite the fact I generally prefer ‘lying down’ to ‘standing up’ first thing in the morning, I found these meetings a great way to focus and ask people for help in advance. In addition to stand up, we also had regular planning meetings, held ‘Sailboat retrospectives’ (another Agile concept) where we defined where we wanted to go as a team and documented our longer-term goals on project planning software.
Settling into the social scene
There is always something to do with your fellow employees at FreeAgent, whether it is sporty like Wakeboarding or gym classes or heading to the pub for a few drinks after work. I got to sample a plethora of the local lunchtime food spots when we went for team lunches and one lunchtime I even went skateboarding with a few other employees!
My favourite kind of ‘board’ meeting…
When I wasn’t galavanting around the local eateries, I often went for walks along the scenic canal and around the many local parks. Although the view from the office was great, it was nice to get out into the fresh air (and sunshine!).
Such a ‘boatiful’ view…I ‘viewed’ it as a privilege to see the countryside from the office…
We also had a beer and wine fridge in the FreeAgent kitchen, with quite a selection! We often had a few drinks after 5pm when we had finished for the day and played pool or table tennis in the office.
We had a ‘ball’ playing pool in the office…
The FreeAgent barbecue was a highlight of my summer; a festival of food, drinks, live music and games held at a large conference centre. I helped to set up the event in the morning and it was thoroughly enjoyable. One of my favourite events that I took part in whilst at FreeAgent was the fundraising scavenger hunt for Shelter. Some of the tasks that my team (fabulously named ‘Simply the Quest’) completed included ‘getting a picture with the largest animal you could find’, ‘acting out one of the lines from Alanis Morissette’s ‘ironic’’ and ‘finding evidence of the presence of an invisible person, ghost or alien’.
We were relieved when we en’cow’ntered this beast!Isn’t it ironic: ‘A traffic jam when you’re already late’There were always some good ‘spirits’ about to help us…
Taking advantage of technical training
I developed my technical skills a great deal more than I expected over the short period I worked in the data science team. Different ways of doing things that I hadn’t considered before suddenly became obvious. When I was struggling to learn Ruby (a programming language that I had never encountered before), I had a team of experts around me that were ready and willing to help and give feedback about my code. I also became much more confident using the command line and Github — tools that I had used before but learnt to use much more efficiently! Of course, the great thing about being in a team of experts is that you can usurp some of their expertise without them even really knowing! Technical words that baffled me were brought up in conversation which I could then ask about or Google when I got home, depending on the situation. I also got feedback about my writing and presenting and the way I work professionally. The whole team participated in ‘Strengthsfinder’, an online skills assessment tool, which was an interesting insight into the way businesses try to understand their employees better.
My manager also spent time helping me plan my future professional development and career goals, which was hugely helpful to me. Part of this development included attending local relevant events with the data science team, which gave me the chance to network with people in similar industries and learn about different techniques.
However, without doubt, my favourite couple of days at FreeAgent was ‘hack days’: a company-wide event that allows people in different departments to collaborate on their own ideas to improve any aspect of the company. Ideas varied from fixing the company arcade game (yes, of course FreeAgent have an arcade game), to planning ways to make the office more environmentally friendly, to designing applications to make life easier for employees/customers. At the end of the second day, everyone presented the projects they had worked on and I was blown away by the amount of talent within the team. I personally was involved in a couple of projects; I shadowed the development of an app (I was even able to contribute!) and designed an experimental survey about a wine and cheese tasting to investigate bias (I admit I was slightly biased about the choice of food).
Inciting impact on important issues
The key project that I was involved in during my internship was designing a sales lead-scoring tool that could be used to predict the success of a lead signing a contract with FreeAgent. This involved several steps:
Analysing and cleaning historic survey data
Designing a predictive mathematical model for success
Designing the tool itself
Collecting further sales lead data and storing this
Implementing the tool in the current sales process
I found the whole process very satisfying and I was really happy to have made a positive impact on the company during the time that I was there. I also worked on a couple of side projects: researching external data sources of commercial interest and using some of these data sources to reveal company characteristics and using Google Trends to provide an insight into company-related internet searches and advising the communications team about more effective ways to report these results. During these projects I always felt fully supported by the data science team, who were always on hand when I needed to ask for help.
Goodbye, but not gone for good
I think the thing that has really hit me the most about my entire time at FreeAgent and the reason why I am most sad to be leaving, is the fact that I met so many genuine people and made some really good friends. Yet even though I have only been sat back at my PhD desk for two days, I have already been invited along to various data science-related events with the team and it is nice to know that even though I am gone from the FreeAgent workplace, I am not ‘gone for good’.
If you want to know even more about the work I did during my internship, I wrote a blog series about my general internship experience, data cleaning and survey design which can be found on FreeAgent’s engineering blog.
