Grinding Gears

Introduction The heart of a reliable data platform are robust and automated data pipelines. As our team migrates our data pipelines to Dagster, re-architecting our automation logic is a crucial task. Dagster offers condition-based approaches to creating or updating a data asset (table or file), moving us toward a modern, asset-centric view of data. This post details the three primary automation strategies we considered and implemented (so far) in Dagster—Schedules,… Continue reading

Introduction Data Engineering at FreeAgent has a mission to ensure our colleagues and customers have reliable, accurate, and secure access to the data they need. Our migration to Dagster, DBT, and DLT is a key part of that. However, it has raised numerous questions, including how we test DBT Macros. This post dives into how we're tackling this by leveraging pytest and Jinja for more efficient unit testing of DBT… Continue reading

Introduction Data is exploding, and so are the tools to manage it. From generating and collecting, to cleaning and analyzing, these tools help create valuable products for customers and give stakeholders decisive insights. As Data Engineering at Freeagent continues to evolve, we're focusing on providing more reliability and quality in our data products. For data pipeline building, we've started to move from a no-code approach toward a software engineering focused… Continue reading

Over the last few years we’ve evolved the way our analytics team works to enable easy access to accurate and reliable data for faster, better decision-making. Recently we made one more change—our Business Intelligence Analysts are now Analytics Engineers! Continue reading