All posts tagged with 'data platform'
Streamlining DBT Macro Testing: A Unit Test Approach with Pytest and Jinja
Introduction Data Engineering at FreeAgent has a mission to ensure our colleagues and customers have reliable, accurate, and secure access to the data they need. Our migration to Dagster, DBT, and DLT is a key part of that. However, it has raised numerous questions, including how we test DBT Macros. This post dives into how we're tackling this by leveraging pytest and Jinja for more efficient unit testing of DBT… Continue reading
Decoding Data Orchestration Tools: Comparing Prefect, Dagster, Airflow, and Mage
Introduction Data is exploding, and so are the tools to manage it. From generating and collecting, to cleaning and analyzing, these tools help create valuable products for customers and give stakeholders decisive insights. As Data Engineering at Freeagent continues to evolve, we're focusing on providing more reliability and quality in our data products. For data pipeline building, we've started to move from a no-code approach toward a software engineering focused… Continue reading
What a data science degree doesn’t teach you
When I enrolled on my data science master’s degree I had limited statistical and coding knowledge. This course was designed to teach these skills from the bottom up. Having now worked as a software engineering intern, I have come to realise a lot of things were missed. Moving beyond ‘if it works… it works!’ Learning to code can seem very daunting. There are so many resources and even languages. Where… Continue reading