All posts written by Paloma Jol
Fine-tuning a DistilBERT classifier with numerical and text inputs
Text classification is often done through fine-tuning of a pretrained foundation model with domain-specific data. In FreeAgent we use transformer based models to automatically classify incoming bank transactions. Specifically we use a DistilBERT model that is fine-tuned on hundreds of millions of bank transactions with customer-labelled accounting categories.
The model inputs are currently text-based, built from a combination of bank transaction descriptions and amounts.
In this post we describe an approach to fine-tuning the DistilBERT model and training the classifier including the numerical amount feature as a single network. Continue reading