ParaCrawl
Date: —
Long-term EU(CEF)-funded project to collect parallel corpora from large-scale web crawls. I participated during my time as a data engineer at one of the project partners — TAUS.
Worked mainly with partners at the University of Edinburgh to optimise, maintain, and run a highly scalable processing pipeline to extract, translate, align, and clean parallel corpora obtained through web crawling.
Released corpora and further information on project website. Released software on GitHub.