Analytics engineering can be weird sometimes.
Scenario: We receive advertising data in a daily email where we have a download link (so no attachment). Urgh…
Write a custom function (potentially with Airflow) that receives the email, parses the body, retrieves the link, load the csv data into memory, save it to Google Cloud storage, then use data transfer to BigQuery – Effort: 2-5d (depending if you have some parts already lying around) – Maintenance for the customer: too high
Using Integromat – create an scenario that receives the email, parses the body with regex, request the csv data and writes it to Google cloud storage: Effort: 3-5h (parsing & encoding part was a bit tricky) – Maintenance for the customer: doable