Software architecture

VIDEO

DATA QUALITY PIPELINEyoutube.com/watch?v=evfdjv_YAGw

IMG040I VIDEO READY. PRESS ENTER OR CLICK TO PLAY.

DATA QUALITY PIPELINE · THE SOFTWARE IN THE VIDEO

It collects the EV charging tariffs published on a European eRoaming platform, reads them from PDFs too, merges them into a single template and checks data quality and compliance with the EU AFIR regulation.

PIPELINE.FEATURES

7 FEATURES
  • Automatic download from 6 sources for each offer, with automatic retries on failed files
  • Reads PDF tariff sheets in 14 languages, including Italian, English, German, French and Spanish
  • Every extracted value carries a declared reliability level: certain, medium, needs manual review
  • A single template that merges APIs, charging points, price lists and PDF rules, with every change tracked
  • 10 quality checks on every row and a report that assigns each error to an owner, with its severity
  • AFIR compliance check (EU Regulation 2023/1804), with a traffic light and the article breached
  • 3270-style terminal interface and an AI assistant to read the results

PIPELINE.STEPS

STEPS 00-09
  • 00DOWNLOADAutomated browser login and download from 6 sources per offer (API, PDF, Excel, CSV), with a preliminary check of every address and a second round of retries on failures.
  • 01PDF ANALYSISRecognizes columns, headings, tables and notes, and flags PDFs with no extractable text.
  • 02RULES FROM PDFSFive stages: price detection, classification in 14 languages, normalization, duplicate removal, tariff rule generation. Every rule is marked certain, medium or needs review.
  • 03INTERMEDIATE TEMPLATEMerges APIs, charging points and three price-list formats, including variable layouts, and consolidates time slots. Every discarded record gets its reason code.
  • 04FINAL TEMPLATEOnly certain PDF rules are applied, and only to authorized fields. Every overwrite is shown in red, with the previous value in a note. Duplicates are removed.
  • 05QA VALIDATION10 checks on every row: tariffs without a time slot, parking without a period, text in numeric fields, ISO 4217 currency, time-slot format, PDF text in structured fields.
  • 06DELIVERYA dated folder for every run, with all the output files.
  • 07ERROR REPORTEvery error has an owner (platform, operator or pipeline) and a severity (blocking, high, medium), with views by error and by operator.
  • 08AFIR COMPLIANCE10 checks against EU Regulation 2023/1804 with the article referenced, prices converted to euro at ECB rates, PASS · WARNING · FAIL per operator.
  • 09FINAL CHECKChecks coverage of new and bilateral offers, flags operators without an identifier and closes with the statistics.
PIPELINE.TECHNOLOGIESPYTHON · PANDAS · OPENPYXL · PYMUPDF · REQUESTS · RICH · PLAYWRIGHT · HTML/JS · AI
>> F4 = START YOUR PROJECT
4BAIMAGINA01 Imagina · VAT no. IT04146980125 · Privacy 001/015