Back to Projects
pipeline
production

Public Procurement Intelligence Pipeline

End-to-end data pipeline extracting and analyzing government procurement data from a national SPA portal. Combines Selenium session auth, internal REST API calls, parallel extraction with 25 workers, exponential backoff, and produces a master dataset for market intelligence analysis.

PythonSeleniumREST APIpandasBeautifulSoupconcurrent.futuresData Analysis

Architecture

ERP + Portal + Registry
→
Normalize Entities
→
Fuzzy Matching
→
Conformed DWH Dim

Code Snippet

-- Full query available on request.