Public Procurement Intelligence Pipeline

End-to-end data pipeline extracting and analyzing government procurement data from a national SPA portal. Combines Selenium session auth, internal REST API calls, parallel extraction with 25 workers, exponential backoff, and produces a master dataset for market intelligence analysis.

PythonSeleniumREST APIpandasBeautifulSoupconcurrent.futuresData Analysis

Architecture

ERP + Portal + Registry

→

Normalize Entities

→

Fuzzy Matching

→

Conformed DWH Dim

Code Snippet

-- Full query available on request.

Detailed write-up, screenshots, and metrics coming in Phase 4.