Corabastos price tracker. Wholesale prices, day by day.
Corabastos publishes its prices as a PDF every weekday. This pipeline turns each bulletin into rows in a time-series database and a trend you can read. Run one sample bulletin below.
Try it: one bulletin into a price trend
Pick a product, then run the daily update.
What the platform did
- cron7:00 Bogotá, a weekday: APScheduler starts the daily update; there are no bulletins on weekends.
- pdfDownloaded Boletin_diario_20260116.pdf from corabastos.com.co and checked that it really is a PDF.
- parsepdfplumber read the tables, skipped the header rows, turned PAPA PASTUSA at $1.760 into a number and normalised the change to Estable, Subio or Bajo.
- dbUpserted into precios, a TimescaleDB hypertable keyed by date: one price per product per day, rows only rewritten when a value changed.
- dbBulletin marked as processed with the count of inserted, updated, unchanged and skipped rows.
- appStreamlit drew Potato (pastusa) with Plotly: current, average, minimum and maximum for the period.
Tap a part of the pipeline
Highlighted wires show what it talks to.
Collect
Transform
Store and show
Runs the daily update at 7:00, Bogotá time, Monday to Friday.
Why: Corabastos publishes around 6 AM and there are no bulletins on weekends.
Builds the day’s bulletin URL or finds it on the prices page, and keeps a local copy.
Why: It checks the content type or the PDF header, so an error page never reaches the parser.
Reads every table, skips header rows, cleans prices and sorts each product into a category by keyword.
Why: Prices come as $16.000 or $16,000, so they are normalized before they become numbers.
PostgreSQL 15 with a precios hypertable on date, plus products and bulletins tables.
Why: A unique index on product and date keeps one price per product per day, and a row is rewritten only when a value changed.
Explore prices, trend analysis with Plotly, and an admin page for imports, backfills and coverage.
Why: Backfills run through the same pipeline as the daily job.
Stack
- Pipeline
- Python 3.11
- requests
- BeautifulSoup
- pdfplumber
- pandas
- Data
- PostgreSQL 15
- TimescaleDB
- SQLAlchemy
- App
- Streamlit
- Plotly
- APScheduler
- Ops
- Docker Compose
- Type
- Personal tool
- Source
- Corabastos daily PDF bulletins
- Schedule
- 7:00, Monday to Friday
- Storage
- TimescaleDB hypertable
- Runs on
- Docker Compose, self-hosted