S/12 · Services
Scrapers & bots
The web’s data, collected — and the busywork, done for you.
Crawlers and scrapers that gather prices, listings and records from sites with no API, and bots that work on your behalf: watching for changes, filling in forms, answering in Telegram and WhatsApp. All of it scheduled, monitored and kept working when the sites change.
- Collects from
- Sites, portals, APIs and PDFs
- Runs
- On a schedule, at scale, monitored
- Delivers to
- Your database, API or spreadsheet
- 01
Someone copies data from websites into spreadsheets by hand.
- 02
You track competitors’ prices or listings and always find out late.
- 03
A portal you depend on has no API and no export.
- 04
Your team repeats the same clicks every day.
- D/01
Crawlers & scrapers
We build collectors for the sources you name, from simple HTML pages to sites that only show their content after JavaScript runs. Each one follows the site’s rules, keeps to a polite pace, and logs every run, so you know what was collected and what was missed.
Included
- A collector for each source, with the fields you need
- JavaScript rendering where a site requires it
- Rate limits and robots.txt respected on every run
- A log of each run: pages, records, errors and retries
crawler/runs/catalog.example.com.log $ crawler run --source catalog.example.com
robots.txt ok · 1 req/s · ua: example-bot/1.2
48 listing pages queued · js render: on
! 14 products without a price → flagged
× 2 pages returned 503 → retry in 10m
✓ 1,912 products · 26m 40s · 0 blocked
- D/02
Clean, structured data
Raw pages are turned into records with consistent fields: prices as numbers, dates in one format, the same product recognised across sources. Records that fail a check are held back instead of polluting your data, and the rest arrive where your team already works.
Included
- A clear schema for every record, documented field by field
- Validation, normalisation and deduplication across sources
- Delivery to PostgreSQL, an API, a CSV or a Google Sheet
- History kept, so you can see how a value changed over time
exports/products-2026-11-02.csv Product Price Stock Source Steel kettle 1.7 L $34.90 in stock shop-a.example Steel kettle 1.7 L $32.50 in stock shop-b.example Toaster, 2 slots $41.00 3 left shop-a.example Hand blender 600 W $27.80 sold out shop-c.example Coffee grinder $58.00 in stock shop-b.example - D/03
Monitoring & alerts
We watch the pages and values that matter to you and tell you when they change: a competitor’s price, a product back in stock, a new listing that matches your filters. You set the thresholds, so alerts are worth reading and don’t become noise.
Included
- A watch list of pages, products or searches
- Thresholds you control: percentage, price or any change
- Alerts in Telegram, email or Slack, with a link to the source
- A daily summary for changes that don’t need an alert
monitor/prices · last 24 h Items watched
1,240
across 6 sources
Price changes
37
12 above your threshold
Alerts sent
12
Telegram · #prices
Time to alert
4 min
median, after the change
- D/04
Chat bots
A bot your customers or staff talk to in the messenger they already use. It answers common questions, takes orders and bookings, sends updates, and reads and writes your real data through your systems, with a hand-over to a person when it should.
Included
- A bot on Telegram, WhatsApp or Bale, or all three
- Menus and flows for the jobs people actually ask for
- Connection to your orders, stock or bookings
- A hand-over to your team, with the conversation attached
telegram · @example_bakery_bot Is the sourdough loaf available tomorrow morning?
Yes, 14 left for tomorrow. Pickup from 8:00 at the Main Street branch. How many would you like?
Two, please.
Order #3318: 2 × sourdough, $9.00, pickup tomorrow from 8:00. Pay now or at the counter?
- D/05
Browser automation
When a portal you rely on has no API, a bot can use it the way a person does: sign in, go to the right page, fill in the form, download the report. It runs on a schedule with its own account, keeps the credentials in a vault, and takes a screenshot when something goes wrong.
Included
- A scripted flow for each task, tested on the real portal
- Credentials stored in a vault, never in the code
- Downloads filed and passed on to your systems
- Screenshots and an alert when a step fails
bots/portal-monthly-report · playwright $ bot run portal-monthly-report
→ open portal.example.com/login
✓ signed in (vault: portal/reports)
→ reports › sales › October 2026 › export CSV
✓ sales-2026-10.csv · 4,218 rows → reports/
✓ done in 1m 52s · next run 1 Dec 06:00
- D/06
Kept working
Sites change without warning, so every run is checked: did it finish, did it collect as much as usual, do the fields still make sense. When a source breaks we hear about it first, and repairing it is part of the maintenance you already have.
Included
- A schedule for every source and bot
- Health checks on record counts, empty fields and errors
- An alert to us and to you when a source drifts
- Repairs for layout changes, included in maintenance
crawler/health · today - catalog.example.com · 1,912 records, as usual06:00
- shop-b.example · 1,184 records06:20
- shop-c.example · price field empty on 30% of pagesfix in progress
- listings.example.org · layout changed, 0 recordsalerted 06:41
- portal-monthly-report · finished in 1m 52s1 Nov
- 01
Define the data
Which sources, which fields, how fresh, and what the data is for.
- 02
Check the ground rules
Terms of use, rate limits and the law reviewed for every source before anything runs.
- 03
Build & schedule
Collectors and bots built, tested on real pages and scheduled at a polite pace.
- 04
Run & maintain
Monitored in production and repaired when sources change.
A typical project takes three to five weeks, depending on the number of sources and how hard they are to read. The first source delivers real data in the first two weeks, so you can check the output before we build the rest.
- Week 1
We agree on the sources, fields and freshness, and review each source’s terms and limits.
- Week 2
The first collector or bot running, with sample data for you to check.
- Week 3
The remaining sources, validation and delivery to your database, API or sheet.
- Week 4
Scheduling, monitoring and alerts set up, and a trial period on live runs.
- Week 5
Handover: documentation, access to the dashboards, and maintenance under way.
The code
Every collector and bot in your repository, with tests and setup instructions.
The data
All collected records, with their history, in your own database or storage.
The schema
A description of every record and field, so anyone can use the data.
Schedules and dashboards
The schedule, run history and source health, visible to your team.
A runbook
How to run, pause and fix each source, and who to call when something breaks.
Maintenance, if you want it
Monitoring and repairs when sources change, on a monthly plan.
- Go
- Python
- Playwright
- Headless Chrome
- Proxies
- Queues & schedulers
- PostgreSQL
- Telegram & WhatsApp APIs
Is scraping legal?
It depends on the source, the data and how it is used. We review the terms and the law for each source first, collect only what you are entitled to, and turn down what you are not.
What happens when a site changes?
Every run is checked. When a source changes its layout we are alerted, and fixing it is part of maintenance.
Won’t it get blocked?
We crawl politely, within rate limits and identified where possible, which keeps blocks rare and access lasting.
What do you need from us?
The list of sources and the fields you want, how often you need them, and where the data should go. For portals, an account the bot is allowed to use.
Can we run it ourselves after handover?
Yes. The code, schedules and runbook are yours, and your team can run it without us. Most clients still keep maintenance, because sources change.
Tell us what you are building.
Every project starts with a consultation: you bring the idea and everything you know, we turn it into a design you can build on.
Book a consultation