Product Research
Automated product discovery for ecommerce. Scrapes Google by category on a schedule, extracts product data, and saves to D1 for analysis.
Tech Stack
- Framework: Next.js + Cloudflare Workers (OpenNext)
- Database: D1 (product-research-db)
- Scraping: CamoFox (anti-detection browser)
- Extraction: Schematron (structured HTML-to-JSON)
- Hosting: Cloudflare Workers
URLs & Repos
- Production: research.coy.gg
- Repo: coydevs/product-research (GitHub, private)
How It Works
- Cron trigger - research-cron worker fires every 3 hours, staggered by category
- Categories: skincare, supplements, nutrition, weightloss
- CamoFox scrapes Google - Searches for trending/emerging products in each category
- Extract products - Schematron parses HTML into structured product data
- Save to D1 - Products stored in product-research-db with metadata
API Endpoints
| Endpoint | Description |
|---|---|
| /api/products | List/search discovered products |
| /api/products/stats | Aggregate stats by category, trend data |
| /api/research | Trigger manual research run |
Cron Worker
- Worker: research-cron (separate Cloudflare Worker)
- Schedule: Every 3 hours, staggered by category
- Flow: Trigger, CamoFox scrape, Schematron extract, D1 insert
Deployment
Cloudflare Workers via OpenNext. Deploy with Wrangler.