dillenberg/ECC
1
0
Fork 0
mirror of https://github.com/Jeuners/ECC.git synced 2026-09-09 15:02:30 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix(agents): read-only reviewer contract + model re-tiering (#2442) + soften data-scraper prose (#2428) (#2480)

Browse source 
* fix(agents): retier reviewer frontmatter (#2442)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs(skills): soften data scraper prose (#2428)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Affaan Mustafa 2026-07-09 03:02:40 -04:00 • committed by GitHub
parent 38a7ebbe32
commit f6e18e0835
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 20 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

18
skills/data-scraper-agent/SKILL.md
View file

@ -1,6 +1,6 @@
---
name: data-scraper-agent
description: Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Scrapes on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
description: Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Runs on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100% free on GitHub Actions. Use when the user wants to monitor, collect, or track any public data automatically.
metadata:
origin: community
---
@ -14,7 +14,7 @@ Runs on a schedule, enriches results with a free LLM, stores to a database, and
## When to Activate
- User wants to scrape or monitor any public website or API
- User wants to gather or monitor any public website or API
- User says "build a bot that checks...", "monitor X for me", "collect data from..."
- User wants to track jobs, prices, news, repos, sports scores, events, listings
- User asks how to automate data collection without paying for hosting
@ -24,7 +24,7 @@ Runs on a schedule, enriches results with a free LLM, stores to a database, and
### The Three Layers
Every data scraper agent has three layers:
Every data collection agent has three layers:
```
COLLECT → ENRICH → STORE
@ -40,7 +40,7 @@ schedule summarises Sheets /
| Layer | Tool | Why |
|---|---|---|
| **Scraping** | `requests` + `BeautifulSoup` | No cost, covers 80% of public sites |
| **JS-rendered sites** | `playwright` (free) | When HTML scraping fails |
| **JS-rendered sites** | `playwright` (free) | When HTML fetching fails |
| **AI enrichment** | Gemini Flash via REST API | 500 req/day, 1M tokens/day — free |
| **Storage** | Notion API | Free tier, great UI for review |
| **Schedule** | GitHub Actions cron | Free for public repos |
@ -95,7 +95,7 @@ Common examples to prompt:
---
### Step 2: Design the Agent Architecture
### Step 2: Design the Collection Architecture
Generate this directory structure for the user:
@ -133,14 +133,14 @@ my-agent/
---
### Step 3: Build the Scraper Source
### Step 3: Build the Source Connector
Template for any data source:
```python
# scraper/sources/my_source.py
"""
[Source Name] — scrapes [what] from [where].
[Source Name] — gathers [what] from [where].
Method: [REST API / HTML scraping / RSS feed]
"""
import requests
@ -182,7 +182,7 @@ def _normalise(raw: dict) -> dict:
}
```
**HTML scraping pattern:**
**HTML fetch pattern:**
```python
soup = BeautifulSoup(resp.text, "lxml")
for card in soup.select("[class*='listing']"):
@ -760,6 +760,6 @@ Before marking the agent complete:
## Reference Implementation
A complete working agent built with this exact architecture would scrape 4+ sources,
A complete working agent built with this exact architecture would collect from 4+ sources,
batch Gemini calls, learn from Applied/Rejected decisions stored in Notion, and run
100% free on GitHub Actions. Follow Steps 1–9 above to build your own.