[WIP] Scraper

Intelligent Scraper to scrape data from the web

Web Scraping Methods

Scrape

Extract data from a single webpage using the /scrape tool. This tool allows you to scrape specific content from a target URL.

Input Parameters

Parameter	Type	Required	Description
url	string	Yes	The webpage URL to scrape data from
selectors	array	No	CSS selectors to target specific elements
wait_for	string	No	Element to wait for before scraping

Output Data

The tool returns scraped content including:

• Raw HTML content
• Text content
• Selected elements based on CSS selectors
• Metadata (title, description, etc.)

Example Prompt

Example Prompt

scrape data from @url using selectors @selectors to #scraped_data using the /scrape tool

Crawl

Navigate and extract data from multiple linked pages using the /crawl tool. This tool follows links and scrapes content across multiple pages.

Input Parameters

Parameter	Type	Required	Description
start_url	string	Yes	Starting URL for the crawl
max_depth	number	No	Maximum depth of pages to crawl
max_pages	number	No	Maximum number of pages to crawl
patterns	array	No	URL patterns to follow

Output Data

The crawler returns:

• Crawled URLs
• Page content for each URL
• Link relationships
• Crawl statistics

Example Prompt

Example Prompt

crawl from @start_url with depth @max_depth to #crawled_data using the /crawl tool

Map

Create structured relationships between scraped data using the /map tool. This tool helps organize and connect scraped content.

Input Parameters

Parameter	Type	Required	Description
data	object	Yes	Scraped data to map
schema	object	Yes	Schema defining relationships
rules	array	No	Mapping rules and transformations

Output Data

The mapper returns:

• Structured data following schema
• Relationship graphs
• Validation results
• Mapping statistics

Example Prompt

Example Prompt

map #scraped_data using schema @schema to #mapped_data using the /map tool

Extract

Parse and extract specific data points using the /extract tool. This tool helps identify and extract targeted information.

Input Parameters

Parameter	Type	Required	Description
content	string	Yes	Content to extract from
extractors	array	Yes	Extraction rules and patterns
format	string	No	Output format (JSON, CSV, etc.)

Output Data

The extractor returns:

• Extracted data points
• Confidence scores
• Validation results
• Extraction metadata

Example Prompt

Example Prompt

extract from #content using rules @extractors to #extracted_data using the /extract tool