n8n Web Scraping: Extract Data from Any Website Automatically

Web scraping with n8n enables competitive intelligence, lead generation, price monitoring, and data collection workflows — all automated and connected to your business tools. This guide covers the best approaches to web scraping in n8n, from simple HTTP fetches to AI-powered data extraction with Claude.

MJK Supplies · May 29, 2026 · 11 min read

ShareX in f ↗

n8n Web Scraping: Extract Data from Any Website Automatically

Web Scraping Options in n8n

n8n supports several scraping approaches:

HTTP Request node: Fetch any URL. Returns raw HTML. Best for simple pages that don't require JavaScript rendering.

Browser automation (Puppeteer/Playwright via Code node): Full browser rendering for JavaScript-heavy sites. More complex to set up; handles dynamic content.

External scraping services: Browserless, ScrapingBee, Apify, Bright Data — they handle proxies, CAPTCHAs, and JS rendering. n8n calls their API.

RSS feeds: For blogs and news sites, RSS is often easier and more reliable than scraping HTML.

Basic HTTP Scraping

For simple HTML pages:

HTTP Request node setup:

Method: GET
URL: target URL
Response Format: String (to get raw HTML)

Code node — extract data:

// Simple HTML extraction const html = $input.first().json.body; // Extract price from specific pattern const priceMatch = html.match(/class="price"[^>]*>([^<]+)</); const price = priceMatch ? priceMatch[1].trim() : null; // Extract title const titleMatch = html.match(/<h1[^>]*>([^<]+)<\/h1>/); const title = titleMatch ? titleMatch[1].trim() : null; return [{ json: { price, title, url: $input.first().json.url } }];

AI-Powered Data Extraction with Claude

For complex or variable HTML structures, pass raw HTML to Claude:

{ "model": "claude-sonnet-4-6", "max_tokens": 1024, "system": "You are a data extraction assistant. Extract structured data from HTML pages. Return only valid JSON.", "messages": [{ "role": "user", "content": "Extract all product names and prices from this HTML. Return JSON array: [{name: string, price: string}]\n\nHTML:\n{{ $json.html }}" }] }

Claude reads messy HTML and returns clean structured data — handling layout variations, encoding issues, and different HTML patterns better than regex.

Competitor Monitoring

Price monitoring workflow:

Schedule trigger: daily at 6am
Loop: for each competitor URL in Airtable
HTTP: fetch competitor pricing page
Claude: extract products and prices
Code: compare to previous prices (stored in Airtable)
IF: price changed → Slack alert
Airtable: update current prices

Content monitoring:

Schedule trigger: weekly
HTTP: fetch competitor blog RSS feed
Loop: for each new article
Claude: "Is this article relevant to our product? Summarise key points."
Slack: send digest of relevant competitor content

Lead Generation via Web Scraping

Job posting scraping:

Company posting "VP Sales" = potential CRM buyer. Company posting "Machine Learning Engineer" = AI tooling buyer.

Schedule trigger: daily
HTTP: scrape job boards (Indeed, Glassdoor, LinkedIn Jobs API)
Claude: "Does this job posting suggest the company needs [your product]? Why?"
Filter: only positive assessments
Clearbit: enrich company data
HubSpot: add to lead pipeline

Directory scraping: Industry directories often list companies with contact information:

HTTP: fetch directory pages
Claude: extract company names, websites, phone numbers, descriptions
Enrichment: Clearbit / Apollo on extracted companies
HubSpot: create contacts

Handling Anti-Scraping Measures

User agent rotation:

const userAgents = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36' ]; const randomUA = userAgents[Math.floor(Math.random() * userAgents.length)];

Rate limiting: Add Wait nodes between requests. 1-3 seconds between requests to the same domain respects server limits and avoids blocks.

Proxy services: For sites with IP-based blocking, route through proxy services: