Web Scraper

Every other node in this module talks to a system that agreed to be talked to. A website did not. There is no schema, no authentication you were given, no promise the markup will look the same next week — and yet most of what you want to know about a company is on its own site, written by them, for free.

The Web Scraper reads it. The size of its configuration is not complexity for its own sake; it is what fetching arbitrary pages politely and reliably actually costs.

crawl_mode decides how many pages you get and how they are chosen: one page, everything in a sitemap, links discovered by following the site, or a list you supply. discover with a small max_pages is the useful default for “tell me about this company” — it follows links and stops.

extract_mode decides how much of each page survives. smart strips navigation, footers, sidebars and advertising and keeps what reads like content — which is what you want unless you are extracting a specific element, in which case selectors and your own CSS selectors do it precisely.

The engine is how the page gets fetched, and it has three real behaviours behind four names.

If you are scraping at any volume, pin the engine rather than leaving it on auto. Run one page each way first and look at {{alias.char_count}} — that tells you which engine that site needs, and then you choose it deliberately.

A company is added to the CRM. The workflow reads a few pages of its own website and files what it found against the record.

The patterns are the interesting part. priority_patterns pushes the crawler toward about, services, products, pricing and contact — the five pages that actually describe a business. exclude_patterns keeps it out of the blog, the news and the careers section, which are enormous and tell you nothing about what the company sells.

Without those, max_pages: 5 spends itself on whatever the homepage links to first, which is usually three blog posts and a cookie policy. Steering the crawl is worth more than raising the limit.

request_delay_ms is set above the default here. A second between requests is polite; a second and a half on a small site is politer, and the difference to your workflow is four seconds.

An empty scrape looks like a successful one. A page that blocks you, or renders entirely in JavaScript over plain HTTP, returns almost no text — not an error. The workflow carries on and files an empty note, or hands a model nothing to summarise and gets a confident summary of nothing. Check {{alias.char_count}} and branch when it is implausibly small. This is the single most useful guard you can add to a scraping workflow.

Robots rules are respected by default, and you should leave them that way. respect_robots_txt defaults on. Turning it off is a decision about somebody else's server and your organisation's reputation, not a configuration tweak — and a site that has asked not to be crawled will often notice.

Selectors rot. If you use extract_mode of selectors against a site you do not control, your workflow depends on someone else's CSS class names. It will work perfectly until a redesign, then return nothing — silently, per the point above. smart mode degrades more gracefully because it has no idea what the class names are.

Scraped text is somebody else's writing. Putting it straight into a CRM field, an email or a model prompt means whatever is on that page is now inside your workflow — including text written to manipulate whatever reads it. Treat it as data, never as instructions, and be careful about pasting it into a prompt without saying where it came from.