Web Search

The scraper reads a page you already knew about. This node is for when you do not — a company you have never heard of, a name on an inbound enquiry, whether anything happened to an account this month.

Its configuration is the smallest in this module. What makes it worth a lesson is everything the configuration does not tell you.

The result count is capped at twenty however you set it. Ask for a hundred and you get twenty. That is a sensible limit — search results past the first page are rarely what you wanted — but it is silent, so a workflow written expecting fifty is quietly working with less.

time_range is the field people forget and then wish they had used. Searching a company name across all time returns whatever ranks best, which is usually their homepage and a directory listing. Restricting it to the last month returns what actually happened recently, which is almost always the question.

search_type of news narrows to news sources; web is everything. For monitoring an account, news plus a month is the pairing that works.

There is no provider field, and the reason is that the choice is made on the server rather than in the node. It tries a paid search API if one is configured for the deployment, then a search library if it is installed, and failing both it falls back to scraping a search page directly.

A question in chat becomes a recent-news search, and the reply is built from what came back — with the empty case handled explicitly, because it is common.

The reply reaches into the results with an indexed path — {{websearch_1.output.results[0].title}} — rather than looping over them. That is deliberate: as the Text Response lesson showed, loop and conditional directives are stripped before a reply node runs, so a template can substitute values but cannot iterate. If you need one line per result, build the list upstream.

The condition exists because empty is a normal outcome, not a failure. A search that finds nothing returns successfully with a count of zero. Without that branch, the reply would be a heading followed by nothing, which reads to the user as a broken product rather than an honest “I could not find anything”.

region is set to India-English here. It changes which results rank, not which are legal to return, and the default of worldwide is rarely the best answer for a business that operates in one market.

Nothing marks the substitution. The count is non-zero, the results look plausible, and the workflow proceeds as though the original question was answered. It is a well-meant feature — a slightly-too-specific query returning something is usually better than nothing — and it is dangerous the moment the result feeds a decision rather than a human's eyes.

Two defences. Keep queries short enough that the fallback would be the same query anyway — three or four words. And when the result matters, put the search terms in the reply so whoever reads it can see what was actually asked.

The pairing this node exists for is search then scrape: find candidate pages here, hand top_urls to the scraper, and read the ones that matter. Search gives you snippets; the scraper gives you the page.

Snippets are not facts. A search result's snippet is an extract chosen by a ranking algorithm to look relevant to your query. Feeding snippets to a model and asking it to summarise produces something that reads authoritative and is assembled from fragments of pages nobody opened. If the answer matters, scrape the page.

Results are not stable. Run the same query twice and the order can change, and so can the set. A workflow that assumes results[0] is the right one is making a bet that gets settled differently each run.