Skip to content

Use secureFetch in WebScraperTool #5678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
christopherholland-workday wants to merge 1 commit into from
Closed

Use secureFetch in WebScraperTool #5678

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/components/nodes/tools/WebScraperTool/WebScraperTool.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import { INode, INodeParams, INodeData, ICommonObject } from '../../../src/Interface'
import { getBaseClasses } from '../../../src/utils'
import { Tool } from '@langchain/core/tools'
import fetch from 'node-fetch'
import * as cheerio from 'cheerio'
import { URL } from 'url'
import { xmlScrape } from '../../../src/utils'
import { secureFetch } from '../../../src/httpSecurity'

interface ScrapedPageData {
url: string
Expand Down Expand Up @@ -60,7 +60,7 @@ class WebScraperRecursiveTool extends Tool {

private async scrapeSingleUrl(url: string): Promise<Omit<ScrapedPageData, 'url'> & { foundLinks: string[] }> {
try {
const response = await fetch(url, { timeout: this.timeoutMs, redirect: 'follow', follow: 5 })
const response = await secureFetch(url, { timeout: this.timeoutMs, redirect: 'follow', follow: 5 })
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The secureFetch function handles redirects manually and overrides the redirect option. The follow option is also not used by secureFetch's internal logic. To avoid confusion and improve clarity, these properties can be removed from the call. secureFetch defaults to a maximum of 5 redirects, which preserves the original behavior of follow: 5.

Suggested change
const response = await secureFetch(url, { timeout: this.timeoutMs, redirect: 'follow', follow: 5 })
const response = await secureFetch(url, { timeout: this.timeoutMs })

if (!response.ok) {
const errorText = await response.text()
return {
Expand Down