Allow using another web provider than Firecrawl

[Tketa]

Hi team,

First of all, thanks for this framework. It's a great starting point and I've been using it for some pharmaceutical analysis use-cases I have.

I've been using Linkup as my web search provider. It's fairly cheap (0.005$ per standard query and has a "deep search" mode for 0.05$ per query). Also it's French 🇫🇷

On my local repo I implemented that by adding a WEB_PROVIDER env variable to manually select linkup (firecrawl being the default for backward compatibility). You can basically get the idea from this PR

I created a tools/web-providers.ts file that defines a common interface and creates the implementation for both providers. That interface will now be used by tools/web.ts so there's a bit of code changes there

export interface WebClient {
  fetch(url: string): Promise<FetchResult>;
  search(query: string, count: number): Promise<SearchResponse>;
}

Alternatively we could have tools/web-providers/{types,firecrawl,linkup.ts} if we don't want it all in one file.

What do you think about this? Let me know if I can make a PR in this direction.

Also, are you planning on implementing tests at some point? There is no tests or CI at the moment, which would be better in case you want people to start contributing I believe. In my PR I initially included some tests using vitest but since it's quite opinionated it's probably better if you decide what you want to use / some guidelines and I'd align with this?

Thanks in advance,
Jérémy

[spolu]

Hi tketa. We plan to transition to using this browse tool always: https://github.com/dust-tt/browse which will remove the dependecny to an external web provider.

ooc what was your rationale for using linkup vs firecrawl?

[spolu]

And more importantly what kind of result did you achieve? Anything interesting?

[Tketa]

I see, I will give dust-tt/browse a try to see how it performs vs my Linkup approach. My use case involves searching broadly across PubMed, Biorxiv, patents, etc.. And then drilling down. Linkup search enables that extremely well for the agents, and converges fast.

I've not run a proper benchmark tbh between Linkup and Firecrawl.

I find the Linkup credits approach working better for me vs subscription. Search is fast, and it often allows to bypass the Cloudflare protection

[Tketa]

The thing I don't understand is that browse only seems to provide fetch capability, not search? Or am I missing something?

[spolu]

browse can search on google directly but that indeed might be less streamlined than those providers.

Happy to incorporate your changes 👍

On tests: this is very much a research project for now, tests will come at a later stage when/if that makes sense 👍

[spolu]

Let's go with a shared interface and one file per providers 👍👍👍