CrawlRouter
This is a API that is Firecrawl-compatible and integrates with Tavily, Searxng, Firecrawl, Jina, Google CSE, Scraping Bee, Scraping Ant, Markdowner and Crawl4ai.
I've developed this tool because the different searching and scraping API available don’t have the same format and are not compatible. This software helps to use the tool of your choice with a software that is compatible with Firecrawl API.
It also allows to rotate between providers to stay within the rate limits.
The backend sare configurable through yml file so you can add as many as you want without modifying the code!
Features
- lightweight: image is 64 mo
- several SERP and scraping backend
- backend rotation: randomly or sequentially
- very basic web UI
- backend are configurable through yml file template
Prerequisites
Install the dependencies:
uv sync
cd app
fastapi run app.py --reload or uvicorn app:app --reload --host 0.0.0.0
This will start the API server at http://0.0.0.0:8000.
Environment Variables
The API relies on the following environment variables:
-
SEARXNG_ENDPOINT: Endpoint for Searxng. JSON needs to be activated in the formats in the search section of settings.yml : https://docs.searxng.org/admin/engines/settings.html#search -
SEARXNG_ENGINES: see https://docs.searxng.org/dev/search_api.html -
SEARXNG_CATEGORIES: see https://docs.searxng.org/dev/search_api.html -
SEARXNG_LANGUAGE: see https://docs.searxng.org/dev/search_api.html -
FIRECRAWL_API_KEY: API key for Firecrawl.