Cannot scrape website for knowledge base

Alexzander Flores · 2025-12-01T20:05:34.891Z

Hey getting this error when adding a site to the KB: "Processing URL failed. Ensure the URL is valid and accessible by bots or crawlers." However nothing in our client's robots.txt should block crawlers, and the URL is valid. Any easy workarounds or is a custom scraper required? Thanks

# Let your agents search the web 🔎 Hey everyone! We just released the web search tool for the agent step! Your agents can now automatically search the web for information, letting your agent supplement the LLM's knowledge and the data in its knowledge base with live, up-to-date information. Plus.... * You can restrict searches to specific domains, so your agent only searches sites that you own * This is a tool, so you remain in control of when the agent searches the web * Results are automatically summarized in a way that your agent can automatically understand Under the hood, we're using OpenAI's web search API. Give it a try, and let us know what you think! https://docs.voiceflow.com/changelog/native-web-search-tool

jacklyn · 4mo ago

jacklynbiggin's Thread

jacklyn · 4mo ago

connor_maclean's Thread

hurt-tomato · 4mo ago

Cannot scrape website for knowledge base

Similar Threads