Cannot scrape website for knowledge base
Hey getting this error when adding a site to the KB: "Processing URL failed. Ensure the URL is valid and accessible by bots or crawlers."
However nothing in our client's robots.txt should block crawlers, and the URL is valid. Any easy workarounds or is a custom scraper required?
Thanks
1 Reply
Hey any ideas? Would a custom scraper be required in these cases?