Cannot scrape website for knowledge base

Hey getting this error when adding a site to the KB: "Processing URL failed. Ensure the URL is valid and accessible by bots or crawlers."

However nothing in our client's robots.txt should block crawlers, and the URL is valid. Any easy workarounds or is a custom scraper required?

Thanks