equal-aqua•15mo ago
KB population with gpt-Crawler
Hello,
I have run into a problem that I did not experience before the new update of Voiceflow.
I created gpt-crawler from https://github.com/BuilderIO/gpt-crawler.git repository. The basic premise of such a function is to crawl a website and upload links to the knowledge base without having to copy and paste many links manually. Since we are using such a function to crawl and upload links for real estate agencies' listings and they change every month this function seems to fit us the most. Now after the recent voiceflow update, this function does not enrich my knowledge base with new links, I checked that API key and Project ID is correct and website links are still present. Can you suggest any options for how I can enrich my knowledge base without having to pay for Tavily AI?
If you have any other Ideas please let me know.
3 Replies
Hello,
The gpt-crawler you are using is a modified version available at this repository. This tool is designed to fetch content from a website, convert it into markdown, and upload everything to your Voiceflow agent knowledge base as text.
If you're looking for other tools provided by Voiceflow, you might consider using the Voiceflow Date Parser API. This API can parse human-like dates such as 'tomorrow at 1000' into ISO 8601 formatted dates. You can find the repository for this API here.
Additionally, Voiceflow provides a variety of other resources and tools. For instance, you can leverage Microsoft's LLMLingua2 for efficient prompt compression, enhancing your Voiceflow agent's performance, tokens usage, and reducing latency. You can also explore integrating the latest OpenAI's GPT-4o model with a fallback to GPT-4 Turbo using Cloudflare Al Gateway. The repository for this can be found here.
For more detailed instructions and updates, you can refer to the following resources:
- Voiceflow Crawler Uploader Repository
- Voiceflow Developer Documentation
FireCrawl - https://github.com/mendableai/firecrawl
GitHub
GitHub - mendableai/firecrawl: 🔥 Turn entire websites into LLM-read...
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API. - mendableai/firecrawl
equal-aquaOP•15mo ago
thx man! Will check it out!