Enhancing your chatbot's effectiveness involves not only uploading documents but also expanding its knowledge base by crawling websites. This guide provides detailed instructions on how to utilize the website crawling feature in Resinq, alongside document uploading, to maximize the information available to your chatbot.
Accessing Web Crawl Interface
Navigate to Sources
Log into your Resinq dashboard.
- Click on Chatbot then navigate to Sources to access crawling functionalities.
Prepare for Crawling
Switch to the Web Crawling tab.
- Enter the URL of the website or a direct link to a sitemap to initiate the crawling process.
Enter Website Link
Input the URL you wish to crawl. You can enter a direct website link or a sitemap for comprehensive crawling.
- Review the links that the system will crawl, displayed for confirmation.
Start Crawling
Click Start Crawling to begin the process. The system will crawl the website to a depth of two links, ensuring substantial yet focused content gathering.
Depth of Crawling Details: When you input a direct website link, Resinq's crawling feature is designed to reach a depth of two links. This means it will access the initial page and any linked pages, but will not go beyond pages linked from these secondary pages.
This helps maintain focus on relevant content without excessive token consumption. Importantly, this depth limitation does not apply when you use a sitemap. With a sitemap, Resinq can crawl more comprehensively based on the structure and links defined within the sitemap itself.
Managing Crawled Content
After the crawling process completes, you can manage and review the crawled content:
- View Details: Inspect the detailed list of crawled pages and the content extracted from each.
- Token Usage: Navigate to the token usage panel to see a summary of tokens used by the crawled content.
Best Practices for Crawling and Uploading
To ensure efficient use of resources and optimal results:
- Quality Over Quantity: Focus on crawling websites that are most relevant to your chatbot's domain.
- Regular Updates: Continuously update the crawled data to keep the chatbot's responses accurate and up-to-date.
Troubleshooting Common Issues
Crawl Not Starting
Low Depth Crawling Issues
By following these steps and tips, you can effectively enhance your chatbot's knowledge base through both document uploading and website crawling on the Resinq platform, making it more capable of delivering precise and helpful responses.