Capria Ventures, manages a wealth of information across its public web pages and a password-protected GAIN website. To help users navigate this vast knowledge base and get precise answers, we developed the GAIN bot. This bot aims to streamline access to Capria’s insights, whether from open web pages or secure internal articles, making it a game-changer for investors, entrepreneurs, and analysts alike.

The Challenge: Scattered Information and Access Barriers

The GAIN website contains valuable content, but it’s password-protected, making it difficult to scrape directly for automated bots. Furthermore, although these articles are indexed by Google, they aren’t secured properly, leaving gaps in how users access this content efficiently.

Capria Ventures needed a solution where a bot can answer user queries by knowing which web page to reference. The process needs to scrape content from the website, search for answers, and provide relevant information to the user without breaching the security of the password-protected site.

How GAIN Bot Works

To solve this, Capria Ventures decided to develop a bot that uses Google search to locate the most relevant web pages, then scrapes these pages for answers. The architecture is designed as follows:

1. Google Search: The GAIN bot first performs a general Google search based on the user’s question or query.
2. Entity Extraction: The bot extracts entities from the search results (keywords or topics relevant to the query).
3. Refined Search: It then refines the search by appending the term “Capria VC” to the entity name for more specific results.
4. Filtering Capria Links: From the refined search, the bot reviews the top five results and identifies Capria’s official pages.
5. Answer Retrieval: If the bot finds relevant content on these pages, it retrieves the answer. If not, it extends the search to sub-links or related articles to find the necessary information.

(The detailed workflow chart is attached at the end of this article.)

The Problem

Password Protection: Since the GAIN landing page is password-protected, the bot encountered errors when trying to scrape it. The initial problem was that the bot could not bypass the login page, leading to failures in retrieving the relevant content.

Content Access: Articles on the GAIN website are accessible via Google search, but direct access to the password-protected pages was hindered by the security measures.

The Solution: Innovative Scraping and Storage

A variety of approaches were tried to resolve the issue, but the most effective solution involved bypassing the security using cookies.

1. Cookie-Based Solution:
   The bot would pass the password once and obtain a session cookie. With this cookie, the bot could bypass the login page for subsequent requests, scraping content from password-protected pages. However, this solution had limitations:
   • Browser Blocking: If too many requests were made using the same cookie, the browser would block it due to excessive traffic.
2. Storing Web Pages in a Vector Database:
   To optimize the process and avoid repeated scraping, all GAIN pages were stored in a vector database. This database contains pre-scraped content, which is chunked and vectorized for easy retrieval.
   

Ongoing Maintenance

• Whenever new pages or articles are added to the GAIN website, they are automatically logged in Google Sheets. The developers can then access these new entries, which are processed by chunking and vectorization, and stored in the database.
• This approach allows the bot to quickly retrieve relevant content without having to scrape the pages from scratch every time, ensuring a smoother experience for users.

Conclusion

By using a combination of cookies for bypassing login security and storing scraped pages in a vector database, the developer was able to create a more efficient architecture for its bot. This solution ensures that the bot can answer users’ queries effectively, even from password-protected content, while also optimizing the scraping process to avoid unnecessary requests and improve response time.

Here is the working flow of the GAIN bot, in detail. In addition to this flow, threads are attached to the 1st 3 steps (i.e. Entity Based Domain Search Layer, Domain Sub-links Search Layer, and Vector Database layer), which processes the steps in parallel, speeding up the process.