Challenge
Beck's Hybrids, a family-owned seed company dedicated to helping farmers succeed, has produced more than 100 studies annually for nearly 50 years through the company’s Practical Farm Research (PFR) tool. This proprietary research features invaluable data about soil, weather conditions and how to prevent disease that can help farmers optimize their practices. However, because the data is stored on Beck’s intranet and saved via brochure-style PDFs, it was difficult to access and search.
Beck's needed a cutting-edge solution to improve the accessibility and usability of their agricultural research data. The company reached out to Six Feet Up to build a secure framework that could transform Beck's extensive research into a digestible, easily searchable format, all without compromising proprietary information.
Six Feet Up set out to develop an AI-Powered, Large Language Model-driven chat interface.
Implementation Details
Six Feet Up leveraged the power of AI and Large Language Models (LLMs) to develop an intuitive chat interface, streamlining the process of accessing Beck's farming research. The system, despite its seemingly simple user interface, comprised three main subsystems:
- Data Gathering Framework: This subsystem was engineered to locate and break down documents into a more usable format. The framework extracted data from numerous PDFs residing on Beck's intranet, overcoming the challenge of decoding research papers laid out like brochures.
- AI Framework: The core of the solution, this framework incorporated AI models from OpenAI, LangChain, and Llama Index. Each model was fine-tuned and tested rigorously to ensure it delivered accurate, relevant answers.
- Layering Framework: This layer effectively integrated Beck's research into the AI models, transforming the data into a format that the "bots" could understand and learn from.
The project was managed using Python — the 'lingua franca of AI.' This language allowed for faster iteration of ideas. Django was used for the backend, while HTMX was used for the UI. For deployment, the team used Kubernetes, simulating a cloud environment on their machine. This approach made it easy to replicate the setup in different environments.
Throughout the development, the Six Feet Up team tackled challenges related to rapidly evolving libraries, and sub-dependency conflicts in this ever-evolving area. They managed these obstacles by locking down every aspect of a working system until new requirements emerged.
The final deliverable was hosted on Beck's own hardware to ensure security. In addition, assurances were sought from OpenAI to ensure they wouldn't retain or learn from Beck's proprietary information.
Results
Despite the data security concerns and the challenge of extracting data from complex research papers, Six Feet Up successfully developed an AI-powered solution that unlocked Beck's wealth of agricultural research data. The project demonstrates how AI can make a significant impact in the agricultural sector, paving the way for more efficient, sustainable and profitable farming practices.
With AI response times reduced from over a minute to around 2-8 seconds on average, the system has shown remarkable efficiency in summarizing information across multiple documents.
In addition to streamlining access to Beck's extensive research data, with the implementation of the new AI-powered chat interface, the company can now aggregate historical data to extract valuable patterns, trends, and correlations from their repository spanning five decades. This information will help farmers make data-driven decisions that advance agricultural practices and boost crop yields.
The tool has also shown the potential to accommodate future upgrades, such as integration with OpenAI’s GPT-4. However, scaling up the project to handle terabytes of data would require different architectural considerations for storing the processed documents.
