Written as part of our AI Upskilling Program
This article was created as part of the Global Devoteam AI Upskilling Program, where employees share their knowledge to accelerate their learning. The program’s key objective is to provide a foundation in AI for every employee and apply these new skills in our work. Do you want to work with us? Check out our career opportunities.
The ability to quickly access and understand data is crucial for making informed business decisions. Many organisations are looking for ways to empower their teams to explore data and find answers without having to learn complex tools or rely on data experts. Databricks is tackling this challenge head-on with AI/BI Genie, a conversational AI experience that allows users to engage with their data using natural language.
What is Databricks Genie?
Databricks Genie is a no-code, self-service tool that enables business teams to ask questions about their data in natural language and receive immediate, relevant answers. Genie uses generative AI tailored to your organisation’s data and terminology, and it continuously learns from user feedback. This allows users to ask questions as if they were talking to an experienced colleague, receiving accurate and trustworthy insights without needing to write SQL or understand complex BI tools.
How does Genie work?
Genie uses a compound AI system to interpret user questions and generate responses. This system combines multiple AI agents, each responsible for specific tasks such as SQL generation, explanation, and visualisation. This approach allows for the use of multiple specialised models over one single monolith. For more information about compound AI systems, see The Shift from Models to Compound AI Systems.
Here’s a breakdown of how Genie generates responses:
- Table Metadata: Genie uses table names, descriptions, and primary/foreign key relationships from the Unity Catalog to understand the data.
- Column Names and Descriptions: Genie filters for relevant column names and descriptions to include in its analysis.
- Example SQL Queries: Genie selects and learns from relevant SQL examples provided as instructions.
- SQL Functions: Genie utilises any SQL functions added to the space.
- Instructions: Plain text instructions provided by space authors offer important context.
- Prompt and Response History: Genie uses the history of the current chat thread to understand context and refine responses.
Genie does not access the row level data itself; instead, it focuses on metadata and instructions to generate the SQL query needed to produce the requested results.
Key Features of Databricks Genie
Natural Language Q&A: Users can ask questions in plain English (or other languages) and receive instant answers, even for questions not covered by existing dashboards.
Continuous Learning: Genie learns and adapts to your organisation’s data and business semantics, ensuring accurate, AI-generated answers. It also refines its understanding through user feedback and clarification requests.
Governed and Secure Insights: Tight integration with Unity Catalog ensures that all insights comply with established governance and access policies.
Self-Service Exploration: Genie enables business users to go beyond dashboards and dive deeper into their data without coding knowledge.
Visualisations: Genie can generate visualisations of your data from the query results.
Benchmarks: Genie authors can create benchmark questions to track the accuracy of the space as they refine the instructions and settings.
Request Review: Users can request that Genie authors verify responses, adding an extra layer of confidence.
Building trust in Genie
To ensure users can trust the insights provided by Genie, Databricks has introduced features to enhance accuracy and reliability:
- Benchmarks: Data practitioners can use benchmarks to systematically evaluate the accuracy of Genie spaces. These benchmark questions should include common user queries with variations in phrasing, and can be associated with a SQL query to compare Genie’s responses to.
- Request Review: Users can request that a data practitioner review an answer that Genie has provided. This can also be done when Genie asks clarifying questions.
- Certified Answers: For commonly asked questions and established metrics, experts can curate instructions and certified answers for use with Genie ahead of time. These predefined SQL queries can be used to ensure accurate responses.
Getting Started with Databricks Genie
To get the most out of Genie, data teams should focus on properly curating Genie spaces, as this will enable Genie to better understand the data and provide more relevant and accurate results:
Enhance Metadata: Use Unity Catalog to enhance table and column descriptions. This will provide Genie with additional context about the data.
Pre-aggregate Data: For complex relationships or calculations, pre-join tables into views or create views that pre-aggregate metrics. This will improve query performance and make it easier for Genie to generate accurate insights.
Supply Instructions: Provide clear instructions to help Genie understand the business context and how to handle specific data scenarios. This will ensure that Genie’s output is aligned with business needs.
Provide Example Queries: Show Genie how to perform calculations by providing example queries. This will help Genie learn how to interact with the data.
Conduct User Testing: Recruit business users to test the space and provide feedback. This will help identify any areas where Genie can be improved.
Conclusion
Databricks Genie is a powerful tool that democratises access to data and empowers business teams to make data-driven decisions. By combining the power of generative AI with the robust data governance of the Databricks platform, Genie is poised to revolutionise how organisations interact with their data. Its ability to learn from user interactions, its focus on accuracy, and its ease of use makes it a valuable asset for any data-centric organisation.
The possibilities for leveraging Genie to drive insights are endless. Start exploring your data with Genie today and experience the future of self-service business intelligence. Discover our Databricks Consulting Services