In today’s fast-paced world, where new AI evolutions emerge daily and companies grapple with monster-sized datasets, creating space for innovation isn’t optional—it’s essential. In 2024, Hydrolix recognized this need when co-founder Hasan Alayi drafted a long list of projects for the team to explore. That list became the foundation for the Hydrolix Innovation Lab, a dedicated environment led by Director Alok Aggrawal and managed by Senior Product Manager Ashley Vassell.
Today, the lab includes fifteen specialized AI, machine learning, and data engineers who focus on a range of projects designed to solve real customer problems and accelerate the company’s ability to innovate.
The Early Days
“The initial focus was on bringing to market a new Splunk connector, built by Hydrolix co-founder Hasan Alayi, to integrate into the Hydrolix streaming data lake,” said Aggrawal. “We’d heard from customers that they value their Splunk dashboards but struggle with the rising costs of retaining terabyte-to-petabyte-scale data. If they wanted to maintain data for more than a week, Splunk costs soared, yet they still wanted to keep their Splunk UI. We wanted to offer them the best of both worlds: continue using your Splunk dashboards, but save money by using Hydrolix on the backend.”
Because Hydrolix can compress data 25–50x, the connector allows customers to retain all their data at a fraction of the cost.
“Due to our work in the Innovation Lab, we have become a supplemental solution to the Splunk installation base,” Aggrawal continued. “Additionally, recognizing that data is the foundational basis for machine learning and AI model development, we sought to provide our customers with a means to retain and query all their data affordably and quickly, regardless of size or age. That meant Hydrolix data had to be available to ML and AI engineers in their most common practice environment, which is Apache Spark. We built a Spark connector that seamlessly integrates with existing tools, such as Databricks, Microsoft Fabric, and AWS EMR. With that connector, customers could store all their data and keep it hot. Those were the first two wins for the Hydrolix Innovation Lab.”
Making Data More Accessible
The next challenge emerged when the team noticed that many customers lacked internal machine learning expertise.
“We observed this trend emerge after building the connectors and decided to expand our focus,” said Aggrawal. “We created our own MCP server to make Hydrolix data available not only to engineers proficient in SQL but also to anyone who wants to interrogate data in natural language. That means even non-technical roles in a company, such as a CMO or HR leader, can query Hydrolix data for their own use cases.”
This step allowed teams outside traditional data roles to access Hydrolix’s full analytical power without writing a single line of code.
Strategic Innovation Meets Practical Development
According to Aggrawal, HIL takes on both strategic, big-picture work and tactical projects that address immediate customer needs.
“It’s a mix of both. We selectively pick tactical work,” he explained. “For example, we heard from Hydrolix customers who primarily use Grafana that they faced issues with the ClickHouse Grafana plugin. To address this issue, our engineers developed a Grafana plugin for Hydrolix data sources, enhancing the query experience. We enriched our offerings by releasing a query assistant. Currently, the assistant is in beta mode and is designed for customers who want to take their Grafana dashboards to the next level, modify queries to meet their business needs, and utilize AI to explain the functionality of their queries. The assistant takes a SQL query, breaks it down, and explains what each variable and macro means so that users who don’t write SQL can build a specific dashboard for their particular use case.”
For users, this approach provides a more accessible and customizable data experience, eliminating the bottleneck of waiting for engineering support.
The Power of Correlation: Anomaly Detection
While the lab’s early projects focused on accessibility and efficiency, its latest efforts are centered on advanced AI-driven analytics.
“As we announced a few weeks ago, Hydrolix purchased intellectual property from Quesma to continue supporting Kibana interoperability for our customers,” said Vassell. “We’re now focusing on integrating the Kibana Gateway (formerly known as the Quesma IP) into our Hydrolix streaming data lake and adding summary tables to support it. We’re also focusing on anomaly detection.”
The lab’s anomaly detection system uses machine learning to correlate data from multiple metrics, identifying the root cause of issues more accurately than traditional systems.
“Our anomaly detection system is ML-based and focuses on correlating data from different metrics to pinpoint the root cause of an anomaly,” Vassell explained. “It’s unique because, with basic detection systems, you typically only access one set of metrics, whereas Hydrolix correlates these metrics. If you observe a spike in one metric and the other correlated metric behaves differently, that indicates an anomaly. And we’re using LLMs for natural language correlation of the anomalous data.”
From Alerting to Action
Traditional anomaly detection systems can tell users that something is wrong, but not what it means or where to look. The Hydrolix Innovation Lab’s approach changes that dynamic.
“With most detection systems, you receive an alert indicating a value is high, but you don’t know what it means or where to look,” said Vassell. “Hydrolix’s patented anomaly detection system, which we developed in the Innovation Lab, enables users to directly identify the root cause of the anomaly and make immediate decisions to mitigate. It’s action-oriented, rather than just another dashboard that people constantly monitor. It’s highly accurate and solves real problems for users instead of alerting them to things they won’t care about.”
She offered a practical example: “Let’s say you’re concerned about cache performance degradation. The CDN has to return to the customer’s origin more frequently to retrieve content because it’s not being cached. We detect this issue by examining how often the CDN cache is being accessed. If that number is decreasing, it means there’s an issue with the CDN provider. Hydrolix also ingests full-fidelity data, and a lot of it. We often see anomaly detection being done with sampled data, which can introduce bias or noise. Since Hydrolix takes in, retains, and analyzes all data, there’s no need to test; anomaly detection is therefore more accurate.”
For companies lacking specialized teams, the system eliminates the manual labor typically required for analysis.
“Our customers don’t have to do any manual operation around our anomaly detection system,” Vassell added. “They receive alerts via email. Those alerts link to a dashboard that displays a root cause analysis report, identifying the affected region where Hydrolix detected anomalous streams and suggests a mitigation action. Actions could be blocking IPs or improving WAF rules. No matter the issue, Hydrolix’s anomaly detection system leads customers to how to solve it.”
The full solution, she said, will be completed by the end of this quarter.
Looking Ahead
When asked what’s next for the lab, Vassell pointed to several key areas of exploration.
“We’re researching agentic AI and how to bring those capabilities into Hydrolix,” she said. “We’re also exploring vectorization and embeddings, which enable users to conduct more effective searches. Piracy detection is another big one. When considering bot detection, whether it’s AI scraping bots or media streaming bots stealing streams, piracy detection, and the ability to more accurately identify those instances with full-fidelity data versus using sampled data are critical.”
The team is also expanding on its natural language capabilities. “Our MCP server already enables some of them,” Vassell explained. “The goal is for a salesperson, for example, to interact with Hydrolix in natural language, such as ‘Tell me what my sales were last week.’ Hydrolix would send a response with a chart. The visual would be a chatbot that’s embedded into our console. We want to add the ability to say ‘Make this into a dashboard,’ and have it generate a dashboard, or it could stay in a chat.”
Scaling the Lab’s Success
Aggrawal reflected on the lab’s achievements so far and the vision ahead.
“I’m pleased with what we’ve built and the projects we’ve delivered in such a short amount of time,” he said. “From the Splunk and Spark connectors to the MCP server, anomaly detection, the Grafana plugin, and now the Kibana Gateway, we’ve released a significant number of updates. I hope that we expand these—and those in the pipeline—to more markets globally. We aim to deliver end-to-end, integrated, turnkey solutions that address a set of key pain points.”
Hydrolix’s Innovation Lab has quickly become the company’s creative core—where problem-solving meets experimentation, and where customer needs drive every line of code. From connectors to AI-powered analytics, the lab continues to shape the future of real-time data management.
To learn more about Hydrolix’s latest connectors and the importance of retaining all high-dimensionality, high-cardinality data, visit the Hydrolix blog.