The context:
A US-based B2B SaaS company set out to build a platform that would function as a “Google for Events” - a unified search and discovery engine for offline events happening across cities, venues, and local communities.
While the product vision was strong, the real challenge lay in the massive and fragmented data ecosystem behind local events. Event information was available across thousands of websites, each with its own format, structure, and reliability. To power their platform, the company needed a way to collect, process, and standardize this data at scale while maintaining high quality.
The Devstark team developed an AI-powered event aggregation and data processing pipeline that automatically collects event information from thousands of municipal and venue websites. The system extracts key details such as dates, locations, and descriptions, and converts them into a clean, standardized format. Built-in validation checks remove duplicates, detect inconsistencies, and help maintain high data accuracy. A custom admin dashboard also allows the team to monitor sources in real time and manage quality assurance workflows.
With the new system in place, the company scaled much faster, growing from onboarding one client every two months to three to five clients per month, while consistently maintaining over 95% data quality.
The challenge:
The client needed a reliable way to continuously collect accurate, real-time information about local offline events. However, this information was spread across hundreds of different websites, municipal portals, venue and club pages, local event listings, community organization sites, and tourism calendars. Each source had its own layout, structure, and update schedule, which made automated data collection extremely challenging.
Before working with our team, the client relied on a large set of custom-built web parsers, each designed to extract data from a specific website. While this approach worked initially, it quickly became difficult to maintain as the number of sources grew.
Several problems began to appear.
- High maintenance costs. Whenever a website changed its layout or structure, the corresponding parser would break and had to be manually fixed.
- Limited scalability. Expanding to a new city or adding new event sources meant building entirely new parsers from scratch.
- Data quality issues. The system frequently produced duplicate events, conflicting dates, or incomplete event details.
- Slow customer onboarding. Because each new client required configuring and validating additional event sources, the company could only onboard one new client every two months.
Over time, all these issues became a major bottleneck. The platform struggled to keep up with new data sources, and the company’s growth slowed as a result. Without a more scalable solution, expanding the platform to new cities or markets would have been nearly impossible.
Solution development:
To solve these challenges, we designed and implemented an AI-powered event aggregation and data processing pipeline that automated the entire lifecycle of event data.
The goal was to replace rule-based parsers with a flexible architecture capable of understanding and structuring event data across thousands of different websites. At its core, the system collects raw data from a wide range of sources, including municipal portals, venue websites (such as pubs, churches, and local clubs), and other local online resources, ensuring broad and consistent coverage of hyper-local events.
Once raw content is collected, AI models—powered by LLMs analyze the pages and automatically extract structured event information. This includes key attributes such as the event name, date and time, venue and location, description, organizer or host, and event category.
The system performs entity extraction and normalization, identifying relevant event details and standardizing them into a unified internal format. It also filters out irrelevant content, noise, and spam to ensure only valid event data is processed.
Unlike traditional parsers that rely on rigid HTML patterns, the AI extraction layer can interpret different page structures and text layouts, making the system far more resilient to website changes. The extracted data is then normalized into a consistent schema, ensuring that all events across thousands of sources are clean, structured, and ready for reliable B2B use.
To ensure production-grade reliability, we implemented a Quality Gate Agent responsible for validating event data before it becomes available to clients. This layer acts as an automated data quality checkpoint thus performing the kind of verification a human team would typically handle, but much faster and at scale.
The system performs several key checks:
- Deduplication across sources identifies duplicate listings and merges them into a single, clean record.
- Conflict detection automatically detects contradictions in the data, such as mismatched dates, times, or venue details.
- Address and venue validation happens with the help of external APIs. The system verifies that event locations are accurate, valid, and properly formatted.
- Edge case flagging occurs if an event contains missing, unclear, or suspicious data, and then it is automatically flagged for manual review.
Overall, this Quality Gate layer performs human-level data quality checks, but in a much faster, more scalable, and more consistent way, providing high accuracy without slowing down the pipeline.
To give the client full visibility and control over the pipeline, we developed a comprehensive Aggregator Admin Dashboard. This interface allows the operations team to monitor and manage the entire event ingestion ecosystem from a single place, combining real-time insights with powerful QA and operational tools.
Key capabilities include:
- Real-time source monitoring, which allows each source to be tracked live with clear statuses such as active, processing, error, or pending review.
- QA workflows with approval and rejection flows, enabling source-level quality assurance. Teams can review extracted events, approve valid entries, reject incorrect ones, and efficiently handle flagged edge cases.
- Performance metrics and failure analysis, offering detailed insights into success rates, extraction statistics, parsing failures, and overall source reliability, helping continuously improve data quality.
- Automated alerts, which notify the team when a source fails, stops updating, or shows signs of data quality issues, allowing for quick intervention.
- Bulk operations and source management make it easy to perform mass approvals, schedule updates, and manage configurations across multiple sources at once, significantly reducing operational overhead.
Overall, the dashboard transforms a complex, distributed data pipeline into a manageable and transparent system, giving the team full control over quality, performance, and scalability.
By automating data processing and streamlining operations through the dashboard, the client increased onboarding capacity from one client every two months to three to five clients per month.
Results and business Impact
After implementing the AI-powered aggregation system, the client quickly saw improvements in efficiency and scalability.
- Onboarding new customers became much faster. Previously, the team could only onboard one client every two months because of the manual work required to configure data sources and fix parser issues. With the new automated pipeline, they can now onboard three to five clients per month.
- The platform also dramatically reduced manual work. Instead of maintaining dozens of parsers, the system now automatically processes thousands of event sources, continuously collecting and structuring new event data.
- At the same time, data quality improved significantly. With built-in AI validation, deduplication, and conflict detection, the platform now maintains over 95% data accuracy, ensuring that clients receive reliable event information.
- Another major benefit was the complete removal of parser maintenance. The old system required constant fixes whenever websites changed their structure. The new AI-based approach eliminated this ongoing operational burden.
- Finally, the platform is now built on a scalable architecture that can easily support expansion into new markets. The client can onboard new cities, tourism platforms, airports, hospitality partners, and event discovery applications without rebuilding their data infrastructure.
By replacing fragile rule-based parsing with an AI-driven aggregation pipeline, the company transformed its product into a robust and scalable engine for hyper-local event discovery. The platform can now collect, structure, and validate event data from thousands of sources, allowing the business to grow faster and expand into new markets with confidence.
Technologies used:
OpenAI
OpenAI provides models like GPT (Generative Pre-trained Transformer), which excels in natural language understanding and generation. These models can be used to build everything from chatbots and content generators to code assistants and data analyzers. OpenAI's API is incredibly versatile, allowing easy integration into web and mobile applications.
Python
Python is an interpreted, object-oriented, high-level programming language with dynamic semantics.
PostgreSQL
A powerful, open source object-relational database system with over 30 years of active development that’s earned a strong reputation for reliability, feature robustness, and performance.
Node.js
Lightweight and efficient, Node.js is perfect for data-intensive real-time applications that run across distributed devices.
TypeScript
TypeScript is an open-source programming language that extends JavaScript by adding static typing. It helps developers catch errors early, improve code quality, and enhance maintainability.TypeScript is compatible with modern JavaScript frameworks like React, Angular, and Vue.js, making it a go-to choice for high-quality software development.
React.js
The library for web and native user interfaces. Whether you work on your own or with thousands of other developers, using React feels the same. It is designed to let you seamlessly combine components written by independent people, teams, and organizations.
AWS
Whether you're looking for compute power, database storage, content delivery, or other functionality, AWS has the services to help you build sophisticated applications with increased flexibility, scalability and reliability
.svg)
