A Big Data Analytics System Cross-Analyzing 30,000 Attributes With 100x Faster Reporting

Summary

INNERLUXES helped a market research company migrate from its legacy analytical system to a future-ready big-data platform. Built on Hadoop, Hive, and Spark across AWS and Azure, the system ingests over a thousand raw data types and cross-analyzes nearly 30,000 attributes — running some queries up to 100 times faster and powering comprehensive advertising-channel analysis across many markets.

About the Client

The Client is a leading market research company.

A Legacy System That Couldn't Scale to Future Needs

Although the Client had a robust analytical system, it believed the system would not meet its future needs and was looking for a future-focused, innovative solution. The new system had to cope with continuously growing data volumes, analyze big data faster, and enable comprehensive advertising-channel analysis. After deciding on the architecture, the Client sought a team highly experienced in big-data implementation and — based on a long-standing relationship — engaged INNERLUXES to handle the entire migration from the old system to the new one.

A Big-Data Platform Built on Hadoop, Hive, and Spark

Throughout the project, the Client's business intelligence architects worked closely with INNERLUXES's big-data team: the architects designed the concept, and INNERLUXES implemented it. For the new system, the architects selected Apache Hadoop for data storage, Apache Hive for data aggregation, querying, and analysis, and Apache Spark for data processing, with Amazon Web Services and Microsoft Azure as cloud platforms. At the Client's request, the old and new systems ran in parallel during migration. The solution comprised five main modules:

• Data preparation. Raw data from many sources — TV views, mobile browsing history, website visits, and surveys — flowed into the system, which had to handle more than 1,000 raw data types (archives, XLS, TXT, and more). Coded in Python, this module covered data transformation, parsing, merging, and loading.
• Staging. Built around Apache Hive, this module kept a structure similar to the raw data, with no established connections yet between respondents from different sources such as TV and the internet.
• Data warehouse 1. Also based on Apache Hive, this module performed data mapping — for example, processing respondent data for radio, TV, internet, and newspaper sources and linking user IDs across sources by mapping rules. Its ETL was written in Python.
• Data warehouse 2. With Apache Hive and Spark at its core, this module processed data on the fly per the business logic — calculating sums, averages, probabilities, and more. Spark DataFrames handled SQL queries from the desktop app, and Spark also filtered query results by each user's access rights. Its ETL was coded in Scala.
• Desktop application. A comprehensive analytics client (detailed below).

Using WPF and C#, INNERLUXES built the desktop application for in-depth cross-analysis of nearly 30,000 attributes, generating detailed intersection matrices for diverse market analytics. The MVVM pattern kept a clean separation of concerns to simplify future enhancements and maintenance. Users could run standard and ad hoc reports — such as Reach Ranking and Share of Time — and define custom parameters for personalized insights, with a responsive XAML-based design presenting complex data through simple, interactive charts. The system also offered forecasting, for example predicting revenue from expected reach and advertising budgets.

Results

By project close, the new system processed several queries up to 100 times faster than the legacy solution. With the insights from analyzing nearly 30,000 attributes, the Client could carry out comprehensive advertising-channel analysis across different markets.

Technologies and Tools

Apache Hadoop, Apache Hive, Apache Spark, Python (ETL), Scala (Spark, ETL), SQL (ETL), Amazon Web Services (cloud storage), Microsoft Azure (cloud storage), .NET, WPF, C#.