The Client
The Quality Education and Skills Training (QUEST) Alliance is a not for profit organisation that focuses on research-led innovation and advocacy in the field of teaching and learning. QUEST aims to create innovative, engaging learning experiences through the use of modern technology, to enable learners and facilitators to develop 21st century skills in their own time, at their own pace. They engage with multiple stakeholders to demonstrate and enable scalable solutions in education and vocational training using Information and Communication Technology (ICT).
QUEST focuses on children & youth, age 10-30, and provides them with a set of real-world skills along with opportunities to build confidence in a fun and engaging way that prepares them for work and life. QUEST is on a mission to empower 4 million learners & facilitators with 21st century skills for social and economic growth by 2023
Srvices Provided
Tech Stack Used
Talend
MySQL
Apache Superset
Firebase - BigQuery
The Problem
The client had earlier tried multiple reporting tools like Zoho Reports, Tableau and PowerBI to create reports and dashboards. But these initiatives did not yield the expected results as it did not have an integrated workflow and data pipeline. Data was in silos and this presented a significant challenge to the team at QUEST. The client required the following:
- A well structured analytics system capable of handling large volumes of data.
- Client preferred moving to an open source solution since they were a not for profit organisation.
- A future proof architecture that can accommodate the changes to existing applications and new applications.
- Develop the analytics platform to support multi company architecture in line with the client’s organisation structure.
Review & Challenges
As a first step, we had discussions with the business stakeholders to understand their requirements and translate them into data requirements. We then reviewed the existing implementation to identify bottlenecks and challenges. The following were the major problems identified by our architects.
- The existing implementation in PowerBI was rudimentary and consumed database views directly resulting in poor performance.
- The dashboards were not designed properly and the system was not designed to handle scale.
- Slow performance resulting from the non-modular approach.
- Loading large unstructured data logs coming from mobile devices was a major challenge.
Proposed System Architecture
Based on our analysis, we proposed a layered architecture which will be scalable, more efficient and easily maintainable. The next step was to identify and evaluate different open source tools. We zeroed in on Metabase and Superset and did proof of concept implementations. We finally chose Apache Superset as it was meeting their requirements and their data team was more comfortable with the tool.
- Introduced a WH and an ETL tool to aggregate the data from various sources and from applications.
- Researched and introduced open source tools without compromising performance and usability.
- Mapped data to different use cases and defined ETL’s that supports multiple organizations.
- We transferred data transformations from PowerBI to the SSIS layer.
Implementation & Results
- Implemented the solution using Talend as ETL, MySQL as Data warehouse and Superset as the BI layer.
- Integration of multiple data sources to the ETL layer.
- Custom business logic for data processing was integrated into the ETL layer.
- Creating reports and dashboards using Apache Superset.
- Testing & QA.
Implementation was done in the following four phases
How can we help you?
Get in touch with us to schedule a consultation
