Data Warehouse and BI
For Data Warehouse and BI we use three types of solutions depending on the DWH size and the data platform the customer uses:
- Small DWH – Uses the same database brand the customer already uses so the application is fully compatible with the database that is already in use.
The databases can be: Oracle, SQL Server, PostgreSQL and MySQL.
The first three listed above are more compatible as they support parallel query execution and analytic functions, MySQL only supports Version 8 and above.
These databases can be hosted on an on-premise machine or database cloud service (RDS, Cloud SQL). These databases can be suited to up to 1TB.
- Medium Data Warehouse – Uses analytic columnar databases such as Exasol, Vertica and Redshift. These databases support scale out for extending CPU, RAM and Disk resources and are best suited for analytic queries. These are best suited for databases from 1 TB up to 10’s TBs of data.
- Huge Data Warehouse – Uses Big Data Platforms Such as Hadoop, Google BigQuery, Amazon Athena or Presto over Cloud Storage. These data platforms are suited for 100’S TB storage. They scale out of the box and can give great response time for huge amounts of data and queries running on huge amounts of data.
Every type of solution will require an ETL process which extracts the data from the source system, transforms and loads it to DWH.
It is important to define the correct DWH data model so queries will not make many joins and return the results quickly. In order to achieve this you need a good Data Architect.
Our Data Engineers use Python (small/medium DWH) and Spark (Huge Data Warehouse) to extract the data from source systems or messages queue systems like Kafka and RabbitMQ transform and load it into the DWH platform.
We use a data lake so all customer dimensions, fields and statistics are stored in one or more tables for easy and fast query analysis.
For vitalization we use Tableau, Qlik Sense and Power BI. Our experts build the optimized data layer above the DWH platform in order to best access the data to the data analysts and business stakeholders.
Contact Us today to hear more about our solutions. We will help build the best DWH to suit your business needs at the lowest cost and the best performance.