Foundation: The Issue with Insights
To compete in today's fast paced economy, organizations must make smart decisions quickly based on up-to-date, accurate, & relevant information. This process of obtaining actionable insights from data is known as Business Intelligence or Analytics. There are many different approaches to gaining insight from data depending on the organizations size and it's needs. Smaller companies with a simple, single source of data may only need a data visualization tool such as Microsoft Excel, Power BI, Tableau, or Qlik to begin gaining insight. However, the amount of data in the world is increasing exponentially, and as the amount and complexity of data increases, it becomes difficult or even impossible to find insights. It then becomes necessary to properly optimize, clean, and format this data in a process known as ETL (Extract, Transform, and Load). The most robust approaches implement what's known as a Data Estate for analytics and include multiple methods for storing this data such as Data Lakes, Data Hubs, Data Warehouses, & Data Marts. Only then, after these complex data sets are consolidated and simplified at varying degrees, you can begin to gain insight into this data.
Unfortunately, the traditional approach to building and maintaining this infrastructure requires teams of people to write and maintain mountains of several types of code. As you can imagine, this can be very costly, time-consuming, and requires a diverse set of hard-to-find skills.