Introduction In today’s business, the faster you can get somewhere, the more beneficial it is. Whether it is speedy delivery or rapid decision making, having the right data at your disposal is a necessity. When moving data from on-premise systems to the cloud, you are restricted to one of two outcomes…

Introduction Data loading into Snowflake can be achieved through various methods, including Snowpipe and Snowpipe Streaming. Both of these services provide automated and continuous data ingestion capabilities. Snowpipe is a serverless data ingestion service offered by Snowflake, designed to simplify the process of loading data into Snowflake data warehouses. When new…

Introduction Great Expectations (GX) is an open-source Python library for data validation that allows you to define, manage, and automate the validation of data inside your data pipelines. In this blog post, we will showcase the power and flexibility of Great Expectations by walking through an end-to-end pipeline, using the tool…

In our previous post, Jason talked about the role of federated data governance in data mesh. In this post, I would like to tackle some of the questions you should have always have the answers to, before deciding how to start with data governance. Let’s dive right into the questions…

In this blog post, the focus will be on the process of deploying OpenMetadata on a Kubernetes cluster hosted on Azure. To find out more about OpenMetadata’s capabilities, check out this review by our data engineers, Kristina and Dominik. Keep in mind that the review was written prior to the…

In a world where data fuels innovation and growth, organizations face the challenge of effectively managing and sharing data to remain competitive. Traditional data governance approaches can be cumbersome and hinder business agility. Federated data governance offers a solution by decentralizing data management, incorporating data mesh principles, and focusing on…

Introduction Change Data Capture (CDC) is the process of extracting changes from a source database system and delivering them to a downstream system or process. CDC enables capturing the state of the database and tracking new changes made to it. This blog post presents a detailed overview of two Kafka Connect…

Introduction Performance and scalability of data warehousing are key concerns for businesses working with high volumes of streaming and historical analytics. Firebolt is a cloud-native solution that offers reliable, scalable performance at high speeds for all types of analytics operations. One of the key benefits of Firebolt is its lightning-fast speed…

BigQuery is just one service that Google Cloud Platform provides, which is a database service, but if your company is using the Cloud mainly for data related purposes like reporting, analytics or data science, you will probably be using BigQuery as your data platform. I would like to share my…