INTRODUCTION The importance of our brain and cognitive abilities is something many of us don’t think twice about during our everyday activities. Nevertheless, none of our cognitive, normal functioning would be possible if it wasn’t for this complex structure we like to call our brain. …

Data mesh is currently a hot topic amongst data architects and as such generates a lot of interest all over the globe. We are brave enough to call it the next generation of enterprise data platform architecture. In this blog we will describe WHAT is data mesh,WHY should it be…

We’re building a recommendation engine that is based on customer usage, billing data and a set of product offers in Product Catalogue. The idea is that the user uploads the input tables from on premise Cloudera cluster, along with a JSON file specifying schemas, tables, table columns and corresponding filenames…

This post will briefly describe how to create Cloud Run service and showcase two different cases for both serverless options followed by performance test results. A previous blog post described the implementation of a data pipeline with simple lookup logic implemented using the Cloud Function. Google Cloud, in addition to…

Description of a data pipeline with simple lookup logic implemented in Google Cloud Platform. This blog post will provide a description of a data pipeline with simple lookup logic implemented in Google Cloud Platform. Lookups are usually used for data enrichment and data mappings. …

For those who develop applications that deal with data fetching and processing, you will know that there are a lot of challenges down that path. Nowadays, applications will sometimes have to deal with large-scale datasets, so every part of the application workflow needs to be well designed and tuned. …

Digital Transformation can be categorised in many ways and mean different things to different companies. At its core, it is a fundamental change to business processes, either by computerising existing manual processes, the modernisation of existing systems that are no longer fit for purpose, or the creation of new systems…

Find out which steps are needed to create simple Dataflow SQL job that reads events from Pub/Sub and combines them with data from BigQuery, as well as some additional examples of streaming processing. In previous blog post we had a short introduction to Dataflow SQL and Apache Beam in general…

Find about the technologies that are backing the Dataflow SQL and the comparison with typical Dataflow pipelines. In this sequence of blog posts, we will take a look at one of features available on Google Cloud Platform, Dataflow SQL. Recently released in Beta (March 3 rd, 2020), Dataflow SQL offers…