Presented by:

46cc05854b61780bfe4c3890eb3f33d5

Jeff Klukas

Jeff Klukas is a data scientist and systems engineer who loves code and automation. He helped discover the Higgs particle at the Large Hadron Collider and now works on data systems for Simple, a consumer technology company that's building banking for the way people think rather than the way banks work.

Logical decoding provides a comprehensive, real-time stream of the changes happening within a Postgres instance, which has tremendous potential for analytics use cases. This talk gives an overview of some common patterns for building pipelines to feed a data warehouse, then dives into the details of Simple's Postgres -> Kafka pipeline, how we use it to warehouse Postgres change history to Amazon Redshift, and how it feeds user-facing streaming applications. We'll finish by comparing Simple's approach to Confluent's bottledwater-pg and where Postgres' integration with the Kafka ecosystem could go in the future.

Date:
2017 March 29 06:40
Duration:
50 min
Room:
Liberty III
Conference:
PGConf US 2017
Language:
Track:
Use Cases
Difficulty:
Medium