Dataphos

Welcome to the official documentation of Dataphos. Dataphos is a collection of microservices designed to solve the common issues arising in the current world of Data Engineering. Although each component is designed and developed independently to support the specific needs of existing architectures, Dataphos is built to inevitably come together as a unified ingestion platform, guiding the journey of your data from on-premise systems into the cloud and beyond.

Started in 2020 by battle-hardened industry professionals, Dataphos was built as a solution to the common issues plaguing Event-Driven Architectures:

• How do you transfer data from an on-premise database as a collection of structured records without relying on CDC solutions to merely replicate the database row-by-row?
• How do you ensure that data consumers don’t break due to sudden changes in the underlying schema of the data?
• How do you ensure proper fail-saves exist while using streaming services? How do you replay messages?
• How can you quickly and efficiently build structured Data Lake architectures?

At its core, Dataphos is an accelerator, enabling your engineers to focus on delivering business value, instead of re-building the same patterns and solutions other businesses spent months or years struggling with.

This documentation will provide a general overview of the key Dataphos use-cases, allow you to quickly set up an environment to deploy individual Dataphos components in (or help you deploy them in already-existing ones with minimal hassle), and provide in-depth information on each individual component’s configuration.