Designed by Agile Lab, Witboost is a versatile platform that addresses a wide range of sophisticated data engineering challenges. It enables businesses to discover, enhance, and productize their data, fostering the creation of automated data platforms that adhere to the highest standards of data governance. Want to know more about Witboost? Check it out here or contact us!
This repository is a guide to our Starter Kit meant to showcase Witboost's integration capabilities and provide a "batteries-included" product.
A Tech Adapter (formerly called Specific Provisioner) is a microservice which is in charge of deploying components that use a specific technology. When the deployment of a system (e.g. a Data Product) is triggered, the platform generates its descriptor and orchestrates the deployment of every component contained in the system. For every such component the platform knows which Tech Adapter is responsible for its deployment, and can thus send a provisioning request with the descriptor to it so that the Tech Adapter can perform whatever operation is required to fulfill this request and report back the outcome to the platform.
Tech Adapters were previously called Specific Provisioners, so you might still find this name in most of the repositories while we perform the name transition. So whenever you encounter Specific Provisioner, just read it as Tech Adapter.
You can learn more about how the Tech Adapters fit in the broader picture here.
We provide two main kinds of projects:
- Tech Adapters: microservice implementations for a specific technology that you can customize to suit your needs
- Scaffolds: base projects that you can start from if you want to implement a tech adapter yourself
| Tech | Kind | Project | Scope | Supported components | Notes |
|---|---|---|---|---|---|
 |
Tech Adapter | Airbyte Adapter | ELT - Airbyte | Workload | |
 |
Tech Adapter | Airflow Adapter | Scheduling - Airflow/MWAA | Workload | |
 |
Tech Adapter | Azure ADLS Storage Area Adapter | Object Storage - Azure Data Lake Storage | Storage Area | Deployable with Azure ADLS Umbrella Chart |
 |
Tech Adapter | CDP Impala Adapter | SQL Query Engine - CDP Impala | Output Port | |
 |
Tech Adapter | CDP S3 Adapter | Object Storage - CDP S3 | Output Port | |
 |
Tech Adapter | CDP Spark Adapter | Data Processing - CDP Spark | Workload | |
 |
Tech Adapter | CDP HDFS Adapter | Distributed File System - CDP HDFS | Output Port, Storage Area | |
 |
Tech Adapter | Databricks Adapter | Data Processing - Databricks Spark | Workload, Output Port | |
 |
Tech Adapter | Hasura Adapter | GraphQL - Hasura | Output Port | Needs the Hasura Authentication Webhook and Role Mapper |
 |
Tech Adapter | Snowflake Adapter | DWH - Snowflake | Output Port, Storage Area | |
 |
Tech Adapter | Azure Data Factory Adapter | ETL - Azure Data Factory | Workload | |
 |
Scaffold | Java Scaffold | Generic - Java | NA | Uses the Java Tech Adapter Framework library. |
 |
Scaffold | Python Scaffold | Generic - Python | NA | |
 |
Scaffold | Terraform Scaffold | Generic - Terraform | NA |
A Template is a tool that helps create components inside Witboost under a specific Data Landscape (e.g. Data Mesh). Templates help establish a standard across the organization. This standard leads to easier understanding, management and maintenance of components. Templates provide a predefined structure so that developers don't have to start from scratch each time, which leads to faster development and allows them to focus on other aspects, such as testing and business logic.
For more information, please refer to the official documentation.
| Tech | Component | Project | Scope | Tech Adapter | Notes |
|---|---|---|---|---|---|
 |
Data Product | Data Product | NA | No Tech Adapter needed | |
|
Storage Area | Azure ADLS Storage Area | Data Lake Storage - Azure | Azure ADLS Storage Area Adapter | |
|
Output Port | CDP CDW Impala Output Port | SQL Query Engine - CDP CDW Impala | CDP Impala Adapter | |
|
Output Port | CDP DL S3 Output Port | Object Storage - CDP DL S3 | CDP S3 Adapter | |
|
Storage Area | CDP HDFS Storage Area | Distributed File System - CDP HDFS | CDP HDFS Adapter | |
|
Output Port | CDP HDFS Output Port | Distributed File System - CDP HDFS | CDP HDFS Adapter | |
|
Output Port | Hasura Output Port | GraphQL - Hasura | Hasura Adapter | |
|
Output Port | Snowflake Output Port | DWH - Snowflake | Snowflake Adapter | |
|
Storage Area | Snowflake Storage Area | DWH - Snowflake | Snowflake Adapter | |
|
Workload | Snowflake SQL Workload | Data processing - Snowflake | No Tech Adapter needed | It's triggered by an orchestrator through the Airflow Workload Template |
|
Workload | Airbyte Workload | ELT - Airbyte | Airbyte Adapter | |
|
Workload | CDP CDE Spark Workload | Data Processing - CDP CDE Spark | CDP Spark Adapter | |
 |
Workload | DBT Workload | Data processing - DBT | No Tech Adapter needed | |
|
Workload | Airflow Workload | Scheduling - Airflow/MWAA | Airflow Adapter | |
|
Workload | Azure Data Factory Workload | ETL - Azure Data Factory | Azure Data Factory Adapter |
Looking to build your own template? Check out the Templates Gallery, which contains howtos and practical examples to kickstart the process.
A Data Catalog Plugin is an extension point for Witboost that allows publishing entities on an external, pluggable Data Catalog. It is invoked at the end of the provisioning flow and receives the whole information about the entity descriptor, provisioning info, etc.
You can learn more about how Data Catalog plugins fit in the broader picture here.
| Tech | Kind | Project | Scope | Notes |
|---|---|---|---|---|
 |
Data Catalog Plugin | Collibra Data Catalog Plugin | Data Catalog - Collibra |
In this section you can find a list of possible integrations. They are not as production ready as the ones above, but are in any way a good starting point to address specific use cases and to understand Witboost capabilities.
| Tech | Kind | Project | Scope | Supported components | Notes |
|---|---|---|---|---|---|
 |
Tech Adapter | Ice Panel | C4 Architecture Diagram | Data Product | Needs an IcePanel license |
 |
Tech Adapter | Tonic.ai | Synthetic Data Generation | Output Port | Needs a Tonic.ai license |
 |
Tech Adapter | DCAT - OWL - RDF | Data Catalog | Output Port | Needs an RDF Triple Store endpoint |
 |
Tech Adapter | GoodData | Analytics | Output Port | Needs a GoodData license |
The Practice Shaper is the main and most impactful Witboost setting that models entities (domains, systems, components, templates) as nodes of a fully-configurable property graph.
This enables data-oriented organizations to shape Witboost based on their unique use cases, structure, and needs.
Thanks to the Practice Shaper, a company can approach any project scenario in data (Data Landscape), such as Data Mesh, Business Intelligence, Machine Learning and others, by defining which practices are enabled and regulated, with the possibility to define technological and methodological guardrails.
Refer to the Witboost documentation to learn more about Practice Shaper and Data Landscapes.
The Practice Shaper Presets repository provides some ready-to-import Data Landscapes, allowing organizations to quickly set up and customize their witboost environment to suit specific business needs.
This project is available under the Apache License, Version 2.0; see LICENSE for full details.
Agile Lab creates value for its Clients in data-intensive environments through customizable solutions to establish performance driven processes, sustainable architectures, and automated platforms driven by data governance best practices.
Since 2014 we have implemented 100+ successful Elite Data Engineering initiatives and used that experience to create Witboost: a technology-agnostic, modular platform, that empowers modern enterprises to discover, elevate and productize their data both in traditional environments and on fully compliant Data mesh architectures.
Contact us or follow us on: