Pulumi

There are three possible sources of resource configuration values: user configuration in the active stack configuration file, retrieved data from existing resources, and default system-level configuration from the application code.

User configuration will always take precedence over other configuration sources. If there is no special user configuration for a parameter, the retrieved value from the resource’s previous configuration will be used. If there wasn’t any data retrieved for the resource (as it is being created for the first time), the default system-level configuration value will be used instead. The default values for parameters are listed in the appropriate section of the configuration options.

If the configuration references an existing cloud resource, the program will retrieve its data from the cloud provider and import the resource into the active stack instead of creating a new one. If the user configuration values specify any additional parameters that differ from the resource configuration while it has not yet been imported into the stack, the deployment will fail. To modify an existing resource’s configuration, import it into the stack first and then redeploy the infrastructure with the desired changes.

Note: Implicit import of an AKS cluster is currently not supported. To use an existing AKS cluster in your infrastructure, set the AKS cluster’s import configuration option to true.

⚠️ WARNING ⚠️

Imported resources will NOT be retained by default when the infrastructure is destroyed. If you want to retain a resource when the infrastructure is destroyed, you need to explicitly set its retain flag to true in the active stack’s configuration file. Retained resources will not be deleted from the backing cloud provider, but will be removed from the Pulumi state on a pulumi destroy. Azure resource groups and GCP projects are set to be retained by default and can be deleted manually. Be careful if you choose not to retain them, as destroying them will remove ALL children resources, even the ones created externally. It is recommended to modify these options only if you are using a dedicated empty project/resource group.

Below is the shared configuration used between all Persistor types.

The namespace and images options at the top-level of the Helm chart configurations are set by default and do not need to be manually configured.

Cloud-specific variables should not be manually configured. Depending on the configured cloud provider, service accounts with appropriate roles are automatically created and their credentials are used to populate these variables.

The variables listed here are required configuration options by their respective Pulumi providers. Your entire infrastructure should reside on a single cloud platform. Deployment across multiple cloud platforms is currently not fully supported.

The stack configuration cluster object is utilized to configure the Kubernetes cluster necessary to deploy the Helm charts that comprise Dataphos products.

The stack configuration brokers object is used to set up the key references to be used by the dataphos components to connect to one or more brokers deemed to be part of the overall platform infrastructure.

Product configs directly reference brokers by their BROKER_ID listed in the broker config. The same applies to TOPIC_ID and SUB_ID – the keys of those objects are the actual names of the topics and subscriptions used.

The Azure storage account type. Valid values: [Storage, StorageV2, BlobStorage, BlockBlobStorage, FileStorage]. The default and recommended value is BlockBlobStorage.

The stack configuration storage object is used to set up the key references to be used by the dataphos components to connect to one or more storage destinations deemed as part of the overall platform infrastructure.

Product configs directly reference storage components by their STORAGE_ID listed in the storage config. The same applies to BUCKET_ID – the keys of those objects are the actual names of the buckets used.