Usage

This page contains all of the information required to further configure the Publisher and ensure proper communication between it and your external resources.

In order to enable Publisher to communicate with the service of your choice, you are required to create and deploy additional secrets in your Kubernetes environment, under the same namespace as Publisher. Google PubSub, for instance, requires the Service Account Key used to connect to your Cloud environment. Kafka, NATS and Pulsar require TLS secrets.

PubSub

Kafka

NATS

Pulsar

When deployed, the Publisher Worker will automatically look for these secrets at the time of running the individual jobs.

Once you are logged into the WebUI, you can add your source, destination and instance YAML files.

• Source defines the source from which Publisher will fetch the data.
• Destination defines to where Publisher will deliver the data (the message broker).
• Instance connects the Source and Destination, and defines the actual publishing job. Both the source and destination may be used in multiple different publishing jobs.

The Source configuration YAML file is used to define the details regarding the source database that the user wants to use as a source of data.

sourceName: publisher-postgres-source
sourceType: Postgres
host: <LoadBalancer IP address>
port: 5432
databaseName: invoices
username: demo_user
password: demo_password

The Destination configuration YAML file is used to define the details regarding the destination service that the user wants to use as a final destination of data. Every destination includes parameters from the Destination tab. Each destination type has specific parameters that have to be added in the YAML file. These parameters are specified in the other tabs.

destinationName: publisher-pubsub-destination
destinationType: PubSub
parameters:
  ProjectID: <your project id>
  TopicID: <your topic>

The Instance configuration YAML file is used to define the job specifics. This is the central type of configuration and its creation and validation depend on the existing active source and destination configurations. The Instance configuration YAML file is fairly minimalistic in approach, but allows enough flexibility to fine-tune the requirements of your publishing job.

Here, we define the query the data will be fetched with, how it will be formatted, serialized, encrypted and, finally, published.

For ease of understanding, we will split the file into three distinct sections:

• The Instance portion, defining how the source and destination should connect.
• The Fetcher portion, defining how the data should be pulled from the source.
• The Business object portion, defining how the data will be transformed directly prior to publishing.

publisherName: publisher-demo
sourceName: publisher-postgres-source
destinationName: publisher-pubsub-destination
serializationType: Avro
encryptionEnabled: false
businessObject:
  description: "Demo Publisher - invoices for client"
  objectGroup: "invoices-client"
  additionalMetadata:
    organization: Syntio
  definition:
    - client_info:
        - client_id
        - client_username
        - client_location
    - invoice_info:
        - invoice_id
        - creation_date
        - due_date
        - fully_paid_date
    - invoice_items:
        - invoice_item_id
        - quantity
        - total_cost
  groupElements:
    - client_id
    - invoice_id
  arrayElements:
    - invoice_items
  keyElements:
    - client_id
    - invoice_id
  idElements:
    - client_id
  query: | 
    SELECT 
      invoice_id, 
      client_id, 
      client_username, 
      client_location, 
      creation_date, 
      due_date, 
      fully_paid_date, 
      invoice_item_id, 
      quantity, 
      total_cost, 
      billing_item_id
    FROM demo_invoices
    WHERE 
      creation_date >= to_timestamp({{ .FetchFrom }},'YYYY-MM-DD HH24:MI:SS') 
      AND 
      creation_date < to_timestamp({{ .FetchTo }}, 'YYYY-MM-DD HH24:MI:SS');
fetcherConfig:
  fetchingThreadsNO: 3
  queryIncrementType: HOUR
  queryIncrementValue: 12
  initialFetchValue: 2020-01-01 00:20:00.000
  useNativeDriver: true

Below is the screen you will be greeted with after first logging into the Publisher Web UI.

Publisher runs - statistical information about the execution of each instance.

Publisher Instances - overview of Publisher’s instance configurations, providing the ability to start and stop instances as required.

Publisher Lineage - data origin, what happens to it, and where it moves over time.

Publisher queries - testing Publisher instance configuration with verbose error messages. You can view the efficiency of query as well as a sample of the data obtained.

WEB CLI - update and process YAML files to update sources, destinations, configuration. First, add source YAML, then destination and then instance. You can add, update and delete your configuration.