APACHE-2.0 License
IMPORTANT: This README file is obsolete and no longer receives updates. For the most current and reliable information, refer to the new documentation at Google Cloud Cortex Framework Overview.
This generator reads the specified CDC table names, keys of the specified tables and merge table name and generates a merge query and a python script for each table for Cloud Composer or Apache Airflow. We recommend reading the instructions in the parent module, the Cortex Data Foundation.
The following steps must be completed before running this generator.
recordstamp
and operation_flag
columns. Adjust the filestemplate_dag
and template_sql
if the fields have different names.config/config.json
file with the values required for SAP deployment configurationConfiguration file for this repository is
config/config.json
.
The cloudbuild.cdc.yaml
for this generator requires the following
parameters:
_GCS_BUCKET
: GCS bucket created for logs that this generator writes to (logs bucket).config/config.json
as described in the Data Foundation README for SAP deploymentconfig/config.json
).settings.yaml
to add / delete the required tables and runsets.yaml
to add / delete the required SAP datasetsThe generator can be run from the Cloud Console using the gcloud builds submit ...
command or
by configuring a Cloud Builds trigger that runs automatically upon push to a Cloud Source
Repository branch
gs://<targetBucket>/dags
gs://<targetBucket>/data/bq_data_replication
targetBucket
- is a GCS bucket created for holding the DAG python scripts and SQL scripts
(targetBucket
in config/config.json
).