optimus
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
APACHE-2.0 License
Bot releases are hidden (Show)
Published by kushsharma over 3 years ago
Changelog
6401fec feat: inherit job specifications based on sub folders (#10)
e08abe6 fix: adding nil check for job spec retry behaviour
16d79f1 refactor: support multiple subquestion/answer at once in plugins (#13)
Docker images
docker pull docker.io/odpf/optimus:0.0.2-alpha.2docker pull docker.io/odpf/optimus:v0.0.2-alpha.2-amd64
Published by kushsharma over 3 years ago
Changelog
95372d6 fix: check for nil project when adapting proto specification
a09aa90 fix: trimming py sufix in gcs store should not replace inline characters
ab07ea2 refactor: task/hook plugins server should have constructor (#11)
Docker images
docker pull docker.io/odpf/optimus:latestdocker pull docker.io/odpf/optimus:0.0.2-alpha.1docker pull docker.io/odpf/optimus:v0.0.2-alpha.1-amd64
Published by kushsharma over 3 years ago
Changelog
7a7662a feat(replay): implement replay dry-run feature (#6)
df8f8ab feat: add & use airflow 2 scheduler (#3)
54719f7 feat: allow job level configuration of retry count and delay (#9)
ec8e023 feat: have one binary that does it all (#8)
1111b1d fix(api): check for nil pointer while deserializing duration proto
042f44c fix: don't print config path if empty
96a5922 fix: handle_pod_overlap accepts a single pod instead of list in airflow2 lib (#7)
065d409 refactor: filter what secrets to be passed to plugins (#4)
df242ba refactor: using INSTANCE_ prefix instead of TASK_ in env injected inside plugin containers (#5)
Docker images
docker pull docker.io/odpf/optimus:latestdocker pull docker.io/odpf/optimus:v0.0.1-amd64
Published by kushsharma over 3 years ago
Changelog
43e2f1d fix: incorrect version passed during binary build
Docker images
docker pull docker.io/odpf/optimus:latestdocker pull docker.io/odpf/optimus:v0.0.1-rc.2-amd64
Published by kushsharma over 3 years ago
Changelog
3e12624 Merge branch 'airflow-integration' into 'master'
aaaa633 Merge branch 'bb-replace-loadmethod' into 'master'
bf82a0b Merge branch 'bq-view-dependency' into 'master'
17eca9e Merge branch 'bq2bq' into 'master'
3308d8e Merge branch 'cross-entity-status-endpoint' into 'master'
809c71b Merge branch 'datastore-management' into 'master'
fd15662 Merge branch 'datastore-management-bq-unit-test' into 'master'
c6cd75c Merge branch 'docs-update' into 'master'
662dded Merge branch 'docs-update' into 'master'
9e3e2c1 Merge branch 'documentation/guides' into 'master'
0b993d6 Merge branch 'external-sensor-truncate-fix' into 'master'
edb6f9d Merge branch 'feature/cross_tenant_dependency' into 'master'
d0b16a3 Merge branch 'feature/hooks_integration' into 'master'
f751316 Merge branch 'feature/metadata_publisher' into 'master'
47a846d Merge branch 'feature/multi_client_integration_for_a_project' into 'master'
3206bb1 Merge branch 'fixing-install-dependency' into 'master'
68fa044 Merge branch 'jinja-templates' into 'master'
8410871 Merge branch 'migration-from-v1' into 'master'
d2b931d Merge branch 'opctl-window-helper' into 'master'
5fbfb6f Merge branch 'predator-hook' into 'master'
c7a2070 Merge branch 'project-config' into 'master'
fc90886 Merge branch 'protos-from-odpf' into 'master'
f222ba6 Merge branch 'runtime-plugins' into 'master'
56a2e14 Merge branch 'store' into 'master'
4a34eef Merge branch 'tag-version-on-dependencies' into 'master'
568abba Merge branch 'task-changes' into 'master'
1b6d18a Optimus Supports creation of Sample JobSpec, Bootstrap a server
bcbaf15 Set theme jekyll-theme-minimal
0e0db1e [Kush/Shekhar] Added airflow scheduler, made server stream deployment ack for each job
396c29c [Kush/Shekhar] Optctl get job command to fetch job assets, init container for airflow pod operator
af543a1 [Kush/Shekhar] Spec compilation, storage and dependency resolution
7008d14 [Kush] Restructure how tasks are being registered
4faa8d7 [Kush] Restructure how tasks are being registered
41a51be [Kush] Store scaffolding
42619d8 [Shekhar|Kush] implement job priority resolver
06719b5 add cross tenant dependency feature, spec dump command, airflow DAG tests and gitignore grpc generated code
1088240 enable support for bq2bq task
c680944 feat: add documentation structure and user guides
b334b9f feat: implement metadata publisher
816a5e6 feat: implement namespace operability for projects
0006dfb feature!: added option to proxy job status request to scheduler
d3b3ae7 feature: adding support for runtime plugins - moved task and hook folders from ext to a different repository - checked in back proto compiled go files - made secret management of task/hook generic - after merging this, Optimus deployment needs to use kube secrets using "optimus-task-taskname" as a name
84d14ee feature: allows hooks to be added as pre/post executable
c68728b feature: opctl to initialize and update project config via optimus.yaml
81119ce feature: option to configure transformation window parameters from opctl create cmd
2274a6b feature: support for bigquery datastore
e222a51 feature: support for jinja template engine for job specifications
cd82083 feature: support for overriding asset compilation
d28553b feature: support for predator hook
bf87ae5 feature: using bq apis to find table dependencies in bq2bq
9b6cdae fix: bq clustering info parsing issue, string/int/primitive types[] as interface[]
e90c157 fix: bump proton version to v1.0.3
eac8399 fix: changes to support v1 migration
817ac79 fix: execution_date macro not correctly generating time
967d270 fix: failing integration test with missing args
796680e fix: increasing context deadline for deployment as BQ times out when used at peak time
fccb4d0 fix: install dependencies
6f1710f fix: integration test for postgres instance repository
bc48f27 fix: omit truncate_to config from external task sensor
e068c99 fix: remove unwanted workdir set in docker template for plugins (#2)
92f519b fix: secret repository integration test
4ae7f6e fix: tag bq2bq image version 0.0.1
4df2c81 fix: update format for job metadata urn
65cd331 fix: use where clause in job spec repo to filter projects
ed46774 hotfix: soft deleted job when inserted again should truncate all related instance records to conform foreign key
594eff8 refactor: added specific versions in makefile dependencies
28d8a9f refactor: allow unknown dependencies, filter expression in bq2bq
a573756 refactor: docs cleanup
9567bfa refactor: fix few golang-ci suggestions
3169094 refactor: moving internal protos to odpf
a0016a9 refactor: project repository will use gorm preload for loading secrets
ac74252 refactor: renames ExecUnit to Transformation
5fb5851 update README for buf debugging
Docker images
docker pull docker.io/odpf/optimus:latestdocker pull docker.io/odpf/optimus:v0.0.1-rc.1-amd64
