optimus

Changelog

• 7645d51 feat: progress bar for long running operations in cli (#130)
• 5f34b27 feat: support for specifying execution resources using metadata (#120)
• b547d74 refactor: protobuf lint issues of Runtime/Plugins/Cluster RPCs (#106)

Changelog

• 4e3ec40 feat: add extension supports
• d84d85b feat: allow manually triggered job using empty scheduler interval (#122)
• 448bfd7 feat: endpoint/rpc for fetching task details of a job (#107)
• cc106f3 feat: enhance replay & backup to support multiple namespaces jobs (#105)
• 4e7c152 feat: fetch info for requested backup runs (#112)
• 9dc68d8 fix: dependency resolution during duplicate destinations of job (#118)
• 330889d fix: deployment fails even if broken job belong to different namespace (#110)
• 1414cd8 fix: priority resolution should not fail because of missing jobs (#117)
• ef394fc refactor: project and namespace configuration to support multiple namespaces (#103)

Changelog

a908f4e feat: add backup dry run (#77)
4622182 feat: add backup feature (#88)
86c3e9b feat: add replay list sub command (#67)
56cbf97 feat: add replay syncer and get replay status (#48)
cf8c7bb feat: custom scheduler for running jobs on demand (#65)
94d5619 feat: pass authorization http header from cli if available (#90)
6da97d0 feat: support alert for the latest binary release (#46)
7150c57 feat: support static inter project dependencies (#93)
d1001c8 feat: telemetry for api traces, database calls and runtime stats (#102)
1ac1718 fix: cross entity sensor in airflow not failing if condition not met (#99)
1ac23a8 fix: cross entity sensor in airflow should use base job windows (#84)
f853468 fix: deploy grpc stream should be concurrent safe (#86)
407057b fix: instances should reuse execution time across a complete run (#96)
5d0fdae fix: skip dependency checks for non DR mods (#54)
ca5771e fix: skip using a prefixed bucket if gcs path is empty
3db9fa5 fix: validation for empty project config during deploy
6b760d4 refactor: airflow scheduler should upload jobs in parallel
587ace4 refactor: inject logger using salt in the required packages (#55)
089e4a6 refactor: optimise ListJob operation for airflow (#95)
2d3c693 refactor: rename dryrun replay methods (#78)
f98a15a refactor: use gorm v2 for postgres storage (#101)
25f4492 refactor: use updated buf lib for inmemory proto generation (#76)

Docker images

• docker pull docker.io/odpf/optimus:0.0.3
• docker pull docker.io/odpf/optimus:v0.0.3-amd64

Changelog

872e8a1 fix: plugin discovery based on plugin type

Docker images

• docker pull docker.io/odpf/optimus:0.0.3-alpha.2
• docker pull docker.io/odpf/optimus:v0.0.3-alpha.2-amd64

Changelog

7aca79c feat: breaking plugins to mods for easier composition (#47)
4bbe355 feat: google sheets external table support for bigquery datastore (#45)
d5c9b08 feat: validate replay request before execution (#35)
bc9912a refactor: pass raw resource with spec yaml parsing errors (#43)
506dff7 refactor: replay errors, replay failure test cases, airflow and replay manager (#37)

Docker images

• docker pull docker.io/odpf/optimus:0.0.3-alpha.1
• docker pull docker.io/odpf/optimus:v0.0.3-alpha.1-amd64

Changelog

d4f27ee chore: update cobra version to v1.2.1 (#29)
987d7d9 feat: support slack based alerts on failure and sla miss (#25)
58ebfc3 fix: return error when fail to parse yaml spec for resource

Docker images

• docker pull docker.io/odpf/optimus:0.0.2
• docker pull docker.io/odpf/optimus:v0.0.2-amd64

Changelog

4cdee6a feat(cli): ask for directories from user before creating new specification (#15)
d11711f feat(cli): support default values inferred from job name (#21)
57a3c9f feat: implement replay without backup and validation (#12)
a577ede fix: cross entity sensor should use updated job metadata api (#23)
ea6cf6a fix: using m instead of h while requesting window dates in airflow sensor

Docker images

• docker pull docker.io/odpf/optimus:0.0.2-alpha.3
• docker pull docker.io/odpf/optimus:v0.0.2-alpha.3-amd64

Changelog

6401fec feat: inherit job specifications based on sub folders (#10)
e08abe6 fix: adding nil check for job spec retry behaviour
16d79f1 refactor: support multiple subquestion/answer at once in plugins (#13)

Docker images

• docker pull docker.io/odpf/optimus:0.0.2-alpha.2
• docker pull docker.io/odpf/optimus:v0.0.2-alpha.2-amd64

Changelog

95372d6 fix: check for nil project when adapting proto specification
a09aa90 fix: trimming py sufix in gcs store should not replace inline characters
ab07ea2 refactor: task/hook plugins server should have constructor (#11)

Docker images

• docker pull docker.io/odpf/optimus:latest
• docker pull docker.io/odpf/optimus:0.0.2-alpha.1
• docker pull docker.io/odpf/optimus:v0.0.2-alpha.1-amd64

Changelog

7a7662a feat(replay): implement replay dry-run feature (#6)
df8f8ab feat: add & use airflow 2 scheduler (#3)
54719f7 feat: allow job level configuration of retry count and delay (#9)
ec8e023 feat: have one binary that does it all (#8)
1111b1d fix(api): check for nil pointer while deserializing duration proto
042f44c fix: don't print config path if empty
96a5922 fix: handle_pod_overlap accepts a single pod instead of list in airflow2 lib (#7)
065d409 refactor: filter what secrets to be passed to plugins (#4)
df242ba refactor: using INSTANCE_ prefix instead of TASK_ in env injected inside plugin containers (#5)

Docker images

• docker pull docker.io/odpf/optimus:latest
• docker pull docker.io/odpf/optimus:v0.0.1-amd64

Changelog

43e2f1d fix: incorrect version passed during binary build

Docker images

• docker pull docker.io/odpf/optimus:latest
• docker pull docker.io/odpf/optimus:v0.0.1-rc.2-amd64

Changelog

3e12624 Merge branch 'airflow-integration' into 'master'
aaaa633 Merge branch 'bb-replace-loadmethod' into 'master'
bf82a0b Merge branch 'bq-view-dependency' into 'master'
17eca9e Merge branch 'bq2bq' into 'master'
3308d8e Merge branch 'cross-entity-status-endpoint' into 'master'
809c71b Merge branch 'datastore-management' into 'master'
fd15662 Merge branch 'datastore-management-bq-unit-test' into 'master'
c6cd75c Merge branch 'docs-update' into 'master'
662dded Merge branch 'docs-update' into 'master'
9e3e2c1 Merge branch 'documentation/guides' into 'master'
0b993d6 Merge branch 'external-sensor-truncate-fix' into 'master'
edb6f9d Merge branch 'feature/cross_tenant_dependency' into 'master'
d0b16a3 Merge branch 'feature/hooks_integration' into 'master'
f751316 Merge branch 'feature/metadata_publisher' into 'master'
47a846d Merge branch 'feature/multi_client_integration_for_a_project' into 'master'
3206bb1 Merge branch 'fixing-install-dependency' into 'master'
68fa044 Merge branch 'jinja-templates' into 'master'
8410871 Merge branch 'migration-from-v1' into 'master'
d2b931d Merge branch 'opctl-window-helper' into 'master'
5fbfb6f Merge branch 'predator-hook' into 'master'
c7a2070 Merge branch 'project-config' into 'master'
fc90886 Merge branch 'protos-from-odpf' into 'master'
f222ba6 Merge branch 'runtime-plugins' into 'master'
56a2e14 Merge branch 'store' into 'master'
4a34eef Merge branch 'tag-version-on-dependencies' into 'master'
568abba Merge branch 'task-changes' into 'master'
1b6d18a Optimus Supports creation of Sample JobSpec, Bootstrap a server
bcbaf15 Set theme jekyll-theme-minimal
0e0db1e [Kush/Shekhar] Added airflow scheduler, made server stream deployment ack for each job
396c29c [Kush/Shekhar] Optctl get job command to fetch job assets, init container for airflow pod operator
af543a1 [Kush/Shekhar] Spec compilation, storage and dependency resolution
7008d14 [Kush] Restructure how tasks are being registered
4faa8d7 [Kush] Restructure how tasks are being registered
41a51be [Kush] Store scaffolding
42619d8 [Shekhar|Kush] implement job priority resolver
06719b5 add cross tenant dependency feature, spec dump command, airflow DAG tests and gitignore grpc generated code
1088240 enable support for bq2bq task
c680944 feat: add documentation structure and user guides
b334b9f feat: implement metadata publisher
816a5e6 feat: implement namespace operability for projects
0006dfb feature!: added option to proxy job status request to scheduler
d3b3ae7 feature: adding support for runtime plugins - moved task and hook folders from ext to a different repository - checked in back proto compiled go files - made secret management of task/hook generic - after merging this, Optimus deployment needs to use kube secrets using "optimus-task-taskname" as a name
84d14ee feature: allows hooks to be added as pre/post executable
c68728b feature: opctl to initialize and update project config via optimus.yaml
81119ce feature: option to configure transformation window parameters from opctl create cmd
2274a6b feature: support for bigquery datastore
e222a51 feature: support for jinja template engine for job specifications
cd82083 feature: support for overriding asset compilation
d28553b feature: support for predator hook
bf87ae5 feature: using bq apis to find table dependencies in bq2bq
9b6cdae fix: bq clustering info parsing issue, string/int/primitive types[] as interface[]
e90c157 fix: bump proton version to v1.0.3
eac8399 fix: changes to support v1 migration
817ac79 fix: execution_date macro not correctly generating time
967d270 fix: failing integration test with missing args
796680e fix: increasing context deadline for deployment as BQ times out when used at peak time
fccb4d0 fix: install dependencies
6f1710f fix: integration test for postgres instance repository
bc48f27 fix: omit truncate_to config from external task sensor
e068c99 fix: remove unwanted workdir set in docker template for plugins (#2)
92f519b fix: secret repository integration test
4ae7f6e fix: tag bq2bq image version 0.0.1
4df2c81 fix: update format for job metadata urn
65cd331 fix: use where clause in job spec repo to filter projects
ed46774 hotfix: soft deleted job when inserted again should truncate all related instance records to conform foreign key
594eff8 refactor: added specific versions in makefile dependencies
28d8a9f refactor: allow unknown dependencies, filter expression in bq2bq
a573756 refactor: docs cleanup
9567bfa refactor: fix few golang-ci suggestions
3169094 refactor: moving internal protos to odpf
a0016a9 refactor: project repository will use gorm preload for loading secrets
ac74252 refactor: renames ExecUnit to Transformation
5fb5851 update README for buf debugging

Docker images

• docker pull docker.io/odpf/optimus:latest
• docker pull docker.io/odpf/optimus:v0.0.1-rc.1-amd64