numaflow

Major Changes

• Fixed a bug related to vertex processing rate calculation.
• Enhanced pipeline spec validation.
• UI improvements to show colored logs and the ability to toggle the order.

Check out our v0.10 release blog.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.10.0...v0.10.1

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.10.1/install.yaml

Major Changes

• Side Inputs support to be able to broadcast slow updated data or configuration automatically without needing to load it during each message processing; see the detail in the proposal.
• Support join vertex and cycle (for retrying) topologies in the pipeline spec for more complex use cases.
• SDK changes for easier user-defined functions implementation.
• Deprecated cooldownSeconds for auto-scaling; use scaleUpCooldownSeconds and scaleDownCooldownSeconds instead.
• Added Vertex template to pipeline spec for common specs (such as metadata, resources, etc.) of the vertices in the same pipeline.
• UI improvements.

🚨 Breaking Change 🚨

SDKs updated to version 0.5.0 with significant changes to the signature and structure. The servers for each operation have been separated, and all types have been moved to specific packages. For example, the types required for writing map UDFs are now available in the mapper package, and similar changes have been made for other operations. Please update your SDKs accordingly.

⚠️ Note: Version 0.10.0 will only work for SDKs with a version greater than or equal to 0.5.0.

Check out our v0.10 release blog.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.9.0...v0.10.0

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.10.0/install.yaml

Major Changes

• Fixed JetStream K/V watch error when there's no entry existing

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.9.1...v0.9.2

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.9.2/install.yaml

Major Features

• Fixed some UI bugs

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.9.0...v0.9.1

Contributors

@yhl25
@veds-g

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.9.1/install.yaml

Major Changes

• Support for multi-partitioned buffers for higher throughput.
• Support to run the map vertex in streaming mode with batch size set to 1. doc
• Fix intermittent failure from Kafka to get consumer offsets. issue
• Fix pods not rescheduled after eviction. issue
• Capability to increase max message size upto 64mb.
• Introduce an optional Validating Admission Webhook to prevent operations such as updating InterStepBufferService immutable fields.
• UI improvements
  - More accurate processing rates in the UI using pod metrics.
  - Support for multi-partitioned edges.
  - better performance.
• Fixed the message size limitation issue (only messages with size < 32K were allowed).

Breaking Changes and Upgrading Guide

• ISBSVC with Redis 6.0.6 is not supported any more, using 7.0.11. Anyone who is using old version, please delete the old object, update the version and recreate.

• The buffer and bucket definitions are revised, therefore we updated the buffer and bucket implementation, which requires recreating the pipeline during upgrade (1. Delete the pipelines; 2. Upgrade; 3. Recreate the pipelines). Meanwhile, we removed the following properties:

  1. Removed spec.edges[].parallelism, use spec.vertices[].partitions.
  2. Removed spec.edges[].limits.*, use spec.vertices[].limits.*, * includes bufferMaxLength and bufferUsageLimit.

• If upgrading from v0.8.0, please delete and recreate the pipeline.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.8.0...v0.9.0

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.9.0-rc2/install.yaml

Major Changes

• Support for multi-partitioned buffers for higher throughput.
• Support to run the map vertex in streaming mode with batch size set to 1. doc
• Fix intermittent failure from Kafka to get consumer offsets. issue
• Fix pods not rescheduled after eviction. issue
• Capability to increase max message size upto 64mb.
• Introduce an optional Validating Admission Webhook to prevent operations such as updating InterStepBufferService immutable fields.
• UI improvements
  - More accurate processing rates in the UI using pod metrics.
  - Support for multi-partitioned edges.
  - better performance.
• Fixed the message size limitation issue (only messages with size < 32K were allowed).

Breaking Changes and Upgrading Guide

• ISBSVC with Redis 6.0.6 is not supported any more, using 7.0.11. Anyone who is using old version, please delete the old object, update the version and recreate.

• The buffer and bucket definitions are revised, therefore we updated the buffer and bucket implementation, which requires recreating the pipeline during upgrade (1. Delete the pipelines; 2. Upgrade; 3. Recreate the pipelines). Meanwhile, we removed the following properties:

  1. Removed spec.edges[].parallelism, use spec.vertices[].partitions.
  2. Removed spec.edges[].limits.*, use spec.vertices[].limits.*, * includes bufferMaxLength and bufferUsageLimit.

• If upgrading from v0.8.0, please delete and recreate the pipeline.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.8.0...v0.9.0-rc2

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.9.0-rc2/install.yaml

Major Changes

• Support for multi-partitioned buffers for higher throughput.
• Support to run the map vertex in streaming mode with batch size set to 1. doc
• Fix intermittent failure from Kafka to get consumer offsets. issue
• Fix pods not rescheduled after eviction. issue
• capability to increase max message size upto 64mb.
• UI improvements
  - More accurate processing rates in the UI using pod metrics.
  - Support for multi-partitioned edges.
  - better performance.
• Fixed the message size limitation issue (only messages with size < 32K were allowed).

Breaking Changes and Upgrading Guide

• ISBSVC with Redis 6.0.6 is not supported any more, using 7.0.11. Anyone who is using old version, please delete the old object, update the version and recreate.

• The buffer and bucket definitions are revised, therefore we updated the buffer and bucket implementation, which requires recreating the pipeline during upgrade (1. Delete the pipelines; 2. Upgrade; 3. Recreate the pipelines). Meanwhile, we removed the following properties:

  1. Removed spec.edges[].parallelism, use spec.vertices[].partitions.
  2. Removed spec.edges[].limits.*, use spec.vertices[].limits.*, * includes bufferMaxLength and bufferUsageLimit.

• If upgrading from v0.8.0, please delete and recreate the pipeline.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.8.0...v0.9.0-rc1

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.9.0-rc1/install.yaml

Major Changes

• Ability to configure different base href for Numaflow UI server.
• Restart the Kafka source pod when there's a server side error.
• Fixed the message size limitation issue (only messages with size < 32K were allowed).

Breaking Changes and Upgrading Guide

• ISBSVC with Redis 6.0.6 is not supported any more, using 7.0.11. Anyone who is using old version, please delete the old object, update the version and recreate.

• The buffer and bucket definitions are revised, therefore we updated the buffer and bucket implementation, which requires to recreate the pipeline during upgrade (1. Delete the pipelines; 2. Upgrade; 3. Recreate the pipelines). Meanwhile, we deprecated following properties:

  1. Deprecated spec.edges[].parallelism, use spec.vertices[].partitions.
  2. Deprecated spec.edges[].limits.*, use spec.vertices[].limits.*, * includes bufferMaxLength and bufferUsageLimit.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.8.0...v0.8.1

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.1/install.yaml

Check out our release blog.

Major Changes

• Ability to assign event time, as well as filtering and transformations at the source vertex using source transformers
• Idle watermark propagation now does watermark progression in idle edges reducing close-of-book latency
• Message delivery count is now being tracked and is exposed to user-defined code
• Support Redis Streams Source
• Performance improvements
  • Message SerDe has been optimized for better encoding/decoding
  • WAL write/read CPU performance was boosted by 20%, and disk space optimization by close to 40%
• Controller to support high availability
• Kill switch feature - the ability to drop a message on demand when the buffer is full
• UX improvements
  • Ability to do namespace scoped installation
  • Switched to namespace search on the main page
  • Be able to show reduce vertex and edge information
• SASL support for both Kafka source and sink
• Late data processing using allowedLateness

Breaking Changes

• Introduced tags for conditional forwarding, removed keyIn, see the detail at here
• SDK changes
  • Please move to the latest 0.4.x release of each SDK
  • Python
    • Major refactor on server startup, please review the README
    • Added Async Map and Async Sink
    • Map has multi-process support via SO_REUSEPORT
    • Removed Python 3.8 support and added support for 3.11

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.7.0...v0.8.0

Contributors

@KeranYang
@vigith
@yhl25
@ashwinidulams
@juliev0
@chromevoid
@dpadhiar
@veds-g
@sandangel
@whynowy
@kohlisid
@xdevxy
@alexanghh

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.0/install.yaml

Major Changes

• Ability to assign event time, as well as filtering and transformations at the source vertex using source transformers
• Idle watermark propagation now does watermark progression in idle edges reducing close-of-book latency
• Message delivery count is now being tracked and is exposed to user-defined code
• Support Redis Streams Source
• Performance improvements
  • Message SerDe has been optimized for better encoding/decoding
  • WAL write/read CPU performance was boosted by 20%, and disk space optimization by close to 40%
• Controller to support high availability
• Kill switch feature - the ability to drop a message on demand when the buffer is full
• UX improvements
  • Ability to do namespace scoped installation
  • Switched to namespace search on the main page
  • Be able to show reduce vertex and edge information
• SASL support for both Kafka source and sink

Breaking Changes

• Introduced tags for conditional forwarding, removed keyIn, see the detail at here
• SDK changes
  • Please move to the latest 0.4.x release of each SDK
  • Python
    • Major refactor on server startup, please review the README
    • Added Async Map and Async Sink
    • Map has multi-process support via SO_REUSEPORT
    • Removed Python 3.8 support and added support for 3.11

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.7.0...v0.8.0-rc1

Contributors

@KeranYang
@vigith
@yhl25
@ashwinidulams
@juliev0
@chromevoid
@dpadhiar
@veds-g
@sandangel
@whynowy
@kohlisid
@xdevxy
@alexanghh

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.0-rc1/install.yaml

Major Changes

• Kill Switch to avoid ISB overruns (#524)
• UX Improvements
• Performance improvements in WAL SerDe
• SASL support for Kafka Source
• Idle Watermark Propagation

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.7.2...v0.7.3

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.3/install.yaml

Major Changes

• Continuous performance improvement and bug fixes for Reduce operation, switched to bidirectional streaming.
• Watermark propagation improvement to reduce delay
• UI changes including reduce reduce vertex support, and enabling namespace scoped installation.
• Controll HA support
• targetBufferUsage for vertex autoscaling is deprecated, use targetBufferAvailability instead, will be removed in v0.9.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.7.1...v0.7.2

Contributors

@KeranYang
@yhl25
@vigith
@chromevoid
@whynowy
@dpadhiar
@ashwinidulams
@kohlisid
@veds-g

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.2/install.yaml

Major Changes

• Major performance improvement for Reduce operation
  • de-multiplexing gRPC streams
  • asynchronous development pattern is recommended for all SDKs

Breaking Changes

• gRPC streams are per window per slot, will work only with latest 0.3.x SDK versions
• only asyncio is supported for Python
• Java uses Akka

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.7.0...v0.7.1

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.1/install.yaml

Major Changes

• Reduce with Windowing is now possible
  • Fixed
  • Sliding

You can read about the new Windowing and Reduce feature in our latest blog.

Breaking Changes

• UDSink is now a streaming RPC call (no longer unary)
  • Please use SDK version 0.3.x

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.6.5...v0.7.0

Contributors

@DevilAeron
@chromevoid
@dseapy
@whynowy
@KeranYang
@yhl25
@xdevxy
@vigith
@ashwinidulams
@si3nloong
@dratler
@edlee2121

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.0/install.yaml

Major Changes

• Windowing is now possible
  • Fixed
  • Sliding
• Aggregation feature (Reduce)

Breaking Changes

• UDSink is now a streaming RPC call (no longer unary)
  • Please use SDK version 0.3.x

Pull Requests

• feat: reduce metrics. Closes #313 (#414)
• feat: udsink grpc stream (#421)
• test: sliding window e2e test (#427)
• chore: validate if pipeline or vertex name too long (#428)
• chore(doc): scope UDF under a dir (#426)
• chore: refactor reduce ctx (#419)
• chore: update PBQ WAL metrics (#422)
• feat: sliding window. closes #339 (#354)
• refactor: nats/jetstream testing (#418)
• feat: nats as source (#411)
• docs: updated CHANGELOG.md (#415)
• chore: Windower interface refactor (#409)
• fix: adding lock while discovering partitions, Closes #412 (#413)
• fix(test): e2e-api-pod can not start on M1 mac (#410)
• chore: e2e tests for reduce. Closes #338 (#386)
• fix: getWatermark to return-1 if any processor returns -1 (#402)
• chore: add pipeline and vertex name validation. Closes #342 (#398)
• chore: default logs to user defined containers if there are. Close #283 (#397)
• fix: e2e testing for PBQ WAL with reduce pipeline (#393)
• chore: pass vertex/pipeline name to pbq wal and add a grafana dashboard template for wal metrics. Closes #365 (#385)
• feat: add Grafana instruction and a dashboard template. Closes #287 (#381)
• chore: reduce max msgs for jetstream config (#395)
• chore:non keyed stream support for reduce (#394)
• chore: fix broken link (#391)
• chore: adding DeleteStore method to store provider interface (#390)
• fix: unit tests for replay. Closes #373 (#377)
• chore: switch to sync publish mode for jetstream isb (#388)
• chore(docs): update docs (#380)
• chore: make tickgen ID unique (#379)
• chore: concurrent acks inside readloop (#376)
• chore: add error metrics for WAL (#374)
• fix: best effort processing during SIGTERM. Closes #371 (#372)
• chore: clean up unused code (#370)
• feat(wal): First pass to implement WAL and hook to PBQ store. (#344)
• feat: watermark otwatcher enhancement (#364)
• refactor(docs): group docs in categories (#362)
• chore: vertex controller stop watching pod create events (#361)
• chore(deps): bump loader-utils from 2.0.3 to 2.0.4 in /ui (#356)
• fix(controller): vertex nil check for edge listing. Fixes #352 (#353)
• chore: minor window refactor (#347)
• fix: data race in pbq manager. Closes #348 (#349)
• Chore: Windower interface. closes #234 (#340)
• chore: fix broken development doc and remove incorrect make cmds (#346)
• feat: add minikube, kind and podman support (#206)
• refactor: Close watermark fetcher and publisher correctly (#336)
• passing window information inside the context (#341)
• feat: timestamp in UI to display milliseconds. closes #280 (#337)
• Simple reduce pipeline. Fixes #289 (#317)
• feat: add blackhole sink. Closes #329 (#330)
• fix: move watermark based on the head of the read batch (#332)
• feat: configurable jetstream storage (#328)
• feat: support adding sidecars in vertex pods. Closes #323 (#325)
• feat: populate watermark settings to vertex spec. Closes #320 (#321)
• chore: use github action services and upgrade Jetstream (#322)
• doc: add few use cases (#318)
• chore: set GOMEMLIMIT and increase graceshutdownperiod (#319)
• Chore: run in sdks-e2e tests, python-udsink log check before go-udsink (#315)
• fix: jetstream build watermark progressors bug (#316)
• feat: update watermark offset bucket implementation (#307)
• chore: upgrade xmldom to address security concerns (#309)
• feat: shuffling support (#306)
• chore: Add boilerplate headers (#308)
• feat: customize init-container resources. Closes #303 (#304)
• feat: watermark - remove non-share OT bucket option (#302)
• feat: customization for batch jobs. Closes #259 (#300)
• refactor: abstract pod template (#296)
• feat: customization for daemon deployment. Closes #223 (#290)
• chore: use milliseconds in HTTP event-time header (#295)
• feat: add pvc support for reduce vertex PBQ (#292)
• fix(doc): hyperlink for security doc (#288)
• feat: support adding init containers to vertices. Closes #284 (#285)
• docs: fix spelling in docs and examples (#286)
• fix: retry when getting EOF error at E2E test (#281)
• feat: Watermark millisecond. Fixes #201 (#278)
• feat: add pipeline node counts and age to printcolumn. Closes #267 (#282)
• feat: introduce reduce UDF. Closes #246 (#262)
• feat: add pandoc to required tools development doc. Closes #276 (#277)
• feat: add isbsvc type and age to printcolumn. Closes #268 (#275)
• fix: watermark consumer fix (#273)
• refactor: generalize watermark fetching as an interface of ISB service. Fixes #252 (#263)
• fix: set default property values for minimal CRD installation (#264)
• fix: validate only one isbsvc implementation type is defined. Fixes #269 (#271)
• fix: main branch make build failure: math.MaxInt64 for int type (#265)
• fix: nil pointer deref when running example with minimal CRD. Fixes #260 (#261)
• fix: retry only the failed offsets (#255)
• docs: update ISB redis doc (#258)
• fix: re-enable build constraint on processor manager test. Fixes #256 (#257)
• fix: container resource for jetstream isbsvc. Fixes #253 (#254)
• fix: update vertex watermark fetching logic. Fixes: #134 (#245)
• chore: refactor to include partition.ID for reducer (#243)
• chore: introduce simple int offset (#250)
• fix: watermark watcher leak (#242)
• fix(docs): fix a typo (#241)
• feat: Support running UX server with namespace scope. Fixes #248 (#249)
• test: reduce test with inmemory watermark store (#239)
• fix(manifests): Include ServiceAccount in namespace scoped install (#240)
• fix: Watermark close fix and removed the nil check (#238)
• fix: skip publishing watermarks to unexpected vertices. Fixes #235 (#236)
• fix: update default watermark to -1. Fixes #133 (#218)
• feat: support disabling TLS and changing port for UX server (#228)
• feat: reducer for stream aggregation without fault tolerance (#208)
• feat: in-memory watermark store for better testing (#216)
• Add USERS.md (#221)
• fix(watermark): generator should not publish wm for every message (#217)
• chore: reference the Payload directly when writing to Redis ISB to avoid extra marshaling (#213)

Contributors

@DevilAeron
@chromevoid
@dseapy
@whynowy
@KeranYang
@yhl25
@xdevxy
@vigith
@ashwinidulams
@si3nloong
@dratler
@edlee2121

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.7.0-rc1/config/install.yaml

Major Changes

• Watermark related bug fixes
• Fine tune JetStream retention policy
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• chore: Windower interface refactor (#409)
• fix: adding lock while discovering partitions, Closes #412 (#413)
• fix(test): e2e-api-pod can not start on M1 mac (#410)
• chore: e2e tests for reduce. Closes #338 (#386)
• fix: getWatermark to return-1 if any processor returns -1 (#402)
• chore: add pipeline and vertex name validation. Closes #342 (#398)
• chore: default logs to user defined containers if there are. Close #283 (#397)
• fix: e2e testing for PBQ WAL with reduce pipeline (#393)
• chore: pass vertex/pipeline name to pbq wal and add a grafana dashboard template for wal metrics. Closes #365 (#385)
• feat: add Grafana instruction and a dashboard template. Closes #287 (#381)
• chore: reduce max msgs for jetstream config (#395)
• chore:non keyed stream support for reduce (#394)
• chore: fix broken link (#391)
• chore: adding DeleteStore method to store provider interface (#390)
• fix: unit tests for replay. Closes #373 (#377)

Contributors

@KeranYang
@whynowy
@yhl25
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.5/config/install.yaml

Major Changes

• Switched JetStream ISB writing from async to concurrent sync
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• chore: switch to sync publish mode for jetstream isb (#388)
• chore(docs): update docs (#380)
• chore: make tickgen ID unique (#379)
• chore: concurrent acks inside readloop (#376)
• chore: add error metrics for WAL (#374)
• fix: best effort processing during SIGTERM. Closes #371 (#372)
• chore: clean up unused code (#370)
• feat(wal): First pass to implement WAL and hook to PBQ store. (#344)

Contributors

@vigith
@whynowy
@yhl25
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.4/config/install.yaml

Major Changes

• Some bug fixes and performance enhancements related to watermark
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• feat: watermark otwatcher enhancement (#364)
• refactor(docs): group docs in categories (#362)
• chore: vertex controller stop watching pod create events (#361)
• chore(deps): bump loader-utils from 2.0.3 to 2.0.4 in /ui (#356)
• fix(controller): vertex nil check for edge listing. Fixes #352 (#353)
• chore: minor window refactor (#347)
• fix: data race in pbq manager. Closes #348 (#349)
• Chore: Windower interface. closes #234 (#340)
• chore: fix broken development doc and remove incorrect make cmds (#346)
• feat: add minikube, kind and podman support (#206)
• refactor: Close watermark fetcher and publisher correctly (#336)
• passing window information inside the context (#341)
• feat: timestamp in UI to display milliseconds. closes #280 (#337)
• Simple reduce pipeline. Fixes #289 (#317)
• feat: add blackhole sink. Closes #329 (#330)
• fix: move watermark based on the head of the read batch (#332)
• feat: configurable jetstream storage (#328)
• feat: support adding sidecars in vertex pods. Closes #323 (#325)

Contributors

@chromevoid
@dseapy
@DevilAeron
@dratler
@ashwinidulams
@whynowy
@yhl25
@vigith

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.3/config/install.yaml

Major Changes

• Fixed some watermarks related bugs
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• feat: populate watermark settings to vertex spec. Closes #320 (#321)
• chore: use github action services and upgrade Jetstream (#322)
• doc: add few use cases (#318)
• chore: set GOMEMLIMIT and increase graceshutdownperiod (#319)
• Chore: run in sdks-e2e tests, python-udsink log check before go-udsink (#315)
• fix: jetstream build watermark progressors bug (#316)
• feat: update watermark offset bucket implementation (#307)
• chore: upgrade xmldom to address security concerns (#309)
• feat: shuffling support (#306)
• chore: Add boilerplate headers (#308)
• feat: customize init-container resources. Closes #303 (#304)
• feat: watermark - remove non-share OT bucket option (#302)
• feat: customization for batch jobs. Closes #259 (#300)
• refactor: abstract pod template (#296)
• feat: customization for daemon deployment. Closes #223 (#290)
• chore: use milliseconds in HTTP event-time header (#295)
• feat: add pvc support for reduce vertex PBQ (#292)
• fix(doc): hyperlink for security doc (#288)
• feat: support adding init containers to vertices. Closes #284 (#285)
• docs: fix spelling in docs and examples (#286)
• fix: retry when getting EOF error at E2E test (#281)
• feat: Watermark millisecond. Fixes #201 (#278)
• feat: add pipeline node counts and age to printcolumn. Closes #267 (#282)

Contributors

@chromevoid
@dseapy
@whynowy
@KeranYang
@vigith

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.2/config/install.yaml