Major Changes

• Fixed some watermarks related bugs
• Fixed the issue that some unexpected behaviors will happen due to missing default values when doing namespace scoped installation
• Initial integration for reduce vertex

Check out v0.6 release blog.

Pull Requests

• fix(manifests): Include ServiceAccount in namespace scoped install (#240)
• fix(docs): fix a typo (#241)
• feat: introduce reduce UDF. Closes #246 (#262)
• feat: add pandoc to required tools development doc. Closes #276 (#277)
• feat: add isbsvc type and age to printcolumn. Closes #268 (#275)
• fix: watermark consumer fix (#273)
• refactor: generalize watermark fetching as an interface of ISB service. Fixes #252 (#263)
• fix: set default property values for minimal CRD installation (#264)
• fix: validate only one isbsvc implementation type is defined. Fixes #269 (#271)
• fix: main branch make build failure: math.MaxInt64 for int type (#265)
• fix: nil pointer deref when running example with minimal CRD. Fixes #260 (#261)
• fix: retry only the failed offsets (#255)
• docs: update ISB redis doc (#258)
• fix: re-enable build constraint on processor manager test. Fixes #256 (#257)
• fix: container resource for jetstream isbsvc. Fixes #253 (#254)
• fix: update vertex watermark fetching logic. Fixes: #134 (#245)
• chore: refactor to include partition.ID for reducer (#243)
• chore: introduce simple int offset (#250)
• fix: watermark watcher leak (#242)
• feat: Support running UX server with namespace scope. Fixes #248 (#249)
• test: reduce test with inmemory watermark store (#239)
• fix: Watermark close fix and removed the nil check (#238)
• fix: skip publishing watermarks to unexpected vertices. Fixes #235 (#236)
• fix: update default watermark to -1. Fixes #133 (#218)
• feat: support disabling TLS and changing port for UX server (#228)
• feat: reducer for stream aggregation without fault tolerance (#208)
• feat: in-memory watermark store for better testing (#216)
• Add USERS.md (#221)
• fix(watermark): generator should not publish wm for every message (#217)
• chore: reference the Payload directly when writing to Redis ISB to avoid extra marshaling (#213)

Contributors

@chromevoid
@dseapy
@whynowy
@edlee2121
@KeranYang
@si3nloong
@yhl25
@ashwinidulams
@vigith

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.1/config/install.yaml

Major Changes

First official Numaflow release, which contains following major features, check the documentation for all the details.

Check out v0.6 release blog.

• Stream processing pipeline with Source, User Defined Function, and Sink vertices
• Pipeline autoscaling
• Native supported sources and sinks, as well as user defined sinks.
• Watermarks support
• Prometheus metrics
• User interface

Pull Requests

• fix(autoscaling): Ack pending should be included in total pending calculation (#212)
• chore: upgrade xmldom to fix potential security issue (#209)
• fix(autoscaling): Skip autoscaling if vertex is not in running phase (#207)
• chore: add read and written bytes counter (#202)
• chore: Replaced node info text with rich format (#200)
• feat: ISBSVC add support for redis cluster mode (#195)
• ci: add sdk test category (#197)
• docs: add and refine some docs. (#196)
• refactor: move controllers package to pkg/reconciler (#192)
• fix: update udf fetchWatermark and publishWatermark initial values (#193)
• chore: fix security alert (#191)
• chore: generate swagger and schema json (#188)
• chore: probe udsink healthiness (#183)
• chore: clean up code of http protocol udf and udsink (#182)
• fix(docs): readme for UI development (#181)

Contributors

@chromevoid
@whynowy
@Krithika3
@qhuai
@sarabala1979
@kohlisid
@vigith
@yhl25

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.0/config/install.yaml

Major Changes

• User Defined Functions and Sinks switched to gRPC protocol
• Liveness check includes UDF health check
• In-memory PBQ implementation (for windowing)
• Watermark implementation for all the sources
• Support to publish watermark to sink buckets
• Shuffle implementation (for windowing)

Pull Requests

• feat: Add udf gRPC support Fixes #145 (#146)
• chore: more advanced install options (#154)
• feat: check if UDF is running in liveness probe (#156)
• docs:Update readme and quickstart (#162)
• feat: PBQ implementation (#155)
• docs:updated quickstart (#164)
• chore: add mkdocs check (#166)
• chore: keep stores together (#167)
• feat: watermark for sources (#159)
• feat: windowing operations (#157)
• chore: publish watermark to sink bucket and refactor. (#170)
• feat: Shuffle implementation (#169)
• chore: Update README.md (#175)
• fix: broken path (#176)
• chore: update README.md (#178)
• fix: use scale.max if it is set (#179)
• fix: numaflow-go udf example & docs (#177)
• feat: grpc udsink (#174)
• chore: enable jetstream 2.9.0 (#173)
• chore: pass watermark to UDF and fix e2e tests (#171)

Contributors

@vigith
@saradhis
@yhl25
@chromevoid
@whynowy
@edlee2121
@amitkalamkar

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.6/config/install.yaml

Major Changes

• Fixed a controller memory leak issue.
• Fixed couple of UI bugs.
• Expose minimal CRD definition and more installation options.

Pull Requests

• fix: daemon service client memory leak (#161)
• chore: more advanced install options (#154)
• refactor: some refactor on watermark (#149)
• chore: use string as the type of key (#153)
• docs: update current docs (#147)
• fix: Fixed JS bug (#144)
• fix: watermark needs nil check
• chore: fix #88 (#140)
• docs: minor autoscaling doc revise (#131)
• fix: pipeline UI broken when vertex scaling down to 0 (#132)

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.5/config/install.yaml

Major Changes

• Vertex autoscaling
• Watermark exposed in UI
• JetStream InterStepBuffer service auto reconnection - addition to Nats connection auto reconnection
• Support namespace scope installation

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.4/config/install.yaml

Major Changes

• UI
• Vertex processing rate and pending messages available in prometheus metrics
• Some breaking changes
  1. udfWorkers is removed from both pipeline and vertex limits;
  2. bufferMaxLength and bufferUsageLimit moved from vertex.limits to edge.limits;
  3. Numaflow controller deployment name changed from controller-manager to numaflow-controller.

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.3/config/install.yaml

Upgrading Guide

Since there are some breaking changes in this release, upgrading from old versions need to follow below steps:

1. Clean up all the limits.udfWorkers from Pipeline spec if any;
2. Clean up udfWorkers from vertices[*].limits if any;
3. Clean up bufferMaxLength and bufferUsageLimit from vertices[*].limits if any;
4. Delete controller-manager deployment with kubectl -n numaflow-system delete deploy controller-manager;
5. Install the new version following the installation guide above;
6. If there's any bufferMaxLength or bufferUsageLimit cleaned up in step 3, add them to edges[*].limits with same names, and apply the new pipeline spec.

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.2/config/install.yaml

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.1/config/install.yaml

Installation

kubectl create namespace numaflow-system
kubectl apply -n numflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.5.0/config/install.yaml