numaflow

Full Changelog: https://github.com/numaproj/numaflow/compare/v1.2.0-rc3...v1.2.0-rc4

Major Changes

• Fix: pass headers to custom UDSink (#1653)
• Fix: pass headers to Source Transformer (#1651)
• Fix: avoid publishing watermarks for duplicate messages (#1649)
• Fix: Dedup not working for multi-partitioned edge (#1639)
• Fix: readonly view (#1640)

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.2.0-rc2...v1.2.0-rc3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc3/validating-webhook-install.yaml

Major Changes

• Fix: dedup in user-defined source #1613
• Fix: allow pipeline to start with redis isbsvc (Fixes: #1513) #1567
• Fix: watermark progression during pods creation/deletion #1619
• Fix: numaflow package style fixes in #1622
• Fix: bug in late message handling for sliding window #1471
• Fix: race condition while publishing wm inside reduce #1599
• Feat: read-only view for UI #1628

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.2.0-rc1...v1.2.0-rc2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.2.0-rc2/validating-webhook-install.yaml

Major Features 🔥

• Introducing Session Window and Streaming Reduce Capability
• Ability to pass headers from Source to Sink

🚨 Breaking Change 🚨

• SDKs have to be updated to the latest version
• Python SDK has interface changes.

New Features 🔥

• feat: health status implementation by @kohlisid in https://github.com/numaproj/numaflow/pull/1406
• feat: Session Window and Reduce Streaming by @yhl25 in https://github.com/numaproj/numaflow/pull/1384
• feat: update tcp client connections by @kohlisid in https://github.com/numaproj/numaflow/pull/1429
• feat: enhance autoscaling peeking logic by @akashjkhamkar in https://github.com/numaproj/numaflow/pull/1432
• feat: pipeline health status for UI by @veds-g in https://github.com/numaproj/numaflow/pull/1460
• feat: terminate reduce vertex pods when pausing pipeline by @dpadhiar in https://github.com/numaproj/numaflow/pull/1481
• feat: configure standardResources via controller configmap by @inishchith in https://github.com/numaproj/numaflow/pull/1490
• feat: add support for SASL SCRAM 256 and 512 for Kafka by @Dnomyar in https://github.com/numaproj/numaflow/pull/1518
• feat: noop persistence store for reduce by @yhl25 in https://github.com/numaproj/numaflow/pull/1532
• feat: unaligned wal by @yhl25 in https://github.com/numaproj/numaflow/pull/1511
• feat: numaflow package by @veds-g in https://github.com/numaproj/numaflow/pull/1579
• feat: support headers for message by @yhl25 in https://github.com/numaproj/numaflow/pull/1578

Major Fixes 🐞

• fix: configmap const name by @whynowy in https://github.com/numaproj/numaflow/pull/1423
• fix: server-secrets-init container restart by @veds-g in https://github.com/numaproj/numaflow/pull/1433
• fix: Kafka source reads duplicated messages by @jy4096 in https://github.com/numaproj/numaflow/pull/1438
• fix: UI Filter by status for pipelines doesn't work as expected by @inishchith in https://github.com/numaproj/numaflow/pull/1444
• fix: GetDownstreamEdges is not cycle safe by @inishchith in https://github.com/numaproj/numaflow/pull/1447
• fix: memory leak inside session windower by @yhl25 in https://github.com/numaproj/numaflow/pull/1445
• fix: bug where dashed line is treated as permalink + spelling/formatting by @ayildirim21 in https://github.com/numaproj/numaflow/pull/1467
• fix(controller): incorrect cpu/mem resources calculation by @whynowy in https://github.com/numaproj/numaflow/pull/1477
• fix: add idle handler offset nil check by @jy4096 in https://github.com/numaproj/numaflow/pull/1489
• fix: unknown for ISB details in pipeline card by @veds-g in https://github.com/numaproj/numaflow/pull/1497
• fix: avoid panic when ctx is canceled by @yhl25 in https://github.com/numaproj/numaflow/pull/1515
• fix: idle manager refactor for multi partitions by @jy4096 in https://github.com/numaproj/numaflow/pull/1512
• fix: podSpec incorrectly configured in case template exist by @inishchith in https://github.com/numaproj/numaflow/pull/1516
• fix: incorrect json schema for tls config by @whynowy in https://github.com/numaproj/numaflow/pull/1520
• fix(controller): vertex template metadata nil check by @inishchith in https://github.com/numaproj/numaflow/pull/1527
• fix: break from retry loop when key is not found by @yhl25 in https://github.com/numaproj/numaflow/pull/1535
• fix: initialize inflightAcks channel to not nil channel by @afugazzotto in https://github.com/numaproj/numaflow/pull/1548
• fix(codegen): protobuf upgrade by @whynowy in https://github.com/numaproj/numaflow/pull/1558
• fix: error message that cause Buttons overflow in the UI by @a49ibrahim in https://github.com/numaproj/numaflow/pull/1591
• fix: flaky e2e tests by @vigith in https://github.com/numaproj/numaflow/pull/1590

New Contributors ❤️

• @akashjkhamkar made their first contribution in https://github.com/numaproj/numaflow/pull/1432
• @inishchith made their first contribution in https://github.com/numaproj/numaflow/pull/1444
• @ayildirim21 made their first contribution in https://github.com/numaproj/numaflow/pull/1467
• @syayi made their first contribution in https://github.com/numaproj/numaflow/pull/1510
• @Dnomyar made their first contribution in https://github.com/numaproj/numaflow/pull/1518
• @afugazzotto made their first contribution in https://github.com/numaproj/numaflow/pull/1548
• @a49ibrahim made their first contribution in https://github.com/numaproj/numaflow/pull/1591

Full Changelog: https://github.com/numaproj/numaflow/compare/v1.1.0...v1.2.0-rc1

Major Changes

• Fixed a Kafka source issue to avoid potential dead lock (#1548)
• A temporary solution to support including message headers for Kafka source (#1569) - Be aware this might be changed later on.

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.1.6...v1.1.7

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.7/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.7/validating-webhook-install.yaml

Major Changes

• Fixed cpu/mem resources calculation issue [#1477]

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.1.4...v1.1.5

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.5/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.5/validating-webhook-install.yaml

Major Changes

• Fixed late data caused sliding window close issue [#1468]

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.1.3...v1.1.4

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.4/validating-webhook-install.yaml

Major Changes

• Enhanced autoscaling peeking logic [#1432]
• Fixed Kafka source duplicate message issue [#1438]
• Fixed a bug of cycle topology pipeline (#1447)

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.1.2...v1.1.3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.3/validating-webhook-install.yaml

Major Changes

• Fix numaflow-ux server secret query permission issue.

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.1.1...v1.1.2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.2/validating-webhook-install.yaml

Major Changes

• Source idle watermark support
• UI access with local user authentication

🚨 Breaking Change 🚨

• User Defined Source interface requires to implement Partitions() function.
• MessageToDrop function in Source Transformer needs to give event time.

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.1.0...v1.1.1

Contributors

@KeranYang
@vigith
@yhl25
@jy4096
@dpadhiar
@veds-g
@whynowy
@kohlisid
@mshakira
@darshansimha

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.1/install.yaml

Major Changes

• Source idle watermark support
• UI access with local user authentication

🚨 Breaking Change 🚨

• User Defined Source interface requires to implement Partitions() function.
• MessageToDrop function in Source Transformer needs to give event time.

Pull Requests

https://github.com/numaproj/numaflow/compare/v1.0.0...v1.1.0

Contributors

@KeranYang
@vigith
@yhl25
@jy4096
@dpadhiar
@veds-g
@whynowy
@kohlisid
@mshakira
@darshansimha

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.1.0/install.yaml

Major Features

• Kubernetes native streaming data processing platform.
• Built-in sources, sinks and UDFs.
• User defined sources, sinks and functions.
• Language agnostic, SDKs in Go, Python, Java, Rust.
• Autoscaling from 0 - N.
• Map Reduce operations.
• Conditional forwarding.
• Side Inputs support.
• Out-of-box watermark support.
• Pipeline topologies in Line, Tree, Join and Cycle.
• Pipeline operations such as pause and resume.
• Built-in observability.
• UX with AuthN/AuthZ.

What's New in v1.0

• CRUD capabilities with new UX experiences
  • Create/update/delete ISB Services
  • Create/update/delete pipelines
  • Update vertices
  • Pause/resume pipelines
• AuthN/AuthZ support for the UX
  • Dex integration to support OAuth authentication
  • Role-based access control (RBAC)
• Unified metrics names [#1290]

Note: The controller and UX server pods will be enforced with a new label app.kubernetes.io/name, if you see complains when applying new manifests, please run kubectl apply with --force.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.11.0...v1.0.0

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0/validating-webhook-install.yaml

Check more installation options here.

Contributors

@alexanghh
@amitkalamkar
@aruwanip
@ashwinidulams
@bbehnke
@bpcarey01
@cdikibo
@chandankumar4
@darshansimha
@dpadhiar
@dratler
@dseapy
@dsosa17
@edlee2121
@jasonzeshengchen
@joelcomp1
@Jorvaulx
@juliev0
@jy4096
@kbaumzie
@KeranYang
@kohlisid
@Krithika3
@mshakira
@muregii
@qhuai
@RohanAshar
@sandangel
@sarabala1979
@saradhis
@shashank10456
@shubhamdixit863
@si3nloong
@tasneemkoushar
@tczhao
@testwill
@veds-g
@vigith
@VishalSoni07
@whynowy
@xdevxy
@yhl25

What's New

• CRUD capabilities with new UX experiences
  • Create/update/delete ISB Services
  • Create/update/delete pipelines
  • Update vertices
  • Pause/resume pipelines
• AuthN/AuthZ support for the UX
  • Dex integration to support OAuth authentication
  • Role-based access control (RBAC)

Note: The controller and UX server pods will be enforced with a new label app.kubernetes.io/name, if you see complains when applying new manifests, please run kubectl apply with --force.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.11.0...v1.0.0-rc4

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc4/validating-webhook-install.yaml

Check more installation options here.

What's New

• CRUD capabilities with new UX experiences
  • Create/update/delete ISB Services
  • Create/update/delete pipelines
  • Update vertices
  • Pause/resume pipelines
• AuthN/AuthZ support for the UX
  • Dex integration to support OAuth authentication
  • Role-based access control (RBAC)

Note: The controller and UX server pods will be enforced with a new label app.kubernetes.io/name, if you see complains when applying new manifests, please run kubectl apply with --force.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.11.0...v1.0.0-rc3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc3/validating-webhook-install.yaml

Check more installation options here.

What's New

• CRUD capabilities with new UX experiences
  • Create/update/delete ISB Services
  • Create/update/delete pipelines
  • Update vertices
  • Pause/resume pipelines
• AuthN/AuthZ support for the UX
  • Dex integration to support OAuth authentication
  • Role-based access control (RBAC)

Note: The controller and UX server pods will be enforced with a new label app.kubernetes.io/name, if you see complains when applying new manifests, please run kubectl apply with --force.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.11.0...v1.0.0-rc2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc2/validating-webhook-install.yaml

Check more installation options here.

Major Changes

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.11.0...v1.0.0-rc1

Installation

kubectl create namespace numaflow-system

# Install numaflow controller
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.0.0-rc1/validating-webhook-install.yaml

Check more installation options here.

Major Changes

• User-Defined Source Support
  Support using the latest Numaflow SDKs(numaflow-go 0.5.2, numaflow-java 0.5.4 and numaflow-python 0.5.3) to craft your own User-Defined Source.

• UX Improvements
  Improved the overall look and feel of the user interface, making it more visually appealing and user-friendly.

Other enhancements

• Enhanced Pipeline Validations to cover more edge cases.
• Optimized Back Pressure Measurement to always treat dropOnFull buffer as not having back pressure.
• Single-Pod JetStream Inter-Step Buffer Service to support the case when availability is not as important as saving cost.
• Simplified Java UDSink SDK.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.10.0...v0.11.0

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.11.0/install.yaml

Major Changes

• Fixed some new UI bugs
• Do not consider it is back pressured if the edge to the vertex is configured as drop latest on full.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.11.0-rc1...v0.11.0-rc2

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.11.0-rc2/install.yaml

Major Changes

• Added user-defined source support.
• Enhanced user experience - Numaflow UI 1.0.
• Enhanced pipeline validation to avoid creating a disjointed pipeline.
• Fixed a bug related to resuming a paused pipeline.

Pull Requests

https://github.com/numaproj/numaflow/compare/v0.10.0...v0.11.0-rc1

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.11.0-rc1/install.yaml