bytewax

Overview

• Fixes a bug when using
  {py:obj}~bytewax.operators.windowing.EventClock where in-order but
  "slow" data results in watermark assertion errors.

What's Changed

• Fix watermark assertion errors with EventClock by @davidselassie in https://github.com/bytewax/bytewax/pull/470
• Prep for v0.20.1 release by @davidselassie in https://github.com/bytewax/bytewax/pull/471

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.20.0...v0.21.1

Overview

• Fixes issue with multi-worker recovery. If the cluster crashed
  before all workers had completed their first epoch, the cluster
  would resume from the incorrect position. This requires a change to
  the recovery store. You cannot resume from recovery data written
  with an older version.

What's Changed

• Fixes link to CD actions page by @davidselassie in https://github.com/bytewax/bytewax/pull/185
• Fix a bug with list serialization. by @whoahbot in https://github.com/bytewax/bytewax/pull/189
• Replace generic epoch with explicit u64 by @davidselassie in https://github.com/bytewax/bytewax/pull/192
• Store execution IDs and worker count for recovery by @davidselassie in https://github.com/bytewax/bytewax/pull/193
• Bumps version to 0.15.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/194

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.14.0...v0.15.0

Overview

• Dataflow continuation now works. If you run a dataflow over a finite
  input, all state will be persisted via recovery so if you re-run the
  same dataflow pointing at the same input, but with more data
  appended at the end, it will correctly continue processing from the
  previous end-of-stream.

• Fixes issue with multi-worker recovery. Previously resume data was
  being routed to the wrong worker so state would be missing.

• The above two changes require that the recovery format has been
  changed for all recovery stores. You cannot resume from recovery
  data written with an older version.

• Adds an introspection web server to dataflow workers.

• Adds collect_window operator.

What's Changed

• Adding manylinux_2_27 wheel building to CI by @miccioest in https://github.com/bytewax/bytewax/pull/169
• Adds webserver by @whoahbot in https://github.com/bytewax/bytewax/pull/175
• Added EXPOSE command in Dockerfiles by @Psykopear in https://github.com/bytewax/bytewax/pull/176
• Adding 3.8, 3.9, and 3.10 python versions to colab CI job by @miccioest in https://github.com/bytewax/bytewax/pull/179
• Use cbfmt with pre-commit by @whoahbot in https://github.com/bytewax/bytewax/pull/180
• Fix issue with resume state not being routed to correct worker by @davidselassie in https://github.com/bytewax/bytewax/pull/182
• Adds collect window operator by @davidselassie in https://github.com/bytewax/bytewax/pull/183
• Preps for v0.14.0 release by @davidselassie in https://github.com/bytewax/bytewax/pull/184

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.13.1...v0.14.0

Overview

• Added Google Colab support.

What's Changed

• Prepare for v0.13.1 release. by @miccioest in https://github.com/bytewax/bytewax/pull/168

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.13.0...v0.13.1

Overview

• Bugfix: Resolves pickling error. KafkaInputConfig now works with
  spawn_cluster.

What's Changed

• Pickling logic for auto_commit in KafkaInputConfig by @blakestier in #102

Overview

• Input is no longer specified using an input_builder, but now an
  input_config which allows you to use pre-built input
  components. See bytewax.inputs for more info.

• Preliminary support for a pre-built Kafka input component. See
  bytewax.inputs.KafkaInputConfig.

• Keys used in the (key, value) 2-tuples to route data for stateful
  operators (like stateful_map and reduce_epoch) must now be
  strings. Because of this bytewax.exhash is no longer necessary and
  has been removed.

• Recovery format has been changed for all recovery stores. You cannot
  resume from recovery data written with an older version.

• Slight changes to bytewax.recovery.RecoveryConfig config options
  due to recovery system changes.

• bytewax.run() and bytewax.run_cluster() no longer take
  recovery_config as they don't support recovery.

What's Changed

• New execution interface by @davidselassie in https://github.com/bytewax/bytewax/pull/19
• Container and Kubernetes related improvements by @miccioest in https://github.com/bytewax/bytewax/pull/24
• Restructure Python imports by @whoahbot in https://github.com/bytewax/bytewax/pull/22
• Restructure rust by @whoahbot in https://github.com/bytewax/bytewax/pull/25
• Renames test_run to test_execution by @davidselassie in https://github.com/bytewax/bytewax/pull/27
• Adds an "order" input helper and allows tumbling "event time" by @davidselassie in https://github.com/bytewax/bytewax/pull/26
• Maturin develop before running tests. by @whoahbot in https://github.com/bytewax/bytewax/pull/28
• Tests ability to interrupt execution by @davidselassie in https://github.com/bytewax/bytewax/pull/23
• Jupyter Notebook Anomaly Detection Example by @awmatheson in https://github.com/bytewax/bytewax/pull/30
• Debug operators by @blakestier in https://github.com/bytewax/bytewax/pull/29
• Running tests using whl file already build by @miccioest in https://github.com/bytewax/bytewax/pull/32
• More comprehensive readme by @awmatheson in https://github.com/bytewax/bytewax/pull/33
• modify inputs by @awmatheson in https://github.com/bytewax/bytewax/pull/35
• Remove Criterion benchmark by @whoahbot in https://github.com/bytewax/bytewax/pull/37
• Add a key param to the stateful_map operator. by @whoahbot in https://github.com/bytewax/bytewax/pull/36
• Runs doctests in CI by @davidselassie in https://github.com/bytewax/bytewax/pull/38
• API Docs by @davidselassie in https://github.com/bytewax/bytewax/pull/39
• Fixes sorted_window() to support items with identical times by @davidselassie in https://github.com/bytewax/bytewax/pull/41
• Apidocs templates by @konradsienkowski in https://github.com/bytewax/bytewax/pull/40
• Testable examples by @davidselassie in https://github.com/bytewax/bytewax/pull/34
• Examples metadata by @konradsienkowski in https://github.com/bytewax/bytewax/pull/43
• Update to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/45
• Runs CI on release publish by @davidselassie in https://github.com/bytewax/bytewax/pull/46
• Feast example by @blakestier in https://github.com/bytewax/bytewax/pull/42
• Adds run_main() by @davidselassie in https://github.com/bytewax/bytewax/pull/48
• Example of Apriori algorithm by @TheBits in https://github.com/bytewax/bytewax/pull/50
• Dockerfile enhancements by @miccioest in https://github.com/bytewax/bytewax/pull/52
• Integrate docs with main repository by @konradsienkowski in https://github.com/bytewax/bytewax/pull/51
• Fix le feast typos by @blakestier in https://github.com/bytewax/bytewax/pull/54
• Adding a Kubernetes example based on manual_cluster by @miccioest in https://github.com/bytewax/bytewax/pull/55
• Update docs with capture operator by @awmatheson in https://github.com/bytewax/bytewax/pull/56
• Update Notebook Example to Bytewax Version 0.8 by @awmatheson in https://github.com/bytewax/bytewax/pull/57
• Re-enable documentation tests and upgrade longform docs to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/58
• Frontier psychiatry by @whoahbot in https://github.com/bytewax/bytewax/pull/59
• Move operator descriptions into API docs by @davidselassie in https://github.com/bytewax/bytewax/pull/60
• Prepare for v0.9.0 release. by @whoahbot in https://github.com/bytewax/bytewax/pull/61
• Moves deployment docs into repo by @davidselassie in https://github.com/bytewax/bytewax/pull/62
• State Recovery by @davidselassie in https://github.com/bytewax/bytewax/pull/53
• Uploading wheels to S3 in CI and using them in CD without building again by @miccioest in https://github.com/bytewax/bytewax/pull/64
• Fix broken links in longform documentation by @konradsienkowski in https://github.com/bytewax/bytewax/pull/63
• Clean up error messages for inputs by @whoahbot in https://github.com/bytewax/bytewax/pull/69
• Adds distribute() helper by @davidselassie in https://github.com/bytewax/bytewax/pull/71
• Order Book example Using Websockets by @awmatheson in https://github.com/bytewax/bytewax/pull/67
• Automatic state recovery / garbage collection by @davidselassie in https://github.com/bytewax/bytewax/pull/73
• Pre commit integration by @kasun in https://github.com/bytewax/bytewax/pull/74
• Fix getting started guide documentation by @kasun in https://github.com/bytewax/bytewax/pull/76
• Fix inputs helper by @blakestier in https://github.com/bytewax/bytewax/pull/77
• cargo fmt hook by @davidselassie in https://github.com/bytewax/bytewax/pull/79
• Uses new JoinHandle::is_finished by @davidselassie in https://github.com/bytewax/bytewax/pull/78
• Kafka recovery store by @davidselassie in https://github.com/bytewax/bytewax/pull/81
• Require state keys to be strings by @davidselassie in https://github.com/bytewax/bytewax/pull/82
• Move recovery_wordcount.py into examples by @davidselassie in https://github.com/bytewax/bytewax/pull/83
• A few touch ups to docs by @blakestier in https://github.com/bytewax/bytewax/pull/85
• Fix API Docs crawling by @konradsienkowski in https://github.com/bytewax/bytewax/pull/88
• Kafka Consumer by @blakestier in https://github.com/bytewax/bytewax/pull/84
• Multi-worker recovery by @davidselassie in https://github.com/bytewax/bytewax/pull/89
• Prep for 0.10.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/91

New Contributors

• @blakestier made their first contribution in https://github.com/bytewax/bytewax/pull/29
• @konradsienkowski made their first contribution in https://github.com/bytewax/bytewax/pull/40
• @TheBits made their first contribution in https://github.com/bytewax/bytewax/pull/50
• @kasun made their first contribution in https://github.com/bytewax/bytewax/pull/74

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.1...v0.10.0

Overview

• Capture operator no longer takes arguments. Items that flow through
  those points in the dataflow graph will be processed by the output
  handlers setup by each execution entry point. Every dataflow
  requires at least one capture.

• Executor.build_and_run() is replaced with four entry points for
  specific use cases:

  • run() for exeuction in the current process. It returns all
    captured items to the calling process for you. Use this for
    prototyping in notebooks and basic tests.

  • run_cluster() for execution on a temporary machine-local cluster
    that Bytewax coordinates for you. It returns all captured items to
    the calling process for you. Use this for notebook analysis where
    you need parallelism.

  • spawn_cluster() for starting a machine-local cluster with more
    control over input and output. Use this for standalone scripts
    where you might need partitioned input and output.

  • cluster_main() for starting a process that will participate in a
    cluster you are coordinating manually. Use this when starting a
    Kubernetes cluster.

• Adds bytewax.parse module to help with reading command line
  arguments and environment variables for the above entrypoints.

• Renames bytewax.inp to bytewax.inputs.

What's Changed

• New execution interface by @davidselassie in https://github.com/bytewax/bytewax/pull/19
• Container and Kubernetes related improvements by @miccioest in https://github.com/bytewax/bytewax/pull/24
• Restructure Python imports by @whoahbot in https://github.com/bytewax/bytewax/pull/22
• Restructure rust by @whoahbot in https://github.com/bytewax/bytewax/pull/25
• Renames test_run to test_execution by @davidselassie in https://github.com/bytewax/bytewax/pull/27
• Adds an "order" input helper and allows tumbling "event time" by @davidselassie in https://github.com/bytewax/bytewax/pull/26
• Maturin develop before running tests. by @whoahbot in https://github.com/bytewax/bytewax/pull/28
• Tests ability to interrupt execution by @davidselassie in https://github.com/bytewax/bytewax/pull/23
• Jupyter Notebook Anomaly Detection Example by @awmatheson in https://github.com/bytewax/bytewax/pull/30
• Debug operators by @blakestier in https://github.com/bytewax/bytewax/pull/29
• Running tests using whl file already build by @miccioest in https://github.com/bytewax/bytewax/pull/32
• More comprehensive readme by @awmatheson in https://github.com/bytewax/bytewax/pull/33
• modify inputs by @awmatheson in https://github.com/bytewax/bytewax/pull/35
• Remove Criterion benchmark by @whoahbot in https://github.com/bytewax/bytewax/pull/37
• Add a key param to the stateful_map operator. by @whoahbot in https://github.com/bytewax/bytewax/pull/36
• Runs doctests in CI by @davidselassie in https://github.com/bytewax/bytewax/pull/38
• API Docs by @davidselassie in https://github.com/bytewax/bytewax/pull/39
• Fixes sorted_window() to support items with identical times by @davidselassie in https://github.com/bytewax/bytewax/pull/41
• Apidocs templates by @konradsienkowski in https://github.com/bytewax/bytewax/pull/40
• Testable examples by @davidselassie in https://github.com/bytewax/bytewax/pull/34
• Examples metadata by @konradsienkowski in https://github.com/bytewax/bytewax/pull/43
• Update to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/45
• Runs CI on release publish by @davidselassie in https://github.com/bytewax/bytewax/pull/46

New Contributors

• @blakestier made their first contribution in https://github.com/bytewax/bytewax/pull/29
• @konradsienkowski made their first contribution in https://github.com/bytewax/bytewax/pull/40

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.1...v0.8.0