bytewax

• KafkaInputConfig now accepts additional properties. See
  bytewax.inputs.KafkaInputConfig.

• Support for a pre-built Kafka output component. See
  bytewax.outputs.KafkaOutputConfig.

What's Changed

• Break up large modules into smaller files. by @whoahbot in https://github.com/bytewax/bytewax/pull/113
• Upgrade dependencies by @Psykopear in https://github.com/bytewax/bytewax/pull/114
• Update KafkaInput to prevent listening to non-existent topics by @awmatheson in https://github.com/bytewax/bytewax/pull/117
• Add additional Kafka configs by @awmatheson in https://github.com/bytewax/bytewax/pull/119
• Kafka Output by @blakestier in https://github.com/bytewax/bytewax/pull/118

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.11.0...v0.11.1

• Added the fold_window operator, works like reduce_window but allows
  the user to build the initial accumulator for each key in a builder function.

• Output is no longer specified using an output_builder for the
  entire dataflow, but you supply an "output config" per capture. See
  bytewax.outputs for more info.

• Input is no longer specified on the execution entry point (like
  run_main), it is instead using the Dataflow.input operator.

• Epochs are no longer user-facing as part of the input system. Any
  custom Python-based input components you write just need to be
  iterators and emit items. Recovery snapshots and backups now happen
  periodically, defaulting to every 10 seconds.

• Recovery format has been changed for all recovery stores. You cannot
  resume from recovery data written with an older version.

• The reduce_epoch operator has been replaced with
  reduce_window. It takes a "clock" and a "windower" to define the
  kind of aggregation you want to do.

• run and run_cluster have been removed and the remaining
  execution entry points moved into bytewax.execution. You can now
  get similar prototyping functionality with
  bytewax.execution.run_main and bytewax.execution.spawn_cluster
  using Testing{Input,Output}Configs.

• Dataflow has been moved into bytewax.dataflow.Dataflow.

What's Changed

• Windowing by @davidselassie in https://github.com/bytewax/bytewax/pull/96
• Input source operators by @davidselassie in https://github.com/bytewax/bytewax/pull/98
• Cooperative manual input by @davidselassie in https://github.com/bytewax/bytewax/pull/99
• Fold window operator by @Psykopear in https://github.com/bytewax/bytewax/pull/95
• Added cargo tests, doctests, small fixes by @Psykopear in https://github.com/bytewax/bytewax/pull/108
• Update examples by @blakestier in https://github.com/bytewax/bytewax/pull/105
• Solved some clippy warnings, enabled doctests by @Psykopear in https://github.com/bytewax/bytewax/pull/109
• Stateful input sources by @davidselassie in https://github.com/bytewax/bytewax/pull/103
• Prepare for 0.11.0 release. by @whoahbot in https://github.com/bytewax/bytewax/pull/110

New Contributors

• @Psykopear made their first contribution in https://github.com/bytewax/bytewax/pull/95

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.10.0...v0.11.0

Overview

• Bugfix: Resolves pickling error. KafkaInputConfig now works with
  spawn_cluster.

What's Changed

• Pickling logic for auto_commit in KafkaInputConfig by @blakestier in #102

Overview

• Input is no longer specified using an input_builder, but now an
  input_config which allows you to use pre-built input
  components. See bytewax.inputs for more info.

• Preliminary support for a pre-built Kafka input component. See
  bytewax.inputs.KafkaInputConfig.

• Keys used in the (key, value) 2-tuples to route data for stateful
  operators (like stateful_map and reduce_epoch) must now be
  strings. Because of this bytewax.exhash is no longer necessary and
  has been removed.

• Recovery format has been changed for all recovery stores. You cannot
  resume from recovery data written with an older version.

• Slight changes to bytewax.recovery.RecoveryConfig config options
  due to recovery system changes.

• bytewax.run() and bytewax.run_cluster() no longer take
  recovery_config as they don't support recovery.

What's Changed

• New execution interface by @davidselassie in https://github.com/bytewax/bytewax/pull/19
• Container and Kubernetes related improvements by @miccioest in https://github.com/bytewax/bytewax/pull/24
• Restructure Python imports by @whoahbot in https://github.com/bytewax/bytewax/pull/22
• Restructure rust by @whoahbot in https://github.com/bytewax/bytewax/pull/25
• Renames test_run to test_execution by @davidselassie in https://github.com/bytewax/bytewax/pull/27
• Adds an "order" input helper and allows tumbling "event time" by @davidselassie in https://github.com/bytewax/bytewax/pull/26
• Maturin develop before running tests. by @whoahbot in https://github.com/bytewax/bytewax/pull/28
• Tests ability to interrupt execution by @davidselassie in https://github.com/bytewax/bytewax/pull/23
• Jupyter Notebook Anomaly Detection Example by @awmatheson in https://github.com/bytewax/bytewax/pull/30
• Debug operators by @blakestier in https://github.com/bytewax/bytewax/pull/29
• Running tests using whl file already build by @miccioest in https://github.com/bytewax/bytewax/pull/32
• More comprehensive readme by @awmatheson in https://github.com/bytewax/bytewax/pull/33
• modify inputs by @awmatheson in https://github.com/bytewax/bytewax/pull/35
• Remove Criterion benchmark by @whoahbot in https://github.com/bytewax/bytewax/pull/37
• Add a key param to the stateful_map operator. by @whoahbot in https://github.com/bytewax/bytewax/pull/36
• Runs doctests in CI by @davidselassie in https://github.com/bytewax/bytewax/pull/38
• API Docs by @davidselassie in https://github.com/bytewax/bytewax/pull/39
• Fixes sorted_window() to support items with identical times by @davidselassie in https://github.com/bytewax/bytewax/pull/41
• Apidocs templates by @konradsienkowski in https://github.com/bytewax/bytewax/pull/40
• Testable examples by @davidselassie in https://github.com/bytewax/bytewax/pull/34
• Examples metadata by @konradsienkowski in https://github.com/bytewax/bytewax/pull/43
• Update to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/45
• Runs CI on release publish by @davidselassie in https://github.com/bytewax/bytewax/pull/46
• Feast example by @blakestier in https://github.com/bytewax/bytewax/pull/42
• Adds run_main() by @davidselassie in https://github.com/bytewax/bytewax/pull/48
• Example of Apriori algorithm by @TheBits in https://github.com/bytewax/bytewax/pull/50
• Dockerfile enhancements by @miccioest in https://github.com/bytewax/bytewax/pull/52
• Integrate docs with main repository by @konradsienkowski in https://github.com/bytewax/bytewax/pull/51
• Fix le feast typos by @blakestier in https://github.com/bytewax/bytewax/pull/54
• Adding a Kubernetes example based on manual_cluster by @miccioest in https://github.com/bytewax/bytewax/pull/55
• Update docs with capture operator by @awmatheson in https://github.com/bytewax/bytewax/pull/56
• Update Notebook Example to Bytewax Version 0.8 by @awmatheson in https://github.com/bytewax/bytewax/pull/57
• Re-enable documentation tests and upgrade longform docs to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/58
• Frontier psychiatry by @whoahbot in https://github.com/bytewax/bytewax/pull/59
• Move operator descriptions into API docs by @davidselassie in https://github.com/bytewax/bytewax/pull/60
• Prepare for v0.9.0 release. by @whoahbot in https://github.com/bytewax/bytewax/pull/61
• Moves deployment docs into repo by @davidselassie in https://github.com/bytewax/bytewax/pull/62
• State Recovery by @davidselassie in https://github.com/bytewax/bytewax/pull/53
• Uploading wheels to S3 in CI and using them in CD without building again by @miccioest in https://github.com/bytewax/bytewax/pull/64
• Fix broken links in longform documentation by @konradsienkowski in https://github.com/bytewax/bytewax/pull/63
• Clean up error messages for inputs by @whoahbot in https://github.com/bytewax/bytewax/pull/69
• Adds distribute() helper by @davidselassie in https://github.com/bytewax/bytewax/pull/71
• Order Book example Using Websockets by @awmatheson in https://github.com/bytewax/bytewax/pull/67
• Automatic state recovery / garbage collection by @davidselassie in https://github.com/bytewax/bytewax/pull/73
• Pre commit integration by @kasun in https://github.com/bytewax/bytewax/pull/74
• Fix getting started guide documentation by @kasun in https://github.com/bytewax/bytewax/pull/76
• Fix inputs helper by @blakestier in https://github.com/bytewax/bytewax/pull/77
• cargo fmt hook by @davidselassie in https://github.com/bytewax/bytewax/pull/79
• Uses new JoinHandle::is_finished by @davidselassie in https://github.com/bytewax/bytewax/pull/78
• Kafka recovery store by @davidselassie in https://github.com/bytewax/bytewax/pull/81
• Require state keys to be strings by @davidselassie in https://github.com/bytewax/bytewax/pull/82
• Move recovery_wordcount.py into examples by @davidselassie in https://github.com/bytewax/bytewax/pull/83
• A few touch ups to docs by @blakestier in https://github.com/bytewax/bytewax/pull/85
• Fix API Docs crawling by @konradsienkowski in https://github.com/bytewax/bytewax/pull/88
• Kafka Consumer by @blakestier in https://github.com/bytewax/bytewax/pull/84
• Multi-worker recovery by @davidselassie in https://github.com/bytewax/bytewax/pull/89
• Prep for 0.10.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/91

New Contributors

• @blakestier made their first contribution in https://github.com/bytewax/bytewax/pull/29
• @konradsienkowski made their first contribution in https://github.com/bytewax/bytewax/pull/40
• @TheBits made their first contribution in https://github.com/bytewax/bytewax/pull/50
• @kasun made their first contribution in https://github.com/bytewax/bytewax/pull/74

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.1...v0.10.0

Overview

• Adds bytewax.AdvanceTo and bytewax.Emit to control when processing
  happens.

• Adds bytewax.run_main() as a way to test input and output builders
  without starting a cluster.

• Adds a bytewax.testing module with helpers for testing.

• bytewax.run_cluster() and bytewax.spawn_cluster() now take a
  mp_ctx argument to allow you to change the multiprocessing
  behavior. E.g. from "fork" to "spawn". Defaults now to "spawn".

What's Changed

• New execution interface by @davidselassie in https://github.com/bytewax/bytewax/pull/19
• Container and Kubernetes related improvements by @miccioest in https://github.com/bytewax/bytewax/pull/24
• Restructure Python imports by @whoahbot in https://github.com/bytewax/bytewax/pull/22
• Restructure rust by @whoahbot in https://github.com/bytewax/bytewax/pull/25
• Renames test_run to test_execution by @davidselassie in https://github.com/bytewax/bytewax/pull/27
• Adds an "order" input helper and allows tumbling "event time" by @davidselassie in https://github.com/bytewax/bytewax/pull/26
• Maturin develop before running tests. by @whoahbot in https://github.com/bytewax/bytewax/pull/28
• Tests ability to interrupt execution by @davidselassie in https://github.com/bytewax/bytewax/pull/23
• Jupyter Notebook Anomaly Detection Example by @awmatheson in https://github.com/bytewax/bytewax/pull/30
• Debug operators by @blakestier in https://github.com/bytewax/bytewax/pull/29
• Running tests using whl file already build by @miccioest in https://github.com/bytewax/bytewax/pull/32
• More comprehensive readme by @awmatheson in https://github.com/bytewax/bytewax/pull/33
• modify inputs by @awmatheson in https://github.com/bytewax/bytewax/pull/35
• Remove Criterion benchmark by @whoahbot in https://github.com/bytewax/bytewax/pull/37
• Add a key param to the stateful_map operator. by @whoahbot in https://github.com/bytewax/bytewax/pull/36
• Runs doctests in CI by @davidselassie in https://github.com/bytewax/bytewax/pull/38
• API Docs by @davidselassie in https://github.com/bytewax/bytewax/pull/39
• Fixes sorted_window() to support items with identical times by @davidselassie in https://github.com/bytewax/bytewax/pull/41
• Apidocs templates by @konradsienkowski in https://github.com/bytewax/bytewax/pull/40
• Testable examples by @davidselassie in https://github.com/bytewax/bytewax/pull/34
• Examples metadata by @konradsienkowski in https://github.com/bytewax/bytewax/pull/43
• Update to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/45
• Runs CI on release publish by @davidselassie in https://github.com/bytewax/bytewax/pull/46
• Feast example by @blakestier in https://github.com/bytewax/bytewax/pull/42
• Adds run_main() by @davidselassie in https://github.com/bytewax/bytewax/pull/48
• Example of Apriori algorithm by @TheBits in https://github.com/bytewax/bytewax/pull/50
• Dockerfile enhancements by @miccioest in https://github.com/bytewax/bytewax/pull/52
• Integrate docs with main repository by @konradsienkowski in https://github.com/bytewax/bytewax/pull/51
• Fix le feast typos by @blakestier in https://github.com/bytewax/bytewax/pull/54
• Adding a Kubernetes example based on manual_cluster by @miccioest in https://github.com/bytewax/bytewax/pull/55
• Update docs with capture operator by @awmatheson in https://github.com/bytewax/bytewax/pull/56
• Update Notebook Example to Bytewax Version 0.8 by @awmatheson in https://github.com/bytewax/bytewax/pull/57
• Re-enable documentation tests and upgrade longform docs to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/58
• Frontier psychiatry by @whoahbot in https://github.com/bytewax/bytewax/pull/59
• Move operator descriptions into API docs by @davidselassie in https://github.com/bytewax/bytewax/pull/60
• Prepare for v0.9.0 release. by @whoahbot in https://github.com/bytewax/bytewax/pull/61

New Contributors

• @blakestier made their first contribution in https://github.com/bytewax/bytewax/pull/29
• @konradsienkowski made their first contribution in https://github.com/bytewax/bytewax/pull/40
• @TheBits made their first contribution in https://github.com/bytewax/bytewax/pull/50

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.1...v0.9.0

Overview

• Capture operator no longer takes arguments. Items that flow through
  those points in the dataflow graph will be processed by the output
  handlers setup by each execution entry point. Every dataflow
  requires at least one capture.

• Executor.build_and_run() is replaced with four entry points for
  specific use cases:

  • run() for exeuction in the current process. It returns all
    captured items to the calling process for you. Use this for
    prototyping in notebooks and basic tests.

  • run_cluster() for execution on a temporary machine-local cluster
    that Bytewax coordinates for you. It returns all captured items to
    the calling process for you. Use this for notebook analysis where
    you need parallelism.

  • spawn_cluster() for starting a machine-local cluster with more
    control over input and output. Use this for standalone scripts
    where you might need partitioned input and output.

  • cluster_main() for starting a process that will participate in a
    cluster you are coordinating manually. Use this when starting a
    Kubernetes cluster.

• Adds bytewax.parse module to help with reading command line
  arguments and environment variables for the above entrypoints.

• Renames bytewax.inp to bytewax.inputs.

What's Changed

• New execution interface by @davidselassie in https://github.com/bytewax/bytewax/pull/19
• Container and Kubernetes related improvements by @miccioest in https://github.com/bytewax/bytewax/pull/24
• Restructure Python imports by @whoahbot in https://github.com/bytewax/bytewax/pull/22
• Restructure rust by @whoahbot in https://github.com/bytewax/bytewax/pull/25
• Renames test_run to test_execution by @davidselassie in https://github.com/bytewax/bytewax/pull/27
• Adds an "order" input helper and allows tumbling "event time" by @davidselassie in https://github.com/bytewax/bytewax/pull/26
• Maturin develop before running tests. by @whoahbot in https://github.com/bytewax/bytewax/pull/28
• Tests ability to interrupt execution by @davidselassie in https://github.com/bytewax/bytewax/pull/23
• Jupyter Notebook Anomaly Detection Example by @awmatheson in https://github.com/bytewax/bytewax/pull/30
• Debug operators by @blakestier in https://github.com/bytewax/bytewax/pull/29
• Running tests using whl file already build by @miccioest in https://github.com/bytewax/bytewax/pull/32
• More comprehensive readme by @awmatheson in https://github.com/bytewax/bytewax/pull/33
• modify inputs by @awmatheson in https://github.com/bytewax/bytewax/pull/35
• Remove Criterion benchmark by @whoahbot in https://github.com/bytewax/bytewax/pull/37
• Add a key param to the stateful_map operator. by @whoahbot in https://github.com/bytewax/bytewax/pull/36
• Runs doctests in CI by @davidselassie in https://github.com/bytewax/bytewax/pull/38
• API Docs by @davidselassie in https://github.com/bytewax/bytewax/pull/39
• Fixes sorted_window() to support items with identical times by @davidselassie in https://github.com/bytewax/bytewax/pull/41
• Apidocs templates by @konradsienkowski in https://github.com/bytewax/bytewax/pull/40
• Testable examples by @davidselassie in https://github.com/bytewax/bytewax/pull/34
• Examples metadata by @konradsienkowski in https://github.com/bytewax/bytewax/pull/43
• Update to 0.8.0 by @davidselassie in https://github.com/bytewax/bytewax/pull/45
• Runs CI on release publish by @davidselassie in https://github.com/bytewax/bytewax/pull/46

New Contributors

• @blakestier made their first contribution in https://github.com/bytewax/bytewax/pull/29
• @konradsienkowski made their first contribution in https://github.com/bytewax/bytewax/pull/40

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.1...v0.8.0

Beta release

What's Changed

• New execution interface by @davidselassie in https://github.com/bytewax/bytewax/pull/19
• Container and Kubernetes related improvements by @miccioest in https://github.com/bytewax/bytewax/pull/24

Updated execution interface

run() now takes a dataflow and some input, runs it synchronously as a single worker in the existing Python thread, and returns the output to that thread. This is what you'd use in tests and simple notebook work.

run_cluster() takes a dataflow and some input, starts a local cluster of processes, runs it, waits for the cluster to finish
work, then collects thre results, and returns the output to that thread. This is what you'd use in a notebook if you need parallelism or higher throughput.

cluster_main() starts up a cluster of local processes, coordinates the addresses and process IDs between them, runs a dataflow on it, and waits for it to finish. This has a partitioned "input builder" and an "output builder" (discussed below). This is what you'd use if you'd want to write a standalone script or example that does some higher throughput processing.

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.1...v0.8.0-beta.0

v0.7.1

Updates to build_and_run() to support running in notebook environments.

What's Changed

• Parse arguments in build_and_run. by @whoahbot in https://github.com/bytewax/bytewax/pull/10
• Adds a capture operator to capture output by @davidselassie in https://github.com/bytewax/bytewax/pull/9
• Introducing: exhash by @davidselassie in https://github.com/bytewax/bytewax/pull/14
• Exceptions and interrupts in multiple worker threads by @davidselassie in https://github.com/bytewax/bytewax/pull/16
• Fix typing for inp.py by @mttcnnff in https://github.com/bytewax/bytewax/pull/17
• Add in tests for inputs by @whoahbot in https://github.com/bytewax/bytewax/pull/18

New Contributors

• @mttcnnff made their first contribution in https://github.com/bytewax/bytewax/pull/17

Full Changelog: https://github.com/bytewax/bytewax/compare/v0.7.0...v0.7.1