TensorFlowOnSpark | Python Ecosystem Directory

Bot releases are hidden (Show)

TensorFlowOnSpark - v2.2.5 Latest Release

Published by leewyang over 2 years ago

Allow use with tensorflow-cpu package.
Dependency updates
Minor fixes.

TensorFlowOnSpark - v2.2.4

Published by leewyang over 3 years ago

Added option to defer releasing temporary socket/port to user map_function for cases where user code may not bind to the assigned port soon enough to avoid other processes binding to the same port, e.g. extensive pre-processing before invoking TF APIs.
Updated screwdriver.cd build template.
Trigger documentation publish after PyPI push.

TensorFlowOnSpark - v2.2.3

Published by leewyang over 3 years ago

Added ability to disable spark barrier execution in TFParallel
Updated with spark 3 + scala 2.12 dependencies
Fixed documentation build

TensorFlowOnSpark - v2.2.2

Published by leewyang almost 4 years ago

Migrated build from travis-ci to screwdriver.cd

TensorFlowOnSpark - v2.2.1

Published by leewyang over 4 years ago

Added support for port ranges in TFOS_SERVER_PORT environment variable.
Updated mnist/keras/mnist_tf.py example with workaround for tensorflow datasets issue.
Added more detailed error message for missing executor_id.
Added unit tests for gpu allocation variants.

TensorFlowOnSpark - v2.2.0

Published by leewyang over 4 years ago

Added support for Spark 3.0 GPU resources
Updated to support Spark 2.4.5
Fixed dataset ordering in mnist_inference.py examples (thanks to @qsbao)
Added optional environment variables to configure TF server/grpc ports and TensorBoard ports on executors
Fixed bug with TFNode.start_cluster_server in backwards-compatibility code for TF1.x
Fixed file conflict issue with compat.export_saved_model in TF2.1
Removed support for Python 2.x

TensorFlowOnSpark - v2.1.3

Published by leewyang over 4 years ago

Detect TF version w/o importing to avoid runtime initialization before GPU allocation.

TensorFlowOnSpark - v2.1.2

Published by leewyang almost 5 years ago

Use tf.config.list_physical_devices() to avoid TF runtime initialization.

TensorFlowOnSpark - v2.1.1

Published by leewyang almost 5 years ago

added compat.is_gpu_available() method to use:
- tf.config.list_logical_devices('GPU') (for TF2.1)
- tf.test.is_cuda_available() (for earlier versions of TF).
added ability to launch TensorBoard on chief:0 or master:0 nodes (for small clusters without worker nodes).

TensorFlowOnSpark - v2.1.0

Published by leewyang almost 5 years ago

Added compat module to manage minor API changes in TensorFlow.
Added compatibility for TF2.1.0rc0 (exporting saved_models and configuring auto-shard policy)
Re-introduced compatibility for TF1.x (except support for InputMode.TENSORFLOW in the ML Pipeline API).
Added TFParallel class for parallelized single-node inferencing via Spark executors.
Updated examples for TF API changes.
Updated to use module-level loggers.

TensorFlowOnSpark - v2.0.0

Published by leewyang about 5 years ago

initial release compatible with TensorFlow 2.x.
API changes:
- removed TFNode.start_cluster_server, which is not required for tf.keras and tf.estimator.
- removed TFNode.export_saved_model, which can be replaced by TF native APIs now.
- added TFNodeContext.num_workers to count master, chief, and worker nodes.
Spark ML Pipeline API changes:
- Scala API has been removed for now, since the Java library for TensorFlow 2.0 is not available yet.
- removed InputMode.TENSORFLOW support for ML Pipelines, since the input data is always a Spark DataFrame for this API.
- added HasMasterNode and HasGraceSecs params.
- removed optional export_fn argument for Spark ML TFEstimator (use TF export APIs instead).
- added standard default values for signature_def_key and tag_set for Spark ML TFModel.
- modified inferencing code in TFModel for TF2.x APIs.
older TF 1.x examples have been replaced with TF 2.x compatible examples.

TensorFlowOnSpark - v1.4.4

Published by leewyang about 5 years ago

last expected release compatible with TensorFlow 1.x (aside from any critical fixes), since the master branch will be moving to TF 2.0 compatibility.
handle multiple outputs with signaturedef (thanks to @markromedia).
handle exceptions after data feeding.
moved API docs to sphinx_rtd_theme.
updated to Spark 2.4.4.

TensorFlowOnSpark - v1.4.3

Published by leewyang over 5 years ago

removed tensorflow as a dependency, in order to support other variants like tensorflow-gpu or tf-nightly.
allow use of evaluator node type in cluster (thanks to @bbshetty)
refactored cluster template generation.
updated wide-deep example to use models/official code.
restore termination of feed in mnist/spark example.
updated sample notebook instructions.
updated to use Spark 2.3.3.

TensorFlowOnSpark - v1.4.2

Published by leewyang over 5 years ago

Set TF_CONFIG for "chief" clusters (required by DistributionStrategy APIs)
Fix GPU allocation for multi-gpu nodes
Updated examples for MNIST
Updated Hadoop and Spark dependency versions

TensorFlowOnSpark - v1.4.1

Published by leewyang almost 6 years ago

Added util.single_node_env(), which can be used to initialize the environment (HDFS compatibility + GPU allocation) for running a single-node instance of TensorFlow on the Spark driver.
Added an example of parallelized inferencing from a pre-trained SavedModel.

TensorFlowOnSpark - v1.4.0

Published by leewyang almost 6 years ago

More deterministic GPU allocation for multi-GPU nodes.
Added timeout argument to TFCluster.shutdown() (default is 3 days). This is intended to shutdown the Spark application in the event that any of the TF nodes hang for any reason. Set to -1 to disable timeout.
Added ability to start reservation server on a specific port (contributed by @AvihayTsayeg).
Updated pipeline API for latest TF APIs (contributed by @AvihayTsayeg)
Added unit test for tf.SparseTensor support.
Updated examples to latest TF APIs (including workaround for https://github.com/tensorflow/tensorflow/issues/21745).
Updated Spark version dependency for Scala Inferencing API.
Added __version__ to module.

TensorFlowOnSpark - v1.3.4

Published by leewyang about 6 years ago

Travis CI integration for Python documentation and Scala Inferencing API builds.
Added sys.path to tensorboard search path.

TensorFlowOnSpark - v1.3.3

Published by leewyang about 6 years ago

Only set TF_CONFIG environment variable if cluster_spec has a "master", i.e. when using tf.estimator.
Updated mnist/keras/mnist_mlp_estimator.py with example of distributed/parallel inferencing via estimator.predict.
Added optional feed_timeout argument to TFCluster.train() for InputMode.SPARK.
Added optional grace_secs argument to TFCluster.shutdown().
Workaround for firewall proxy issue with get_ip_address (contributed by @viplav).
Add support for all Hadoop-compatible File System schemes (contributed by @vishnu2kmohan).
Added error messages to assert statements.
Initial Travis CI integration.

TensorFlowOnSpark - v1.3.2

Published by leewyang over 6 years ago

add grace period to TFCluster.shutdown()
add wide & deep example (contributed by @crafet)
update mnist/pipeline examples to tf.data, add instructions, and misc code cleanup (from @yileic)
parameterize versions in pom.xml and code cleanup (from @tmielika)
update Scala Inferencing pom.xml to latest tensorflow-hadoop artifact (contributed by @psuszyns)