TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
APACHE-2.0 License
Bot releases are visible (Hide)
tensorflow-cpu
package.Published by leewyang over 3 years ago
Published by leewyang over 3 years ago
Published by leewyang almost 4 years ago
Published by leewyang over 4 years ago
TFOS_SERVER_PORT
environment variable.mnist/keras/mnist_tf.py
example with workaround for tensorflow datasets issue.executor_id
.Published by leewyang over 4 years ago
mnist_inference.py
examples (thanks to @qsbao)TFNode.start_cluster_server
in backwards-compatibility code for TF1.xcompat.export_saved_model
in TF2.1Published by leewyang over 4 years ago
Published by leewyang almost 5 years ago
tf.config.list_physical_devices()
to avoid TF runtime initialization.Published by leewyang almost 5 years ago
compat.is_gpu_available()
method to use:
tf.config.list_logical_devices('GPU')
(for TF2.1)tf.test.is_cuda_available()
(for earlier versions of TF).chief:0
or master:0
nodes (for small clusters without worker
nodes).Published by leewyang almost 5 years ago
compat
module to manage minor API changes in TensorFlow.Published by leewyang about 5 years ago
TFNode.start_cluster_server
, which is not required for tf.keras
and tf.estimator
.TFNode.export_saved_model
, which can be replaced by TF native APIs now.TFNodeContext.num_workers
to count master
, chief
, and worker
nodes.InputMode.TENSORFLOW
support for ML Pipelines, since the input data is always a Spark DataFrame for this API.HasMasterNode
and HasGraceSecs
params.export_fn
argument for Spark ML TFEstimator
(use TF export APIs instead).signature_def_key
and tag_set
for Spark ML TFModel
.TFModel
for TF2.x APIs.Published by leewyang about 5 years ago
master
branch will be moving to TF 2.0 compatibility.Published by leewyang over 5 years ago
tensorflow
as a dependency, in order to support other variants like tensorflow-gpu
or tf-nightly
.evaluator
node type in cluster (thanks to @bbshetty)Published by leewyang over 5 years ago
Published by leewyang almost 6 years ago
util.single_node_env()
, which can be used to initialize the environment (HDFS compatibility + GPU allocation) for running a single-node instance of TensorFlow on the Spark driver.Published by leewyang almost 6 years ago
timeout
argument to TFCluster.shutdown()
(default is 3 days). This is intended to shutdown the Spark application in the event that any of the TF nodes hang for any reason. Set to -1 to disable timeout.tf.SparseTensor
support.__version__
to module.Published by leewyang about 6 years ago
sys.path
to tensorboard search path.Published by leewyang about 6 years ago
tf.estimator
.mnist/keras/mnist_mlp_estimator.py
with example of distributed/parallel inferencing via estimator.predict
.feed_timeout
argument to TFCluster.train()
for InputMode.SPARK.grace_secs
argument to TFCluster.shutdown()
.get_ip_address
(contributed by @viplav).assert
statements.Published by leewyang over 6 years ago
TFCluster.shutdown()
tf.data
, add instructions, and misc code cleanup (from @yileic)Published by leewyang over 6 years ago
tf.keras
apis