training-operator

Distributed ML Training and Fine-Tuning on Kubernetes

APACHE-2.0 License

Downloads
15.6K
Stars
1.5K
Committers
179
View Code on GitHub Visit Website
Ecosystems: Kubernetes, Python, PyTorch, Tensorflow

Bot releases are hidden (Show)

training-operator - v1.0.1-rc.5

Published by gaocegege over 3 years ago

training-operator - v1.0.1-rc.4

Published by gaocegege over 3 years ago

v1.0.1-rc.4 (2021-02-04)

Full Changelog

Closed issues:

  • I have some questions about the function createNewPod in pkg/controller.v1/tensorflow/pod.go #1221

Merged pull requests:

training-operator - v1.0.1-rc.3

Published by gaocegege over 3 years ago

v1.0.1-rc.3 (2021-01-27)

Full Changelog

Closed issues:

  • Error with release tag v1.0.1 "invalid memory address or nil pointer dereference" #1223

Merged pull requests:

  • feat(server): Use apiextension client to check if crd exists #1228 (gaocegege)
training-operator - v1.0.1-rc.2

Published by gaocegege over 3 years ago

v1.0.1-rc.2 (2021-01-27)

Full Changelog

Merged pull requests:

training-operator - v1.0.1-rc.1

Published by gaocegege over 3 years ago

v1.0.1-rc.1 (2021-01-18)

Full Changelog

Closed issues:

  • checkCRDExists func return true when k8s cluster is not connected #1206
  • How to install it without kubeflow #1195
  • Pod get re-created after it exited and get garbage collected #1186
  • Surface Pod and other Errors that Prevent TFJob from starting #1131
  • Jobs failing when a node is preempted #999

Merged pull requests:

training-operator - v1.0.1-rc.0

Published by gaocegege almost 4 years ago

v1.0.1-rc.0 (2020-12-22)

Full Changelog

Closed issues:

  • tf-operator panic without worker role #1192
  • TFJob completion with active services/endpoints resources #1191
  • Having trouble viewing logs using Kubernetes dashboard #1189
  • [feature] Support SuccessPolicy/FailurePolicy Based on % of Succeeded/Failed Workers #1188
  • TFJob cannot utilize GPUs in the node. #1184
  • [bug] With Python SDK, TFJob won't stop running #1183
  • [bug] [Python SDK] tfjob_client.get_logs broken #1182
  • How to create a python sdk for mxnet-operator #1181
  • [feature] python sdk should report errors in created TFJobs #1180
  • Could not introduce k8s.io/kube-openapi@master #1174
  • can tf-operator used in distribute scene, such as Multi-node #1173
  • Multi-worker training with Keras only use one GPU #1169
  • NCCL WARN Failed to open libibverbs.so[.1] #1168
  • tf-job-operator pod restarts #1167
  • swagger-codegen-cli-2.4.6.jar not found #1166
  • Cut release for tf-operator project #1163
  • Replace reconciler implementation with kubeflow/common JobController #1161
  • Error while replicating mnist_with_summaries #1159
  • Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: out of memory #1158
  • TFjob pods hang without explanation #1156
  • [Proposal] Support ClusterSpec Propagation Feature in TF 1.14 #1141
  • evaluator should be set in TF_CONFIG when using Estimator distribute strategy #1139
  • Is there any case to run the different command in tfReplicaSpecs? #1138
  • should gpu resource be released when tfjob failed because of image pull problem? #1136
  • tf-job-operator CrashLoopBackOff #1135
  • How to change the log level of tf-job-operator #1132
  • Support getting the training process via Python SDK #1129
  • Popgroup is not created automatically. #1121
  • TFConfig should be demonstrated more specifically. #1115
  • [chore] Remove tfjob dashboard #1113
  • read TF_CONFIG env from configMap #1112
  • Long job names result in jobs stuck forever #1101
  • [Question] can't the base image "registry.access.redhat.com/ubi8/ubi:latest" in Dockerfile be replaced with "debian:buster" ? #1099
  • can i install tf-operator alone without kubeflow? #1096
  • c #1095
  • TFJob test is failing on master and v0.7 branch for kubeflow/kubeflow #1094
  • TFJob tests should use pytest #1093
  • Multiple Evaluator replicas gives InvalidTFJobSpec #1091
  • Java client for current version of TFjob #1090
  • [enhancement] Replace common with kubeflow/common #1087
  • Lack of documents for deployment #1086
  • Performance problem about pod informer #1079
  • [bug] Cannot initialize the training job with TF Estimator when the user uses 1 worker and 0 PS #1078
  • Separate cluster scoped and namespace scoped resources #1077
  • TFJob 1.0 #1076
  • [bug] Keep tf-job-role as deprecated label in this version #1068
  • GenLabels may select wrong Pods #1066
  • Can I create a tf-operator pod without using GO? #1065
  • tf-job-dashboard cannot work #1060
  • [discussion] Should We Add CleanPodPolicy PS? #1059
  • Refactor dockerfile #1058
  • remove v1beta1 in v0.5.3 cause incompatible issue when using go mod #1057
  • Invalid value: "v1beta1": must appear in spec.versions #1056
  • Example on EKS: Device or resource busy #1053
  • can we add PriorityClassName when we create TF-job Podgroup? #1048
  • TFjob still running while chief pod is completed #1045
  • Is there any document for how to run TFJob in AllReduce Strategy #1039
  • tf-operator version conficts #1035
  • Add E2E test for gang-scheduling #1033
  • gang schedule annotation #1031
  • [feature] Can we use one headless service for one job? #1030
  • Will tf-operator upgrading k8s to 1.13? #1029
  • no error log for create tfjob fail #1026
  • Creating tfjob in dashboard usability issues #1024
  • Deleting tf-job through the dashboard is not working #1019
  • Create common CRD validate and mutating webhook for all operator #1016
  • error with kubeflow instalation #996
  • Shall we consider upgrading k8s to 1.11.3 #985
  • TFJob Dashboard is not support pvc #980
  • ERROR handle object: patching object from cluster: merging object with existing state: unable to recognize "/var/folders/tl/zzfcr4zs53vgnpqqjq4n08sh0000gn/T/ksonnet-mergepatch020443124": no matches for kind "TFJob" in version "kubeflow.org/v1beta1" #976
  • Create CRD conversion webhook #967
  • Performance issue when there is a lot of completed jobs #965
  • Failed to marshal the object to TFJob; the spec is invalid: Failed to marshal the object to TFJob #964
  • Proposal for a Common Operator #960
  • Delete pod with unknown status in reconcilePods #956
  • Create distributed training example for TF 2.0 #953
  • Consider using KubeBuilder to reduce boilerplate code #925
  • e2e test for dashboard/backend/handler/api_handler.go #921
  • Use pod group instead of PDB for gang scheduling #916
  • shareProcessNamespace not working with TFJob #902
  • [feasibility-research] Handle machine failure #900
  • Should limit the size of logs of tf_operator container #888
  • Log message severity isn't properly reported in stackdriver #864
  • E2E test for invalid spec errors #810
  • [v1alpha2] Delete resources according to cleanuppolicy exactly once #804
  • refactor the code of TFJobController for unittest #757
  • e2e test for cleanupTFJob #756
  • [build] Replace Python with Make or Bazel #739
  • Export TF/Tensorboard/TF Summaries to prometheus #722
  • [discussion] Maintain Helm Chart #716
  • [discussion] Capacity planning #708
  • [v1alpha2] Generate CRD validation in Kubernetes 1.11 #622
  • Set labels and annotations for svc created by tf_operator #609
  • mnist test isn't part of CI #597
  • [v1alpha2] Push the example docker image to google or dockerhub registry #590
  • feat: use fake client-set and informer add controller unittest. #540
  • Run submit_release_job.sh in CI #519
  • Add environment name in ControllerConfig #450
  • [dashboard] How to handle storage? #449
  • [dashboard] GPU limits are not taken into account #448
  • [dashboard] Ability to create a TensorBoard instance #447
  • [examples] Add termination policy in examples/tf_job.yaml #438
  • add boilerplate header #430
  • [logging] Extra flag problem #427
  • [CI] Add hack/verify-codegen.sh in Travis CI #426
  • E2E workflows should ignore failures #423
  • [enhancement] Add OWNERS in subdirectories #415
  • [enhancement] Fix the warnings reported by goreportcard.com #394
  • [discussion] Separate the operator and UI dashboard #389
  • [enhancemnet] Separate release image and test image #385
  • [enhancement][CI] Replace Travis CI with Prow #382
  • use Python3 for all python code? #377
  • What to do about example TFJob YAML specs? #375
  • E2E test for non-default namespace #170
  • OpenAPI Client Generation for Java, Python #167
  • Prevent scheduling deadlocks #165
  • TfDebugger support #132
  • Refactor code in py into a proper python package #114
  • Update instructions and code to work with Kubernetes 1.8 #108
  • Build sample container as part of release process #81
  • Run lint (Python, Go) as a presubmit test #53
  • Optimize scheduling of TF Processes #35
  • E2E test that verifies invalid jobs are failed #30
  • E2E test(s) to verify that permanent and retryable errors are handled correctly. #29

Merged pull requests:

training-operator - v1.0.0-rc.0

Published by kunmingg over 5 years ago

tf-operator pre-graduation

training-operator - v0.5.3

Published by richardsliu over 5 years ago

training-operator - v0.5.2

Published by richardsliu over 5 years ago

training-operator - v0.5.1

Published by richardsliu over 5 years ago

training-operator - v0.5.0

Published by richardsliu over 5 years ago

training-operator -

Published by richardsliu over 5 years ago

training-operator - v0.4.0-rc.1

Published by richardsliu almost 6 years ago

training-operator - v0.4.0-rc.0

Published by richardsliu almost 6 years ago

Initial version of 0.4.0

  • TFJob v1beta1 API
training-operator - v0.3.0 Release

Published by jlewi about 6 years ago

The v0.3.0 release of the TFJob operator.

training-operator - v0.2.0-rc1

Published by kunmingg over 6 years ago

tf-operator release v0.2.0, part of Kubeflow release v0.2.0.

Features and improvements:

  • [v1alpha2] Set event for tfjob when spec is not valid #620
  • [enhancement] Fix the gofmt support #586
  • [go] Use dep instead of glide to reduce the size of vendor #556
  • [v1alpha2] Enhance the logic about sync #547
  • [v1alpha2] Use structured log #537
  • [log] investigate zap #534
  • [v1alpha2] Try to not to always claim pods #533
  • [v1alpha2] Suppport customized port #532
  • [v1alpha2] start using kubeconfig #522
  • v1alpha2 integration #521
  • TFJob operator surface queue metrics #503
  • [api] Remove pending pods from active pods #484
  • [enhancement] Set StartTime for TFJob status #475
  • [Feature] Support "eval" worker in tf-operator #444
  • Add appropriate logging fields to the tf-operator log messages #424
  • [enhancement] Refactor docs #379
  • Deprecate TfPort and set default port for users #327
  • [enhancement] Add e2e test cases for recorder #317
  • Make the TfJob controller more event driven #314
  • Potential data race, maybe #302
  • Don't leave pods running just to get logs #128
  • Add hyperparameter tuning? #112
  • Use headless services for Training jobs #40
  • More validation of TfJob #25

Fixed bugs:

  • [v1alpha2] RealServiceControl does not set owner reference #616
  • TfJob operator stops working on invalid spec #561
  • [v1alpha2]tfjob restartPolicy for Never #555
  • [v1alpha2] Potential bugs when there is one worker succeeded #538
  • [v1alpha2][test] Avoid potential data race problem #530
  • Phase is wrong unexpected TfJob phase: Done #110

Closed issues:

  • [v1alpha2] Make restart policy a pointer #692
  • [v1alpha2] Need conditions Succeeded and Failed indicating when job is done #673
  • [v1alpha2] add pod label with job name (without namespace) #672
  • [v1alpha2] Pods not deleted when job finishes #671
  • [v1alpha2] conditions not updated #668
  • [v1alpha2] Move control interface to separate pakckage #665
  • [v1alpha2] Move test util to separate package #664
  • Speedup E2E test by running build and setup cluster in parallel #659
  • In TFjob, when the workers Completed, i want the ps Completed too, how can i do? #657
  • [v1alpha2] service names are prefixed with namespace #654
  • [v1alpha2] Create a simple python server to be used for E2E tests of controller behavior #653
  • dep ensure give warning on k8s.io/apiserver #647
  • [v1alpha2] pod names don't include random salt #644
  • [v1alpha2]Unable to create pod #641
  • GPU tests failing; ks env doesn't exist #640
  • TFJob not marked as success when master exits but not workers #634
  • v1alpha2 - pod names don't include replica type #633
  • tensorflow on kubernetes how to pass in worker_host and ps_host to container if I use tf-operator #630
  • tf_job_client blocks forever #606
  • [v1alpha2] Need to add the v1alpha2 binaries to our Docker image #600
  • [v1alpha2] Need ksonnet package #599
  • Support deploying v1alpha2 and v1alpha1 controllers simultaneously #598
  • [v1alpha2] Remove controller_utils.go #591
  • [v1alpha2] Add CI test #589
  • [question] dist_mnist example failed to run #588
  • can not set labels #580
  • v1alpha2 should use headless services #574
  • TFJob operator should pass through annotations to the pod #573
  • [test] Test failed because of ImagePullBackOff #567
  • Servable not found for request: Latest(mnist) #552
  • [v1alpha2] The state of distributed model training. #544
  • [test] copy labels and anotations to pod from tfjob #543
  • Unable to deploy the example TfJob in the user guide #535
  • [v1alpha2] Do not set default to always for restartpolicy #524
  • E2E test steps should exit with non zero exit code if test fails #514
  • [v1alpha2] Sync commits with v1alpha1 #490
  • Use OpenAPI validation for CRDs in k8s 1.9 #437
  • default install of kubeflow no longer install tf-job-dashboard #435
  • Use DAG functionality of Argo in our E2E tests #422
  • Post submits are failing with Argo #370
  • tf-job-operator pod hangs and doesn't restart if it can't delete one of the TfJob pods #366
  • Refactor TFJobStatus in CRD API #333
  • Deprecate the TfImage field #330
  • [discussion] Differences between tensorflow/k8s and caicloud/kubeflow-controller #283
  • Does TfJob controller need to do master election? #263
  • Setup Prow PR Dashboard #255
  • API: some comments about API changes from PR #215 review #249
  • e2e test for the case that the chief is not master #235
  • Use conditions instead of phase #223
  • Submitted tfjobs cease to start running under unknown conditions #203
  • Tutorials #195
  • Copy chart to kubernetes/charts #93
  • Create a web page to list releases #70
  • tensorflow 1.4 and estimator support #61
  • Set a default value for restartPolicy #55

Merged pull requests:

training-operator - Initial release of the TFJob operator

Published by jlewi over 6 years ago

gcr.io/kubeflow-images-staging/tf_operator@sha256:1a3d1a2ee90f0108fff3e29023228fc686afbfa311752e8b3bf71859d488b435

v0.1.0 (2018-03-29)

Closed issues:

  • [v1alpha2] Implement condition update #502
  • E2E tests timing out; job appears to remain in running state even though job is done. #500
  • [v1alpha2] TF_CONFIG should be configurable by user #499
  • [test] All log is 404 in argo #496
  • Presubmit shows succeeded, but some test actually failed. #479
  • Waiting pods start too long #461
  • [test] Add unit test for pkg/controller #455
  • Create a suitable OWNERS file in /dashboard #443
  • Tide is misconfigured for this repository. #433
  • CI failed to setup the cluster #420
  • [docs] Add dashboard readme #411
  • Make coverall results advisory and not report as failure #406
  • Presubmits failing due to lint #404
  • [enhancement] Fix go vet errors which not caught by the compilers #395
  • User facing website for Kubeflow that details how to choose a stack #371
  • [discussion] How to set clusterspec #369
  • [enhancement] Rename the cmd/tf_operator to cmd/tf-operator #363
  • Local releaser fails due to version_tag #360
  • Helm test failure not reported to gubernator #355
  • [discussion] Whether to create CRD in helm charts #353
  • Should resourcelock be in the same namespace as controller? #352
  • Helm test tf-job does not pass validation #351
  • Move tensorflow/k8s to kubeflow/tf-operator #350
  • Get rid of TensorBoard replica #347
  • Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs #346
  • Deprecate the ENV MY_POD_NAMESPACE and MY_POD_NAME #341
  • [feature] Does tfJob support setting different label/envVar for each worker(replicas >1)? #340
  • [Discussion] Time to start tagging releases for the TF operator? #339
  • [discussion] Should group name be tensorflow.org or kubeflow.io or kubeflow.org? #337
  • dashboard silient error during calling non-existent tfjob #335
  • in dashboard, silent error when nonexistent namespace is specified #334
  • Deprecate the IsDefaultPS field #329
  • [Convention] Replace Tf with TF in CRD #328
  • Standardise labels for issues and PRs #326
  • Manage Pods directly instead of using Job controllers #325
  • TfJobs dashboard not showing jobs #324
  • TfJobs dashboard doesn't work with K8s API server proxy or envoy proxy #323
  • Recreating a failed/successful job with same name doesn't work #322
  • Releaser incorrectly tags images as "dirty" #321
  • Reenable the releaser #320
  • E2E tests are not isolated #318
  • Need to mark prow job as failed if any tests fail #315
  • Remove outdated branch wbuchwalter-patch-1 #311
  • E2E test delete and recreate job with same name #310
  • TrainingJob.reconcile not called periodically #309
  • rename master to chief #306
  • Assign resource quota for TensorBoard #304
  • Jobs evicted for lack of memory, potentially add resource field to tf-job prototype #301
  • [Discussion] Operators vs. controller pattern #300
  • [bug] Add a default pod template for PS #297
  • Bunch of pylint error messages #294
  • Fix Head #293
  • Operator deployment fails post-v20180108-190394d #292
  • Promote last known good release #290
  • [bug] metadata.ownerReferences.apiVersion is not set #288
  • fail to run example job. invalid job spec: tfReplicaSpec.TfPort can''t be nil #284
  • [bug] Build log 404 in https://prow.k8s.io/?repo=tensorflow%2Fk8s #282
  • [feature] Seperate the CRD and controller #281
  • Gaps in test coverage #280
  • Regression in flag name: controller-config-file #279
  • [bug] glog before flag.Parse() #275
  • build new code to new image and find some problem #274
  • Fix the releaser so we can build new images #270
  • deploy.py gives gcloud api error '... Version "1.8.1-gke.1" is invalid.' #268
  • Pods terminated without waiting #267
  • Attach appropriate header (copyright) to go files #266
  • suppose i've install the tfjob in my k8s cluster #265
  • what's the folder pkg for? #264
  • Build failing because of lint issues #256
  • what's the main change between version 0.2 and version 0.3? #247
  • SetupCluster failures unexpected keyword argument 'client_configuration' #242
  • GPU test marked as succeeded but airflow step is failing #240
  • Use Kubeflow & ksonnet to install TfJob #239
  • tf_smoke.py distributed computing doesn't work on minikube #238
  • example-job can not work in private k8s cluster #233
  • Test failures aren't properly reported in Gubernator #229
  • [CRD] Request for input and output dirs in TFJobSpec #224
  • TfJob should be marked as failed if setup fails #218
  • panic: runtime error: invalid memory address or nil pointer dereference can not run in k8s 1.8.5 #212
  • Rethink the TFJob CRD #209
  • ksonnet configs for deploying the TfJob CRD & Controller #208
  • Make default TfImage configurable by users #207
  • refactor the TfJob to use Informer and Controller #206
  • Use Argo workflow engine for CI/CD or releases #205
  • Potential issue with Tensorboard / value of simple best-practices example with tboard #202
  • Investigate using buildah to build our images #201
  • E2E tests pre & postsubmits are failing #196
  • Publishing a client to pypi #193
  • Don't require a master or chief #192
  • Make cloning the repo and building the artifacts separate commands in py/release.py #189
  • Handle the case where grpcServerFilePath is the empty string #188
  • Make Airflow logs accessible #185
  • Complement docs for Python 3rd party dependencies #181
  • Helm Test fails because grpcServerFilePath is the empty string #179
  • Helm should only set --controller_config_file conditionally #175
  • Troubleshooting Guide: no matches for tensorflow.org/, Kind=TfJob #174
  • no matches for tensorflow.org/, Kind=TfJob #173
  • Failed to build TFOperator #171
  • E2E test for GPUs #164
  • TfJob doesn't work on minikube #160
  • Deleted jobs re-starting #156
  • Use coveralls.io to report and check code coverage #155
  • Clarify scope of tensorflow/k8s #150
  • After init helm, install chart failed #149
  • Helm test; insufficient permissions on RBAC clusters #135
  • Need to trim trailing slash of host string in TfJobRestClient.Watch() #130
  • results of lint test aren't reported in junit file used by gubernator #126
  • Collaborators need to be K8s members to trigger tests #122
  • Extend Test Infrastructure to run multiple E2E tests in parallel #120
  • initResource() failed; findAllTfJobs returned error: #118
  • Latest tag on gcr.io is not up to date #116
  • duplicate #115
  • postsubmit results aren't showing up in testrgrid #113
  • TensorBoard replica set not deleted when job deleted. #107
  • helm permission issue on 1.8.1 #106
  • Run python unittests as part of pre/post/periodic tests #101
  • E2E tests are failing #96
  • E2E Test log should capture output from helm-test #95
  • Rename TfJob kind to remove mlkube.io #89
  • Setup travis for tensorflow/k8s #88
  • Update repo to use its new location tensorflow/k8s #86
  • mlkube.io -> tensorflow/k8s #85
  • Update prow to use repo tensorflow/k8s #84
  • periodic test is failing #83
  • runner.py needs to create build-log.txt with stdout/stderr of test #82
  • E2E tests leaking GKE clusters #80
  • No results show up if you click on mlkube-build-periodic #76
  • No results show up in prow test grid for presubmit jobs #75
  • Include TfJob name in labels #72
  • Simplify/Clarify Accelerators config #71
  • Clean up examples; don't require cloning the repo #68
  • How to create TF Jobs from the user side? #67
  • Change version from beta -> alpha #65
  • API Review #64
  • Setup release process for CRD #63
  • Post submit jobs don't correctly upload artifacts to GCS #62
  • presubmit test(bootstrap.py) doesn't properly check out PRs #59
  • E2E Test for default PS server #58
  • UI / Kubernetes Dashboard Integration #57
  • E2E test for GPUs #54
  • Integrate with Prow for Continuous Testing #46
  • Consider how we manage replicas (stateful sets, managing pods directly) #45
  • Use K8s Garbage Collection #42
  • func c.findAllTfJobs() in controller.go will never reach #41
  • Rename project #34
  • Structured (Json) logging for Tf Processes #32
  • Permanent errors don't cause job failure #28
  • If handling Add event fails, TfJob should be marked as failed with appropriate error #26
  • Structured Logging For the operator #24
  • Operator Log Spam; replicas.go:287] No container named: tensorflow found for pod; assuming POD is running #23
  • Provide a default value for TfPort, replicas, and tfReplicaType #22
  • Setup continuous build of containers #19
  • Should this be converted to a Custom Resource Definition (CRD) in anticipation of 1.7 #17
  • Run TensorFlow server for parameter servers by default #16
  • TensorBoard Integration #13
  • Dependency management #7
  • Better GPU support #6
  • TfJobRestClient.Create doesn't set kind appropriately #5
  • Add a creationTimestamp #4

Merged pull requests:

Package Rankings
Top 4.64% on Pypi.org
Top 1.18% on Proxy.golang.org
Badges
Extracted from project README
Build Status Coverage Status Go Report Card
Related Projects
refinery-python-sdk

Official Python SDK for Kern AI refinery.

06 Jun 2021 18
torchx

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iter...

04 May 2021 328
skypilot

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availabilit...

11 Aug 2021 5,717
kopf

A Python framework to write Kubernetes operators in just a few lines of code

17 Aug 2020 1,974
pytorch-dist

08 Nov 2016 35
ignite

High-level library to help with training and evaluating neural networks in PyTorch flexibly and t...

23 Nov 2017 4,484
gradsflow-automl

An open-source AutoML Library based on PyTorch

11 Aug 2021 306
pytorch-lightning

Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.

31 Mar 2019 27,777
opyrator

🪄 Turns your machine learning code into microservices with web API, interactive GUI, and more.

06 Apr 2021 3,063