enterprise_gateway

Jupyter Enterprise Gateway release 2.0.0 officially introduces support for Kubernetes! 🎉

This includes support for kernel-based images consisting of:

• Python (IPython), Python w/ Spark, Python w/ Tensorflow, and Python w/ Tensorflow and GPUs
• Scala (Toree) w/ Spark
• R (IRKernel), R w/ Spark

Other items included in this release:

• Support for Docker and Docker Swarm
• Helm chart for Kubernetes, Compose file for Docker
• Spark 2.4 support
• Dask-Yarn support
• Gateway Client (experimental) - demonstrates how to use Enterprise Gateway programmatically
• Jupyter Hub integration (configuration)
• Drop support for Python 2.x
• Kernel images based on Jupyter Docker-stacks
• Support for multiple YARN resource managers (HA)
• Bug fixes

Many thanks to the following contributors:

• Alan Chin (@akchinSTC)
• Alexey (@xhumanoid)
• Andrew Otto (@ottomata)
• Atin Sood
• Charlie Yang (@charlieeeeeee)
• Chia-liang Kao (@clkao)
• Dan Helfman (@witten)
• Dmitri Khokhlov (@dkhokhlov)
• Eric Charles (@echarles)
• Eric Dill (@ericdill)
• Eunsoo Park (@esevan)
• Grigoriy (@lamOrigin007)
• gsaibhar
• Hagen Hoferichter (@h4gen)
• IMAM KHURSHEED (@IMAM9AIS)
• Inki Hwang (@fisache)
• Jim Crist (@jcrist)
• Kevin Bates (@kevin-bates)
• Kuisong Tong (@ktong)
• Luciano Resende (@lresende)
• Matt Todd (@mattjtodd)
• Michał Zimniewicz (@michzimny)
• Naresh Sankapelly (@nareshsankapelly)
• Philip Jones (@philip-p-jones)
• Ray Yamamoto Hilton (@rayh)
• Ron Wu (@ronnnwu)
• Solaris (@SolarisYan)
• Todd (@toddrme2178)
• Troy Flagg
• sudheer0553 (@sudheer0553)
• Vikas Garg (@vikasgarg1996)
• Zain Kabani (@zain-kabani)

Jupyter Enterprise Gateway release 2.0.0rc2 officially introduces support for Kubernetes! 🎉

This includes support for kernel-based images consisting of:

• Python (IPython), Python w/ Spark, Python w/ Tensorflow, and Python w/ Tensorflow and GPUs
• Scala (Toree) w/ Spark
• R (IRKernel), R w/ Spark

Other items included in this release:

• Support for Docker and Docker Swarm
• Helm chart for Kubernetes, Compose file for Docker
• Spark 2.4 support
• Dask-Yarn support
• Gateway Client (experimental) - demonstrates how to use Enterprise Gateway programmatically
• Jupyter Hub integration (configuration)
• Drop support for Python 2.x
• Kernel images based on Jupyter Docker-stacks
• Support for multiple YARN resource managers (HA)
• Bug fixes

Thanks to the following contributors:

• Alan Chin (@akchinSTC)
• Alexey (@xhumanoid)
• Andrew Otto (@ottomata)
• Atin Sood
• Charlie Yang (@charlieeeeeee)
• Dan Helfman (@witten)
• Eric Charles (@echarles)
• Eric Dill (@ericdill)
• Eunsoo Park (@esevan)
• Grigoriy (@lamOrigin007)
• gsaibhar
• Hagen Hoferichter (@h4gen)
• Inki Hwang (@fisache)
• Jim Crist (@jcrist)
• Kevin Bates (@kevin-bates)
• Kuisong Tong (@ktong)
• Luciano Resende (@lresende)
• Matt Todd (@mattjtodd)
• Michał Zimniewicz (@michzimny)
• Philip Jones (@philip-p-jones)
• Ray Yamamoto Hilton (@rayh)
• Ron Wu (@ronnnwu)
• Solaris (@SolarisYan)
• Todd (@toddrme2178)
• Troy Flagg
• sudheer0553 (@sudheer0553)
• Zain Kabani (@zain-kabani)

Jupyter Enterprise Gateway 2.0.0rc1

Jupyter Enterprise Gateway release 2.0.0rc1 officially introduces support for Kubernetes! 🎉

This includes support for kernel-based images consisting of:

• Python (IPython), Python w/ Spark, Python w/ Tensorflow, and Python w/ Tensorflow and GPUs
• Scala (Toree) w/ Spark
• R (IRKernel), R w/ Spark

Other items included in this release:

• Support for Docker and Docker Swarm
• Spark 2.4 support
• Dask-Yarn support
• Gateway Client (experimental) - demonstrates how to use Enterprise Gateway programmatically
• Jupyter Hub integration (configuration)
• Drop support for Python 2.x
• Kernel images based on Jupyter Docker-stacks

The Jupyter Enterprise Gateway 1.2.0 release provides some dependency updates and multiple bug fixes as listed below:

• Tolerance to Tornado 6.x
• Update to Jupyter Kernel Gateway 2.3.0
• Update to Apache Toree 0.3.0
• Update to Yarn Client API 0.3.3
• Tolerate errors when closing response sockets
• Properly use provided YARN RM port
• Reset watch-activity callback on remote-proxy restarts

Jupyter Enterprise Gateway 2.0.0-beta.1

Jupyter Enterprise Gateway release 2.0.0-beta.1 officially introduces support for Kubernetes! 🎉

• This includes support for kernel-based images consisting of:
  • Python (IPython), Python w/ Spark, Python w/ Tensorflow, and Python w/ Tensorflow and GPUs
  • Scala (Toree) w/ Spark
  • R (IRKernel), R w/ Spark

Other items included in this release:

• Support for Docker and Docker Swarm
• Spark 2.4 support
• Dask-Yarn support
• Gateway Client (experimental) - demonstrates how to use Enterprise Gateway directly
• Jupyter Hub integration (configuration)

Thanks to the following contributors:

• Alan Chin (@akchinSTC)
• Alexey (@xhumanoid)
• Andrew Otto (@ottomata)
• Atin Sood
• Charlie Yang (@charlieeeeeee)
• Eric Dill (@ericdill)
• Grigoriy (@lamOrigin007)
• Hagen Hoferichter (@h4gen)
• Inki Hwang (@fisache)
• Jim Crist (@jcrist)
• Kevin Bates (@kevin-bates)
• Kuisong Tong (@ktong)
• Luciano Resende (@lresende)
• Ray Yamamoto Hilton (@rayh)
• Ron Wu (@ronnnwu)
• Solaris (@SolarisYan)
• Todd (@toddrme2178)
• Troy Flagg
• sudheer0553 (@sudheer0553)
• Zain Kabani (@zain-kabani)

This release incorporates the following commits:

• Properly set PYTHONUSERBASE relative to user home folder
• Fix max-kernels-per-user when KERNEL_USERNAME not set
• Avoid use of backslash operator when working with JSON

The Enterprise Gateway 1.1.0 release provides enhanced security support adding support for YARN endpoints protected by SPNEGO and other bug fixes as listed below:

• Add support to YARN endpoints protected by SPNEGO
• Allow enterprise gateway to ignore the SIGHUP signal
• Add the Enterprise Gateway version stamp to the startup message
• Update Scala and SBT dependencies to the latest version

Thanks @tf3193 for helping out with the fix for SIGHUP.

The Enterprise Gateway 1.0.2 release provides minor bug fixes around incompatible changes on 3rd party dependencies.

• Use yarn-api-client release version below 0.3.0 to avoid issues with incompatible API changes.

• Fixes issue seen in Python 3 environments to detect launch failures (#436)
• Adds optimization for using DistributedProcessProxy on local servers (#430)

Thanks to @ziedbouf for helping out with the python 3 issue.

Jupyter Enterprise Gateway 1.0.0 builds on its 0.9.4 release with various issue resolutions, but, more importantly, marks an important milestone for the project itself. With this release, Enterprise Gateway 1.0.0 officially supports Hadoop YARN and IBM Spectrum Conductor resource managers, enabling remote kernel distribution across a managed cluster by leveraging its pluggable process proxy framework. In addition, general ssh-based distribution can be accomplished via the DistributedProcessProxy plugin.

Here are some of the additions since our 0.9.4 release:

• Code coverage monitoring #415
• Bundle Toree assembly jar into Kernelspec distro #411
• Update Apache Toree dependencies to Release 0.2.0-incubating #412
• Update JKG dependency to 2.1.0 release #410
• Detect launch failures prior to timeout #404
• Update java build in yarn-spark image #391
• Update project readme file #388
• UTF-8 decode stdout from curl subprocess in YarnClusterProcessProxy #379

Thank you to the following contributors since 0.9.4:

• Alan Chin (@akchinSTC)
• Andrew Otto (@ottomata)
• Kevin Bates (@kevin-bates)
• Luciano Resende (@lresende)

• Minor bug fix for IBM Spectrum Conductor #396
• Allow access from remote hosts (Notebook 5.6.0 change) #386

Minor bug fixes - primarily targeting IBM Spectrum Conductor.

Jupyter Enterprise Gateway release v0.9.2 focuses on minor bug fixes and test automation and includes the following:

• Honor PYSPARK_PYTHON during R Kernel launches
• Add unittests specific to enterprise gateway
• Add authorization integration tests

See the v0.9.2 milestone on GitHub for a complete list of pull requests involved in this release.

Jupyter Enterprise Gateway release v0.9.1 focuses on minor bug fixes and test automation and includes the following:

• Address issues with kernel interrupt in local mode
• Fix Apache Toree dependency location required for Scala kernel launcher
• Add integration tests as part of the regular build
  • Basic kernel tests for multiple kernel manager mode
  • Basic runtime tests

Thanks to the following contributors:

• Alan Chin (akchinSTC)
• Kevin Bates (kevin-bates)
• Luciano Resende (lresende)

See the v0.9.1 milestone on GitHub for a complete list of pull requests involved in this release.

Jupyter Enterprise Gateway release v0.9.0 focuses on stabilization and usability and includes the following:

• Support for IBM Spectrum Conductor
• Support restriction of number of kernels per user
• Support port-ranges for kernels and gateway
• Support blacklisting specific network class
• Usability enhancements around error message and docker images
• Spark context initialization modes
• Address known resource leaks in jupyter stack

Thanks to the following contributors:

• Alan Chin (akchinSTC)
• Charlie Yang (charlieeeeeee)
• Kevin Bates (kevin-bates)
• Luciano Resende (lresende)
• Sanjay Saxena (sanjay-saxena)

See the v0.9 milestone on GitHub for a complete list of pull requests involved in this release.

Jupyter Enterprise Gateway release v0.8.0 focuses on security-related updates and includes the following:

• Add initial authorization and impersonation support (#235)
• Add JupyterLab option to elyra/nb2kg image. (#237)
• Connection info not returned from launchers in Python 3 (#241)
• Add proxy configuration to kernelspec (#240)
• Integration test enhancement (#245)
• First version implementation of impersonation integration test (#246)
• Fix R-kernel to show kernel ID as app name in YARN RM (#248)
• Fix SSH tunnel termination on kernel shutdown (#244)
• Validate cell execution results immediately after execution (#249)
• Fix signal_kernel() to nolonger assume its only used for interrupts (#252)
• Update NB2KG references to reflect recent repo move and release. (#255)
• Apply recent Toree kernel command-line additions to scala sample configurations (#231)
• Updated YARN cluster mode kernelspecs to use new spark conf parameter (#256)
• Encrypt handshake between the Kernel Launchers and the EG (#266)
• Initial changes of itests to use pytest (#268)
• Add support for kernel interrupts for R kernels (#269)
• Use UTF-8 encoding while encrypting and decrypting (#273)
• Limit the dependency on Tornado to versions less than 5.0 (#276)
• Update troubleshooting guide for pycrypto install (#275)
• Add a trouble shooting section for Kerberos authentication (#284)
• Add support for specifying port range to use for kernel and EG ports (#270)

Thanks to the following contributors:

• Alex Bozarth (ajbozarth)
• Alan Chin (akchinSTC)
• Charlie Yang (charlieeeeeee)
• Kevin Bates (kevin-bates)
• Kun Liu (liukun1016)
• Luciano Resende (lresende)
• Sanjay Saxena (sanjay-saxena)

See the v0.8 milestone on GitHub for a complete list of pull requests involved in this release.

Jupyter Enterprise Gateway release v0.7.0 is primarily a stabilization release that includes the following feature updates:

• Support for SSH tunneling between the Enterprise Gateway server and remote kernels. Currently disabled by default, this option can be enabled via env EG_ENABLE_TUNNELING=True. We look forward to making SSH tunneling the default behavior in v0.8.0.
• Creation of docker images for Jupyter + NB2KG, Enterprise Gateway and a Spark YARN base image to assist with gateway development. Visit our Elyra organization on docker hub (https://hub.docker.com/u/elyra/dashboard/) as well as our git repo for more information.
• Added the ability to interrupt Python and Scala kernels running remotely as different users. Support for R in this capacity is pending.
• Incorporated both the scala launcher code and integration tests into the enterprise-gateway repo - previously externally maintained.
• Added support for deploying releases to PyPi and Conda-forge frameworks.
• Refactored documentation increasing its readability. Check it out at http://jupyter-enterprise-gateway.readthedocs.io/en/latest/

Jupyter Enterprise Gateway 0.6.0 release