sagemaker-ssh-helper

New in v2.2.0:

• (HOT!) Introducing FQDN scheme for SSH by host name ssh xxx.yyy.sagemaker. See ssh_config section in README. Closes #33 for smoothly switching instances.
• (HOT!) UI: Introducing sm-ssh CLI tool with connect command as the alternative to sm-local-ssh-* and the list command to show all instances that you can connect to. Moving closer to pure Python implementation as mentioned in #38. See sm-ssh section in README.
• UI: New script sm-ssh-deregister-instances to be used in place of deregister_old_instances_from_ssm.py now as mentioned in the FAQ.md.
• API: New method print_ssh_info() for the wrappers to print the useful links and commands for connection
• Added fail-on-error options to all setup scripts, resulting stronger error handling that helps with troubleshooting of SSH Helper issues. Discussed in #37.
• Improved testing for IDE notebook through Selenium and Firefox. Introduced the class for SageMaker Studio automation (can be used to test your own notebooks, too) – sagemaker_ssh_helper/browser_automation.py.
• FAQ: How to improve performance of SageMaker Studio when using remote interpreter?
• Fixed Windows installation and updated instructions in FAQ.md, as discussed in #38.
• Improved logic for Low GPU utilization Lambda described in FAQ.md.
• Improved logging
• Resource monitoring in tests with pytest-monitor
• Introducing timedelta() syntax instead of counting seconds in API.
• MXNet SageMaker Studio kernel image support
• Refined the docs

New in v2.1.0:

Set your SageMaker Studio domain ID and user profile name, to allow multiple users connecting to multiple domains with the same kernel gateway name:

sm-local-ssh-ide set-domain-id <<domain_id>>
sm-local-ssh-ide set-user-profile-name <<user_profile_name>>

IAM action ssm:DeregisterManagedInstance removed from SSHSageMakerClientPolicy, because instance cleanup is to be run by an admin.

#32 - switch to regional STS endpoints in wrappers, to fetch the local user ID.

#35 - Support HF accelerate and DeepSpeed for inference.

#27 - Support Hugging Face Accelerate for training.

Support SSH into training with SageMaker Training Compiler.

Support for SSH into Jupyter Server instance.
FAQ: Can I connect from my local machine to Jupyter Server in addition to Kernel Gateways?

Improved support for SSH Agent keys forwarding into SageMaker.
FAQ: How do I securely forward my local private SSH keys from local machine to the remote host with SSH Agent?

API: get_instance_ids() method receives an optional timeout_in_sec parameter that is by default 900 seconds (15 minutes), to highlight the difference from connection_wait_time_seconds passed to the wrapper, which is by default 600 seconds (10 minutes).

Added development journey to README.

Improved logging.

Improved troubleshooting instructions.

New in v2.0.0:

Internet-free mode support: I'm running SageMaker in a VPC. Do I need to make extra configuration?

Amazon Linux support for SageMaker Studio

MATE desktop is now the default environment for VNC both for Amazon Linux and Debian-based SageMaker Studio images

Windows procedure now uses virtual env and doesn't need admin rights

Documentation improvements, more detailed architecture diagrams

Improve CI/CD testing of the code

New in v1.12.0:

#31 - fixed sm-local-configure on macOS

FAQ:

• When debugging a training job with SageMaker SSH Helper and train_placeholder.py, I want to automatically stop the job when there are no users connected and there's no GPU utilization. How to do that?
• I want to send users the SMS or email notification when the placeholder training job has issues with low GPU utilization. How to do that?

sm-wait stop now have to be used instead of pkill to gracefully shut down the waiting loop

IAM policy SSHSageMakerServerPolicy changes – ssm:ListInstanceAssociations requires * as a resource name to avoid errors in logs (see IAM_SSM_Setup.md for more details).

CDK apps has been split (see IAM_SSM_Setup.md for more details):

• sagemaker_ssh_helper.cdk.iam_ssm_app
• sagemaker_ssh_helper.cdk.advanced_tier_app

API Changes:

• ssh_wrapper.training_job_name() instead of ssh_wrapper.latest_training_job_name()
• sm-ssh-ide get-metadata to print SageMaker Studio details to connect
• sm-ssh-ide set-local-user-id "$LOCAL_USER_ID" and sm-ssh-ide init-ssm instead of a single init-ssm command.

Added the tip to set the Web VNC screen resolution with xrandr

sm-save-env now handles special symbols and quotes in variables

Improve CI/CD testing of the code

New in v1.11.0:

IAM/SSM policy update - needs ssm:GetCommandInvocation permission for better performance – 2628c4f
IAM/SSM policy update - needs * for ssm:ListInstanceAssociations to avoid errors in logs – 049f97b

#28 – fixed issue with new SageMaker Python SDK: Invalid bucket name "sagemaker.config INFO - Fetched defaults config from location: ": Bucket name must match the regex. The change related to SageMaker defaults :
https://github.com/aws/sagemaker-python-sdk/pull/3872/files

Added support for SageMaker defaults
Improved MME support for TF
Added support for pre-trained HuggingFace models
Better support for popular SageMaker Studio images
Support attaching to already submitted training jobs with SSHEstimatorWrapper.attach()

Documented alternative way to add SSH Helper dependency through requirements.txt

New in v1.10.4:

#25 - fix for IDE notebook / kernel lifecycle configuration script: installing sudo that is missing in some kernels

New in v1.10.3:

WebVNC through noVNC and AWS Jupyter Proxy

#25 - fix for sm-local-configure on Linux: sudo for installing AWS CLI and Session Manager Plugin

New in v1.10.2:

#21 - don't update the pre-installed pip in SageMaker Studio that can cause issues with PyYAML

The sm-local-ssh-ide script now supports extra SSH args for forwarding additional ports:

sm-local-ssh-ide connect <kernel_gateway_name> [--ssh-only] [<extra_ssh_args>]

The new method is_last_session_timeout() to prevent unused resources for the placeholder job if there's were no SSM or SSH sessions for the specified period of time.

Documentation updates

New train_placeholder.py script that ends when sessions timeout.

Added section on Forwarding TCP ports over SSH tunnel - useful to access remote apps like Dask or Streamlit

New in v1.10.1:

#20 - fixing location of SSH authorized keys to prevent sm-local-ssh-ide script from asking root password

Simplified CDK app deployment step in IAM_SSM_Setup.md (no need to clone the source code repo anymore)

Documentation updates

Simplified procedure with instructions for ~/.ssh/config

FAQ - I'm running SageMaker in a VPC. Do I need to make extra configuration?

FAQ - I'm using boto3 Python SDK instead of SageMaker Python SDK, how can I use SageMaker SSH Helper?

New in v1.10.0:

CDK deployment automation in IAM_SSM_Setup.md

Demonstrated the least privilege principle

#12 - Removed EC2 instance from SSM setup

Speed up instance ID resolution with SSMManager (not using CloudWatch logs and SSHLog anymore, except for endpoints)

Speed up instance ID resolution (not using CloudWatch logs anymore, except for endpoints)

#4 - Notebook instances support

The command sm-local-ssh-ide <<kernel_gateway_name>> is becoming sm-local-ssh-ide connect <<kernel_gateway_name>> (added connect for consitency with other scripts)

New command: sm-local-configure to run on the local machine to install AWS CLI v2 and Sessions Manager plugin

#17 - An option to start only SSH server inside SageMaker Studio: sm-ssh-ide start --ssh-only

Deregistering instances with timestamp: new parameter --delete-older-than-n-days <N>

New tags attached to an SSM instance, in addition to SSHOwner: SSHCreator, SSHTimestamp, SSHResourceName and SSHResourceArn.

Env variables passed to SSH helper change: instead of SSH_SSM_TAGS it now accepts only SSH_OWNER_TAG, other tags are calculated automatically. SSH_LOG_TO_STDOUT parameter is needed for notebook instances.

#16 - China AWS Regions support

Stability, usability and performance improvements

Documentation updates

FAQ - Are SageMaker notebook instances supported?

FAQ - What if I want to use an estimator in a hyperparameter tuning job (HPO) and connect to a stuck training job with SSM?

FAQ - How to configure an AWS CLI profile to work with SageMaker SSH Helper? (resolves #14)

New in v1.9.1:

Support connecting to a different AWS Region by providing region_name in boto3 session and in AWS_REGION / AWS_DEFAULT_REGION env vars in CLI.

Stability fixes.

Documentation updates

FAQ - How to troubleshoot jobs that are failing with the exception or error?
FAQ - I see folders like Desktop, Documents, Downloads, Pictures in SageMaker Studio, is it fine?
FAQ - How do I automate my pipeline with SageMaker SSH Helper end-to-end?

New in v1.9.0:

• Added support for batch transform - #10

• Fixed SKLearn support - #2

• Added support for basic estimator and improved support for other frameworks, including HuggingFace, TensorFlow and XGBoost.

• Added MultiDataModel support without a model object - #4

• Fix the deregister_old_instances_from_ssm.py to allow clean up instances with any role, not only those that contain ‘sagemaker’ - #8

• Possibility to override the public keys location path in S3 with SSH_AUTHORIZED_KEYS_PATH environment variable - #8

• Additional sm-local-ssh-* scripts to connect with SSH to processing, training, inference and batch transform containers.

• Fixing issue with sudo installation in some containers.

• Better support for the distributed multi-GPU scenarios.

New in v1.8.1:

• Improved logging (less noise)
• Fix minor issue with missing jq util in older Ubuntu versions
• IDE / SageMaker Studio scenario improvements:
  • Adding lifecycle configuration script support for SageMaker Studio
  • Refine IDE notebook
  • Make it possible to override IDE settings by per-user ~/. files
  • Update docs on switching IDE instances
  • Make bootstrap procedure smoother – not need to stop old instance when starting a new one, the whole notebook can be restarted as many times as needed

New in v1.8.0:

• Install the latest stable version from PyPI instead of downloading source code - pip install sagemaker-ssh-helper .
• Script to cleanup stale SSM instances – [sagemaker_ssh_helper/deregister_old_instances_from_ssm.py] (sagemaker_ssh_helper/deregister_old_instances_from_ssm.py) . Will remove all offline instances created by SageMaker with SSHOwner tag
• Limit the number of training instances for distributed job with an optional parameter ssh_instance_count passed to SSHEstimatorWrapper.create() (default is 2) – useful to prevent Throttling API errors in SSM
• Added FAQ
• Windows support for local machine (more details in FAQ)

Initial public release