A helper library to connect into Amazon SageMaker with AWS Systems Manager and SSH (Secure Shell)
MIT-0 License
New in v2.2.0:
ssh xxx.yyy.sagemaker
. See ssh_config section in README. Closes #33 for smoothly switching instances.sm-ssh
CLI tool with connect
command as the alternative to sm-local-ssh-*
and the list
command to show all instances that you can connect to. Moving closer to pure Python implementation as mentioned in #38. See sm-ssh section in README.sm-ssh-deregister-instances
to be used in place of deregister_old_instances_from_ssm.py
now as mentioned in the FAQ.md.print_ssh_info()
for the wrappers to print the useful links and commands for connectionpytest-monitor
timedelta()
syntax instead of counting seconds in API.Published by ivan-khvostishkov about 1 year ago
New in v2.1.0:
Set your SageMaker Studio domain ID and user profile name, to allow multiple users connecting to multiple domains with the same kernel gateway name:
sm-local-ssh-ide set-domain-id <<domain_id>>
sm-local-ssh-ide set-user-profile-name <<user_profile_name>>
IAM action ssm:DeregisterManagedInstance
removed from SSHSageMakerClientPolicy
, because instance cleanup is to be run by an admin.
#32 - switch to regional STS endpoints in wrappers, to fetch the local user ID.
#35 - Support HF accelerate and DeepSpeed for inference.
#27 - Support Hugging Face Accelerate for training.
Support SSH into training with SageMaker Training Compiler.
Support for SSH into Jupyter Server instance.
FAQ: Can I connect from my local machine to Jupyter Server in addition to Kernel Gateways?
Improved support for SSH Agent keys forwarding into SageMaker.
FAQ: How do I securely forward my local private SSH keys from local machine to the remote host with SSH Agent?
API: get_instance_ids() method receives an optional timeout_in_sec
parameter that is by default 900 seconds (15 minutes), to highlight the difference from connection_wait_time_seconds
passed to the wrapper, which is by default 600 seconds (10 minutes).
Added development journey to README.
Improved logging.
Improved troubleshooting instructions.
Published by ivan-khvostishkov over 1 year ago
New in v2.0.0:
Internet-free mode support: I'm running SageMaker in a VPC. Do I need to make extra configuration?
Amazon Linux support for SageMaker Studio
MATE desktop is now the default environment for VNC both for Amazon Linux and Debian-based SageMaker Studio images
Windows procedure now uses virtual env and doesn't need admin rights
Documentation improvements, more detailed architecture diagrams
Improve CI/CD testing of the code
Published by ivan-khvostishkov over 1 year ago
New in v1.12.0:
#31 - fixed sm-local-configure
on macOS
FAQ:
sm-wait stop
now have to be used instead of pkill to gracefully shut down the waiting loop
IAM policy SSHSageMakerServerPolicy
changes – ssm:ListInstanceAssociations
requires *
as a resource name to avoid errors in logs (see IAM_SSM_Setup.md for more details).
CDK apps has been split (see IAM_SSM_Setup.md for more details):
API Changes:
ssh_wrapper.training_job_name()
instead of ssh_wrapper.latest_training_job_name()sm-ssh-ide get-metadata
to print SageMaker Studio details to connectsm-ssh-ide set-local-user-id "$LOCAL_USER_ID"
and sm-ssh-ide init-ssm
instead of a single init-ssm command.Added the tip to set the Web VNC screen resolution with xrandr
sm-save-env
now handles special symbols and quotes in variables
Improve CI/CD testing of the code
Published by ivan-khvostishkov over 1 year ago
New in v1.11.0:
IAM/SSM policy update - needs ssm:GetCommandInvocation
permission for better performance – 2628c4f
IAM/SSM policy update - needs *
for ssm:ListInstanceAssociations
to avoid errors in logs – 049f97b
#28 – fixed issue with new SageMaker Python SDK: Invalid bucket name "sagemaker.config INFO - Fetched defaults config from location: ": Bucket name must match the regex. The change related to SageMaker defaults :
https://github.com/aws/sagemaker-python-sdk/pull/3872/files
Added support for SageMaker defaults
Improved MME support for TF
Added support for pre-trained HuggingFace models
Better support for popular SageMaker Studio images
Support attaching to already submitted training jobs with SSHEstimatorWrapper.attach()
Documented alternative way to add SSH Helper dependency through requirements.txt
Published by ivan-khvostishkov over 1 year ago
New in v1.10.4:
#25 - fix for IDE notebook / kernel lifecycle configuration script: installing sudo that is missing in some kernels
Published by ivan-khvostishkov over 1 year ago
New in v1.10.3:
WebVNC through noVNC and AWS Jupyter Proxy
#25 - fix for sm-local-configure
on Linux: sudo for installing AWS CLI and Session Manager Plugin
Published by ivan-khvostishkov over 1 year ago
New in v1.10.2:
#21 - don't update the pre-installed pip in SageMaker Studio that can cause issues with PyYAML
The sm-local-ssh-ide
script now supports extra SSH args for forwarding additional ports:
sm-local-ssh-ide connect <kernel_gateway_name> [--ssh-only] [<extra_ssh_args>]
The new method is_last_session_timeout()
to prevent unused resources for the placeholder job if there's were no SSM or SSH sessions for the specified period of time.
Documentation updates
New train_placeholder.py
script that ends when sessions timeout.
Added section on Forwarding TCP ports over SSH tunnel - useful to access remote apps like Dask or Streamlit
Published by ivan-khvostishkov over 1 year ago
New in v1.10.1:
#20 - fixing location of SSH authorized keys to prevent sm-local-ssh-ide
script from asking root password
Simplified CDK app deployment step in IAM_SSM_Setup.md (no need to clone the source code repo anymore)
Documentation updates
Simplified procedure with instructions for ~/.ssh/config
FAQ - I'm running SageMaker in a VPC. Do I need to make extra configuration?
FAQ - I'm using boto3 Python SDK instead of SageMaker Python SDK, how can I use SageMaker SSH Helper?
Published by ivan-khvostishkov over 1 year ago
New in v1.10.0:
CDK deployment automation in IAM_SSM_Setup.md
Demonstrated the least privilege principle
#12 - Removed EC2 instance from SSM setup
Speed up instance ID resolution with SSMManager (not using CloudWatch logs and SSHLog anymore, except for endpoints)
Speed up instance ID resolution (not using CloudWatch logs anymore, except for endpoints)
#4 - Notebook instances support
The command sm-local-ssh-ide <<kernel_gateway_name>>
is becoming sm-local-ssh-ide connect <<kernel_gateway_name>>
(added connect
for consitency with other scripts)
New command: sm-local-configure
to run on the local machine to install AWS CLI v2 and Sessions Manager plugin
#17 - An option to start only SSH server inside SageMaker Studio: sm-ssh-ide start --ssh-only
Deregistering instances with timestamp: new parameter --delete-older-than-n-days <N>
New tags attached to an SSM instance, in addition to SSHOwner
: SSHCreator
, SSHTimestamp
, SSHResourceName
and SSHResourceArn
.
Env variables passed to SSH helper change: instead of SSH_SSM_TAGS
it now accepts only SSH_OWNER_TAG
, other tags are calculated automatically. SSH_LOG_TO_STDOUT
parameter is needed for notebook instances.
#16 - China AWS Regions support
Stability, usability and performance improvements
Documentation updates
FAQ - Are SageMaker notebook instances supported?
FAQ - How to configure an AWS CLI profile to work with SageMaker SSH Helper? (resolves #14)
Published by ivan-khvostishkov almost 2 years ago
New in v1.9.1:
Support connecting to a different AWS Region by providing region_name in boto3 session and in AWS_REGION / AWS_DEFAULT_REGION env vars in CLI.
Stability fixes.
Documentation updates
FAQ - How to troubleshoot jobs that are failing with the exception or error?
FAQ - I see folders like Desktop, Documents, Downloads, Pictures in SageMaker Studio, is it fine?
FAQ - How do I automate my pipeline with SageMaker SSH Helper end-to-end?
Published by ivan-khvostishkov almost 2 years ago
New in v1.9.0:
Added support for batch transform - #10
Fixed SKLearn support - #2
Added support for basic estimator and improved support for other frameworks, including HuggingFace, TensorFlow and XGBoost.
Added MultiDataModel support without a model object - #4
Fix the deregister_old_instances_from_ssm.py
to allow clean up instances with any role, not only those that contain ‘sagemaker’ - #8
Possibility to override the public keys location path in S3 with SSH_AUTHORIZED_KEYS_PATH environment variable - #8
Additional sm-local-ssh-*
scripts to connect with SSH to processing, training, inference and batch transform containers.
Fixing issue with sudo installation in some containers.
Better support for the distributed multi-GPU scenarios.
Published by ivan-khvostishkov almost 2 years ago
New in v1.8.1:
jq
util in older Ubuntu versionsPublished by ivan-khvostishkov almost 2 years ago
New in v1.8.0:
pip install sagemaker-ssh-helper
.SSHOwner
tagssh_instance_count
passed to SSHEstimatorWrapper.create()
(default is 2) – useful to prevent Throttling API errors in SSMPublished by ivan-khvostishkov about 2 years ago
Initial public release