Summary
SageMaker Python SDK is an open source library for training and deploying machine learning models on Amazon SageMaker. An issue where the HMAC secret key is stored in environment variables and disclosed via the DescribeTrainingJob API has been identified.
Impact
- Function and Payload Tampering: Attackers with DescribeTrainingJob permissions may extract HMAC secret keys and forge serialized function payloads stored in S3. These tampered payloads would be processed and executed without triggering integrity validation errors, enabling unintended code substitution.
- Arbitrary Code Execution in the Training Environment: An third party with both DescribeTrainingJob permissions and write access to the job's S3 output location can extract the HMAC key, craft inappropriate Python objects, and achieve remote code execution in the client's Python process when the victim retrieves remote function results.
- Data and Credentials Handling: Arbitrary remote code execution may interact with sensitive data, model artifacts, environment variables, and potentially AWS metadata.
- Cross-Tenant or Shared Environment Risks: In multi-tenant, shared S3 bucket, a disclosed HMAC key could act as a pivot point to perform inappropriate actions against other users' remote function workloads. This could leverage the IAM permissions, shared S3 buckets, or VPC resources to compromise adjacent services or data.
Impacted versions
- SageMaker Python SDK v3 < v3.2.0
- SageMaker Python SDK v2 < v2.256.0
Patches
This issue has been addressed in SageMaker Python SDK version v3.2.0 and v2.256.0. Upgrading to the latest version immediately and ensuring any forked or derivative code is patched to incorporate the new fixes is recommended.
Workarounds
Customers using self-signed...