fix: Remove sagemaker_job_name from hyperparameters in TrainingStep by staubhp · Pull Request #2950 · aws/sagemaker-python-sdk

staubhp · 2022-02-22T23:54:29Z

Issue #, if available:
2940

Description of changes:
Some framework classes add a sagemaker_job_name hyperparameter with a dynamic training job name. This is both inaccurate (the job name will be set when the step runs) and it breaks caching for those steps.

Testing done:
Unit, manual

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter · 2022-02-23T00:12:38Z

Codecov Report

Merging #2950 (2dfde22) into dev (086258d) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##              dev    #2950   +/-   ##
=======================================
  Coverage   89.80%   89.80%           
=======================================
  Files         196      196           
  Lines       16563    16565    +2     
=======================================
+ Hits        14875    14877    +2     
  Misses       1688     1688

Impacted Files	Coverage Δ
src/sagemaker/workflow/steps.py	`97.81% <100.00%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 086258d...2dfde22. Read the comment docs.

sagemaker-bot · 2022-02-23T00:21:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 298f965
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-23T01:13:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 298f965
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-23T02:25:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 298f965
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T18:43:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6ff0c7a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:13:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dbddcc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:34:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6ff0c7a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:35:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6ff0c7a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T19:50:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dbddcc2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jerrypeng7773 · 2022-03-02T20:57:50Z

        )
        request_dict = self.estimator.sagemaker_session._get_train_request(**train_args)
        request_dict.pop("TrainingJobName")
+        request_dict["HyperParameters"].pop("sagemaker_job_name", None)


wonder if it is possible that this request dict does not have HyperParameters in it?

It will always be there, but let me add a safety check

sagemaker-bot · 2022-03-02T21:00:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dbddcc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T21:20:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dbddcc2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-02T23:49:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dbddcc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-03T00:30:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2dfde22
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-03T01:21:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2dfde22
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-03T01:48:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2dfde22
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

shreyapandit

As in comments

…2950) Co-authored-by: Payton Staub <pstaub@amazon.com>

* change: update code to get commit_id in codepipeline (#2961) * feature: Data Serializer (#2956) * change: reorganize test files for workflow (#2960) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> * feature: TensorFlow 2.4 for Neo (#2861) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (#2950) Co-authored-by: Payton Staub <pstaub@amazon.com> * fix: Style update in DataSerializer (#2962) * documentation: smddp doc update (#2968) * fix: container env generation for S3 URI and add test for the same (#2971) * documentation: update sagemaker training compiler docstring (#2969) * feat: Python 3.9 for readthedocs (#2973) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: Shreya Pandit <shreya.pandit25@gmail.com> Co-authored-by: Ahsan Khan <ahsan.al.zaki@gmail.com> Co-authored-by: Mufaddal Rohawala <89424143+mufaddal-rohawala@users.noreply.github.com>

* change: update code to get commit_id in codepipeline (aws#2961) * feature: Data Serializer (aws#2956) * change: reorganize test files for workflow (aws#2960) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> * feature: TensorFlow 2.4 for Neo (aws#2861) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (aws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com> * fix: Style update in DataSerializer (aws#2962) * documentation: smddp doc update (aws#2968) * fix: container env generation for S3 URI and add test for the same (aws#2971) * documentation: update sagemaker training compiler docstring (aws#2969) * feat: Python 3.9 for readthedocs (aws#2973) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: Shreya Pandit <shreya.pandit25@gmail.com> Co-authored-by: Ahsan Khan <ahsan.al.zaki@gmail.com> Co-authored-by: Mufaddal Rohawala <89424143+mufaddal-rohawala@users.noreply.github.com>

…ws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com>

* change: update code to get commit_id in codepipeline (aws#2961) * feature: Data Serializer (aws#2956) * change: reorganize test files for workflow (aws#2960) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> * feature: TensorFlow 2.4 for Neo (aws#2861) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (aws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com> * fix: Style update in DataSerializer (aws#2962) * documentation: smddp doc update (aws#2968) * fix: container env generation for S3 URI and add test for the same (aws#2971) * documentation: update sagemaker training compiler docstring (aws#2969) * feat: Python 3.9 for readthedocs (aws#2973) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: Shreya Pandit <shreya.pandit25@gmail.com> Co-authored-by: Ahsan Khan <ahsan.al.zaki@gmail.com> Co-authored-by: Mufaddal Rohawala <89424143+mufaddal-rohawala@users.noreply.github.com>

…ws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com>

* change: update code to get commit_id in codepipeline (aws#2961) * feature: Data Serializer (aws#2956) * change: reorganize test files for workflow (aws#2960) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> * feature: TensorFlow 2.4 for Neo (aws#2861) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> * fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (aws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com> * fix: Style update in DataSerializer (aws#2962) * documentation: smddp doc update (aws#2968) * fix: container env generation for S3 URI and add test for the same (aws#2971) * documentation: update sagemaker training compiler docstring (aws#2969) * feat: Python 3.9 for readthedocs (aws#2973) Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> * fix doc structure * archive 1.6.0 doc * add new args, refs, and links * fix version number * incorp eng feedback, update docstrings, improve xref * Trigger Build * minor fix, trigger build again * fix typo Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Jeniya Tabassum <jeniya.tabassum@gmail.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: Ben Crabtree <bencrab@amazon.com> Co-authored-by: Dewen Qi <qidewen@amazon.com> Co-authored-by: Qingzi-Lan <83724147+Qingzi-Lan@users.noreply.github.com> Co-authored-by: Payton Staub <staubhpa@gmail.com> Co-authored-by: Payton Staub <pstaub@amazon.com> Co-authored-by: Shreya Pandit <shreya.pandit25@gmail.com> Co-authored-by: Ahsan Khan <ahsan.al.zaki@gmail.com> Co-authored-by: Mufaddal Rohawala <89424143+mufaddal-rohawala@users.noreply.github.com>

BasilBeirouti force-pushed the dev branch from 0e4fd55 to 26b984a Compare February 23, 2022 02:20

Payton Staub added 2 commits March 2, 2022 10:34

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep

8118a81

Fix unit tests

6ff0c7a

staubhp force-pushed the fix-training-step-caching branch from 298f965 to 6ff0c7a Compare March 2, 2022 18:34

black-format

dbddcc2

jerrypeng7773 reviewed Mar 2, 2022

View reviewed changes

qidewenwhen approved these changes Mar 2, 2022

View reviewed changes

Add property check

2dfde22

shreyapandit reviewed Mar 3, 2022

View reviewed changes

Comment thread tests/unit/sagemaker/workflow/test_step_collections.py

shreyapandit suggested changes Mar 3, 2022

View reviewed changes

shreyapandit added the type: question label Mar 3, 2022

jerrypeng7773 approved these changes Mar 3, 2022

View reviewed changes

shreyapandit approved these changes Mar 3, 2022

View reviewed changes

shreyapandit merged commit 20df3d7 into aws:dev Mar 3, 2022

shreyapandit removed the type: question label Mar 3, 2022

shreyapandit pushed a commit that referenced this pull request Mar 4, 2022

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (#…

067cac2

…2950) Co-authored-by: Payton Staub <pstaub@amazon.com>

jerrypeng7773 pushed a commit to jerrypeng7773/sagemaker-python-sdk that referenced this pull request Apr 19, 2022

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (a…

5ea9f21

…ws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com>

Evan-W-ang added a commit to Evan-W-ang/sagemaker-python-sdk that referenced this pull request Jun 8, 2026

fix: Remove sagemaker_job_name from hyperparameters in TrainingStep (a…

4da87bf

…ws#2950) Co-authored-by: Payton Staub <pstaub@amazon.com>

Uh oh!

Conversation

staubhp commented Feb 22, 2022

Merge Checklist

General

Tests

Uh oh!

codecov-commenter commented Feb 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Feb 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

jerrypeng7773 Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

staubhp Mar 3, 2022

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 3, 2022

AWS CodeBuild CI Report

Uh oh!

Uh oh!

shreyapandit left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov-commenter commented Feb 23, 2022 •

edited

Loading