Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix-66 #74

Merged
merged 1 commit into from
Oct 9, 2024
Merged

fix-66 #74

merged 1 commit into from
Oct 9, 2024

Conversation

MichaelClifford
Copy link
Collaborator

@MichaelClifford MichaelClifford commented Oct 8, 2024 •
edited
Loading

Closes #66

These changes have been tested on the MOC cluster and the training run completes as part of a Kubeflow pipeline.

This PR changes:

  • The image for model training has beed changed from a custom model to the RHEL AI 1.1 image
  • Replaced the custom script for running training with a direct torchrun call instead

@MichaelClifford MichaelClifford mentioned this pull request Oct 8, 2024
Open
@MichaelClifford
WIP: fix-66
9b0fd55 
Signed-off-by: Michael Clifford <mcliffor@redhat.com>
@MichaelClifford MichaelClifford changed the title WIP: fix-66 fix-66 Oct 9, 2024
@MichaelClifford MichaelClifford marked this pull request as ready for review October 9, 2024 14:13
Shreyanand
Shreyanand approved these changes Oct 9, 2024
View reviewed changes
Copy link
Member

@Shreyanand Shreyanand left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, The PR adds the torchrun command to the pytorch job component. The command directly calls the instructlab main_ds training script that eliminates the need for additional scripts such as run_main_ds.py that were previously used.

@Shreyanand Shreyanand merged commit 249dfd3 into redhat-et:main Oct 9, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Shreyanand Shreyanand Shreyanand approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Remove need for custom scripts in PyTorchJob step
2 participants
@MichaelClifford @Shreyanand