Skip to content

Fix llama 2 10#520

Merged
gkumbhat merged 2 commits into
foundation-model-stack:mainfrom
gkumbhat:fix_llama_2_10
Mar 19, 2026
Merged

Fix llama 2 10#520
gkumbhat merged 2 commits into
foundation-model-stack:mainfrom
gkumbhat:fix_llama_2_10

Conversation

@gkumbhat

Copy link
Copy Markdown
Collaborator

Changes

  • llama started giving compilation failures on pytorch 2.10 when running the model on TP > 1. There seems to be changes in torch 2.10.0, which made the tracking of fx graph more strict. In current implementation, we were wrapping TP strategy around embedding and head, which seemed not necessary, since llama like granite, doesn't use tied weights and doesn't need special handling for head or embedding. This likely created a differences in the graph and resulted in symbolic shape issues. We are already applying distributed strategy around individual Llama blocks.
  • This PR simplifies this by removing those conditional blocks and this makes the model run on torch 2.10.

@gkumbhat gkumbhat marked this pull request as draft March 18, 2026 16:12
@gkumbhat gkumbhat closed this Mar 18, 2026
Signed-off-by: Gaurav-Kumbhat <Gaurav.Kumbhat@ibm.com>
@gkumbhat gkumbhat reopened this Mar 18, 2026
Signed-off-by: Gaurav-Kumbhat <Gaurav.Kumbhat@ibm.com>
@gkumbhat gkumbhat marked this pull request as ready for review March 18, 2026 19:10

@flaviabeo flaviabeo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@gkumbhat gkumbhat merged commit 479b5c7 into foundation-model-stack:main Mar 19, 2026
4 checks passed
@gkumbhat gkumbhat deleted the fix_llama_2_10 branch March 19, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants