More efficiently use diskspace - #1012
Conversation
346387f to
5479f8e
Compare
fd90dbe to
0f3e9e4
Compare
| linux_train_mock, | ||
| is_macos_with_m_chip_mock, | ||
| ): | ||
| # pylint: disable=consider-using-with |
There was a problem hiding this comment.
nit: you may be able to avoid arguing with the linter about this if you declare a helper function instead of using a lambda?
There was a problem hiding this comment.
thanks, the lambda has been replaced
0f3e9e4 to
28d6abd
Compare
| if os.path.exists(final_results_dir): | ||
| shutil.rmtree(final_results_dir) |
There was a problem hiding this comment.
I'm a bit uncomfortable blindly removing this whole directory. I'd rather only operate on exactly the files we know are relevant to the current run of ilab and not make assumptions about what can be removed otherwise. I just want to avoid any surprising destructive behavior.
There was a problem hiding this comment.
Actually looking at this again I think I can just remove these two lines. The large model files that it contained is now moved instead of copied so this directory isn't taking up a lot of space.
28d6abd to
368db67
Compare
In order to allow ilab to run e2e on colab where diskspace is limited we move rather then copy large files around, also rm files we're done with to save space for the next step. Fix instructlab#1011 Signed-off-by: Derek Higgins <derekh@redhat.com>
368db67 to
e0a08b6
Compare
| for file in glob(final_results_dir + "/*.safetensors"): | ||
| os.remove(file) | ||
|
|
||
| shutil.move(final_results_dir + "/ggml-model-f16.gguf", gguf_models_file) |
There was a problem hiding this comment.
only if you have to make another change anyway, would be good to make this os.path.join(). I know this isn't new code though, so it's not worth blocking over
In order to allow ilab to run e2e on colab where diskspace is limited we move rather then copy large files around, also rm files we're done with to save space for the next step.
Fix #1011