Cache Shell Script of `spack load` and `spack env activate` by RikkiButler20 · Pull Request #47755 · spack/spack

RikkiButler20 · 2024-11-23T02:39:33Z

This will fix the problem of spack load and spack env activate taking too long and currently iterating over all the package's dependency's python package files. This will also prevents the drifting of the installed spec from the current, updated version on Spack

adamjstewart · 2024-11-23T11:14:05Z

Very excited for this and happy to test on fish!

aweits · 2024-11-24T13:47:59Z

What if we cached the environment changes at install-time instead? That would also solve issues related to allowing older installs to still function/load... (willing to help with this, would allow me to jettison some local code that is getting harder and harder to maintain)

tgamblin · 2024-11-24T14:06:12Z

What if we cached the environment changes at install-time instead?

This is the plan. There are some details in the slack discussion linked from #47603, and the bit about “package drift” there.

i think @RikkiButler20 is doing this in steps (see checklist) — probably need one or two more to move the logic to package install time and env install time.

@aweits I think if you want to help, iterating on reviews would be great so we can get this in for 1.0!

RikkiButler20 · 2024-11-27T01:07:41Z

@spackbot Fix style

spackbot-app · 2024-11-27T01:07:44Z

Let me see if I can fix that for you!

spackbot-app · 2024-11-27T01:09:19Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/load.py
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
2 files reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2024-12-05T01:15:56Z

@spackbot Fix style

spackbot-app · 2024-12-05T01:15:59Z

Let me see if I can fix that for you!

spackbot-app · 2024-12-05T01:17:28Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

adamjstewart · 2025-01-15T11:01:48Z

Going to unsubscribe from this PR while it's still a WIP (too many notifications) but please ping me to review fish support once it is complete.

RikkiButler20 · 2025-01-16T20:27:11Z

@spackbot Fix style

spackbot-app · 2025-01-16T20:27:14Z

Let me see if I can fix that for you!

spackbot-app · 2025-01-16T20:28:19Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/hooks/__init__.py
  lib/spack/spack/hooks/cache_shell_script.py
  lib/spack/spack/test/environment_modifications.py
  lib/spack/spack/test/hooks/cache_shell_script.py
  lib/spack/spack/test/util/environment.py
  lib/spack/spack/util/environment.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
Fixing /tmp/tmpu87evsa3/spack/lib/spack/spack/test/hooks/cache_shell_script.py
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/hooks/cache_shell_script.py
reformatted lib/spack/spack/util/environment.py
All done! ✨ 🍰 ✨
2 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/cmd/load.py:12: [F401] 'spack.user_environment as uenv' imported but unused
lib/spack/spack/cmd/load.py:13: [F401] 'spack.util.environment' imported but unused
  flake8 found errors
==> Running mypy checks
lib/spack/spack/util/environment.py:662: error: Item "NameModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/util/environment.py:662: error: Item "NameValueModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/version/version_types.py:135: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Tuple[str]")  [assignment]
lib/spack/spack/variant.py:130: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [misc]
Found 6 errors in 4 files (checked 636 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Signed-off-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

…kkiButler20/spack into feature/cache-spack-load-and-activate

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Ensures activation scripts are written correctly when locking is disabled Signed-off-by: John Parent <john.parent@kitware.com>

Signed-off-by: John Parent <john.parent@kitware.com>

RikkiButler20 · 2026-07-08T21:50:08Z

@spackbot re-run pipeline

spackbot-app · 2026-07-08T21:50:13Z

I've started that pipeline for you!

Signed-off-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Signed-off-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

becker33 · 2026-07-28T04:18:47Z

+The shell scripts contain all the environment modifications needed for the package, including:
+
+1. Updating PATH to include the package's bin directory
+2. Setting LD_LIBRARY_PATH for shared libraries


By default we don't do LD_LIBRARY_PATH because it breaks system libraries.

This should probably have a link to the prefix_inspections docs for how to modify the behavior.

Is this what you mean? https://spack.readthedocs.io/en/latest/configuration.html#environment-modifications

I was talking about this: https://spack.readthedocs.io/en/latest/module_file_support.html#global-environment-modifications

I don't know that we document it well, but we reuse these module configurations when doing module-like operations (spack load, spack env activate).

becker33 · 2026-07-28T04:49:09Z

-        cmds += "function despacktivate;\n"
-        cmds += "   spack env deactivate;\n"
-        cmds += "end;\n"
+        if "color" in os.getenv("TERM", ""):


These are the commands that are part of env activation, not part of putting the view in the path.

The code is basically taking the cached activation script, and prepending a short header that is computed on the fly. This is the small portion computed on the fly, for things like e.g. spack env activate -p which modifies the behavior of the activation header but not the cached script.

becker33 · 2026-07-28T04:52:08Z

+def generate_script(shell_script_path: str, mods: str, comments: str):
+    """Helper function to write spec's shell scripts
+
+    Args:
+        shell_script_path: Path to the shell script.
+        mods: Modifications to write to the script.
+        comments: Comment character(s) to use in the script
+            (e.g. "::" for bat and ### for all other shells)
+    """
+
+    try:
+        fs.mkdirp(os.path.dirname(shell_script_path))
+
+        header = (
+            f"{comments} This file is auto-generated by Spack. Do not edit this file directly.\n"
+            f"{comments} Generated on: {datetime.now().strftime('%Y-%m-%d')}\n\n"
+        )
+        with fs.write_tmp_and_move(shell_script_path, encoding="utf-8") as f:
+            f.write(header)
+            f.write(mods)
+    except OSError as e:
+        raise OSError(f"Error generating to {shell_script_path}\n{e}")


This method seems to share a lot of code with the analogous method for environments.

I think we can move some core functionality out of this hooks module into the spack.user_environment module, and then we can call it from both the generate_spec_scripts hooks and the spack.environment.generate_scripts module.

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Remove update_env_script Signed-off-by: Kayla Butler <butler59@llnl.gov>

Signed-off-by: Kayla Butler <butler59@llnl.gov>

RikkiButler20 added feature A feature is missing in Spack commands shell-support labels Nov 23, 2024

RikkiButler20 self-assigned this Nov 23, 2024

spackbot-app Bot added core PR affects Spack core functionality tests General test capability(ies) labels Nov 23, 2024

adamjstewart added this to the v0.24 milestone Nov 23, 2024

becker33 modified the milestones: v1.0.0, v1.0.0 stretch goals Nov 25, 2024

tgamblin modified the milestones: v1.0.0 stretch goals, v1.0.0 Nov 26, 2024

aweits reviewed Dec 2, 2024

View reviewed changes

Comment thread lib/spack/spack/cmd/load.py Outdated

spackbot-app Bot added the utilities label Dec 18, 2024

spackbot-app Bot added the documentation Improvements or additions to documentation label Jan 3, 2025

tldahlgren reviewed Jan 14, 2025

View reviewed changes

Comment thread lib/spack/spack/hooks/cache_shell_script.py Outdated

tldahlgren reviewed Jan 14, 2025

View reviewed changes

Comment thread lib/spack/spack/test/hooks/cache_shell_script.py Outdated

RikkiButler20 marked this pull request as draft January 16, 2025 20:26

RikkiButler20 and others added 4 commits July 7, 2026 13:48

Fix import

6992ae5

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix tty imports

636ddd0

Signed-off-by: Kayla Butler <butler59@llnl.gov>

[@spackbot] updating style on behalf of RikkiButler20

ce62e7b

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Add test for generated env scripts

2bdcc46

Signed-off-by: Kayla Butler <butler59@llnl.gov>

RikkiButler20 force-pushed the feature/cache-spack-load-and-activate branch from d8dedb4 to 2bdcc46 Compare July 7, 2026 20:49

github-actions Bot added the ci Issues related to Continuous Integration label Jul 7, 2026

RikkiButler20 and others added 9 commits July 7, 2026 13:54

Merge branch 'develop' into feature/cache-spack-load-and-activate

3d83fa6

Signed-off-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>

Remove unnecessary import

438c595

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Merge branch 'feature/cache-spack-load-and-activate' of github.com:Ri…

d4caba2

…kkiButler20/spack into feature/cache-spack-load-and-activate

Clean imports

fbaefef

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix string import

6f495e0

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Fix style

22c353c

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Env scripts exists when activated

65d92db

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Ensure dot env directory is present before writing

345e041

Ensures activation scripts are written correctly when locking is disabled Signed-off-by: John Parent <john.parent@kitware.com>

quote env typing

0c9d923

Signed-off-by: John Parent <john.parent@kitware.com>

RikkiButler20 removed the ci Issues related to Continuous Integration label Jul 8, 2026

RikkiButler20 and others added 6 commits July 21, 2026 14:36

Merge branch 'develop' into feature/cache-spack-load-and-activate

89f0b1d

Signed-off-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>

Fix style

b87960c

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Merge branch 'develop' into feature/cache-spack-load-and-activate

c27bd6c

Signed-off-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>

Fix pulled changes

706309b

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Remove calls to ev.active_env

ff52f4b

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Merge branch 'develop' into feature/cache-spack-load-and-activate

b2417e9

becker33 requested changes Jul 28, 2026

View reviewed changes

RikkiButler20 added 5 commits July 29, 2026 10:01

Remove LD_LIBRARY_PATH from docs

e4042d2

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Cache default view

34f2639

Remove update_env_script Signed-off-by: Kayla Butler <butler59@llnl.gov>

Add source_env_script

f3c7e4a

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Add source_script

82b6be5

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Don't regenerate env scripts when activating

a7cd614

Signed-off-by: Kayla Butler <butler59@llnl.gov>

Uh oh!

Conversation

RikkiButler20 commented Nov 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Nov 23, 2024

Uh oh!

aweits commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RikkiButler20 commented Nov 27, 2024

Uh oh!

spackbot-app Bot commented Nov 27, 2024

Uh oh!

spackbot-app Bot commented Nov 27, 2024

Uh oh!

Uh oh!

RikkiButler20 commented Dec 5, 2024

Uh oh!

spackbot-app Bot commented Dec 5, 2024

Uh oh!

spackbot-app Bot commented Dec 5, 2024

Uh oh!

Uh oh!

Uh oh!

adamjstewart commented Jan 15, 2025

Uh oh!

RikkiButler20 commented Jan 16, 2025

Uh oh!

spackbot-app Bot commented Jan 16, 2025

Uh oh!

spackbot-app Bot commented Jan 16, 2025

Uh oh!

RikkiButler20 commented Jul 8, 2026

Uh oh!

spackbot-app Bot commented Jul 8, 2026

Uh oh!

Uh oh!

becker33 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

RikkiButler20 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

becker33 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

becker33 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

becker33 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

RikkiButler20 commented Nov 23, 2024 •

edited

Loading

aweits commented Nov 24, 2024 •

edited

Loading

tgamblin commented Nov 24, 2024 •

edited

Loading