Automate dependencies

[actuarysailor]

What problem are you facing?

If you use the integrated pre-commit GitHub application, the fact that the docker images do not automatically download, install, or have the dependencies pre-defined causes issues.

Could you please update the hooks so that they either download or come pre-loaded with a version of the tools many of these hooks are dependent on?

How could pre-commit-terraform help solve your problem?

Either add a parameter in the hooks to download a specific version of the tools, or pre-load with latest version of tools.

[yermulnik]

I'm not familiar with https://pre-commit.ci platform, though under the Github Actions section of README there's an example of how to use pre-commit-terraform Docker image, which has all required tools pre-installed — is it any helpful in regards to your use case? 🤔

[MaxymVlasov]

It can be resolved by implementing #418 (comment)

[webknjaz]

@yermulnik pre-commit.ci separates what it does into two stages.

The first one is preparing the testing environment, it installs the deps (including additional_dependencies), creates venvs and such. It has network access, and I'm pretty sure it uses the feature I requested years ago: pre-commit/pre-commit#456 (comment). This environment is then cached.

The second step is running the checks. It does not provision anything. Moreover, it has networking disconnected from the env where it's being executed. This stage is not supposed to rely on network access in any way or attempt to provision things, since it's expected that the provisioned env is cached and never restarted until said cache is invalidated.

So the only way to address this is to provide the dependencies in a way that is native to pre-commit.

[MaxymVlasov]

This stage is not supposed to rely on network access in any way or attempt to provision things,

That's a problem for tflint and terrascan in some setups 🤔

tflint requires tflint --init for .tflint.hcl which includes plugins:

plugin "aws" {
    enabled    = true
    version    = "0.37.0"
    source     = "github.com/terraform-linters/tflint-ruleset-aws"
}

In theory, we could init all available plugins and pack them for pre-commit.ci somewhere, but, that's will not work if user uses outdated version or we not fast enough to update our build. In other words, that requires perfect sync from us and all pre-commit.ci users of terraform_tflint hook if user will have any plugin.
So, if we not figure out how to utilize additional_dependencies for tflint plugins to allow users set their own deps, terraform_tflint usage in pre-commit.ci will be highly limited.

Regarding terrascan - it requires terrascan init to download rules before run terrascan scan
And it could try to update rules right before scan:

debug   downloader/getter.go:136  download with remote type: "", remote URL: "", destination dir: "/tmp/nquawb"
debug   cli/run.go:253            remote url and type not configured, proceeding with regular scanning

And I not sure what better:

• pack it with at least some cached policies to catch at least something
• not pack it at all, to eliminate falsy-passed checks to users, as they can end in security holes.

Other hooks should be fine, just need to find out how to deal with additional_dependencies

[actuarysailor]

Not sure how my repo doing a PR to test the work for this issue auto closed it.... that is interesting

[actuarysailor]

This stage is not supposed to rely on network access in any way or attempt to provision things,

That's a problem for tflint and terrascan in some setups 🤔

tflint requires tflint --init for .tflint.hcl which includes plugins:

plugin "aws" {
enabled = true
version = "0.37.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}
In theory, we could init all available plugins and pack them for pre-commit.ci somewhere, but, that's will not work if user uses outdated version or we not fast enough to update our build. In other words, that requires perfect sync from us and all pre-commit.ci users of terraform_tflint hook if user will have any plugin. So, if we not figure out how to utilize additional_dependencies for tflint plugins to allow users set their own deps, terraform_tflint usage in pre-commit.ci will be highly limited.

Regarding terrascan - it requires terrascan init to download rules before run terrascan scan And it could try to update rules right before scan:

debug   downloader/getter.go:136  download with remote type: "", remote URL: "", destination dir: "/tmp/nquawb"
debug   cli/run.go:253            remote url and type not configured, proceeding with regular scanning

And I not sure what better:

• pack it with at least some cached policies to catch at least something
• not pack it at all, to eliminate falsy-passed checks to users, as they can end in security holes.

Other hooks should be fine, just need to find out how to deal with additional_dependencies

I'm not so sure... the machine running the docker image has web access etc... and most of these opentofu/terraform commands download from the web to the current directory, your just containerizing the code of the commands etc... basically like a lambdafunction, the code still executes in the current repo

I will post some screenshots from the pre-commit app website that runs these.

[OJFord]

This isn't specific to 'pre-commit.ci' (which seems to be some low-effort unaffiliated project squatting on the search terms 'set up pre-commit in ci'?) - any use of pre-commit but especially apparent in a fresh environment such as CI (whether that be GH Actions, pre-commit.ci, whatever) would expect that pre-commit --install-hooks installs the necessary tools.

I've found this issue in exactly this way: everything is installed for my other hooks, but terraform_fmt/validate is failing with 'terraform not found'. Note the commit just above linking to this issue - 'Does seem weird we have to do this.' - indeed!

It could defer the heavy-lifting to https://github.com/tofuutils/tenv, just setting the install dir appropriately; maybe passing through version override from pre-commit params.