Consider standardising usage of artifacts folder.

[jmalloc]

From what I can tell, our usage of this is a bit of a mishmash at the moment.

IIRC we started trying to unify the structure of this directory across packages by putting coverage output in the same location across languages, etc. I am wondering if it might be better have each package (go, js, docker, etc) use a fixed subdirectory (./aftifacts/go, etc).

Perhaps even within that we could adopt a POSIX-esque directory structure. We've sort-of-kinda-maybe already done this in a few places. For example:

• ./artifacts/protobuf/bin <-- install protoc here (I think this is already the location)
• ./artifacts/go/bin <-- install protoc-gen-go and other go-specific tools here
• ./artifacts/<whatever>/etc
• ./artifacts/<whatever>/var <-- build logs, coverage reports, etc

The upshot of this is that we can be confident that two packages wont interfere with each other, and also automatically set PATH within the makefiles.

[ezzatron]

Per-language directories might make sense, but I don't think I'm convinced about POSIX directories. Wouldn't we be better off with less esoteric names? And 99% of all things will end up under var, meaning that we're adding an extra path segment that essentially becomes meaningless.

[jmalloc]

Yep, that's fair enough. I would have thought the POSIX names were the opposite of esoteric, but I am in no way set on that idea, it has its flaws. FWIW, the things I would expect to go in var would be things like touch files (by analogy to pid files) and logs, so at best we'd probably still be inventing non-standard names for some things anyway.

These are the kinds of artifacts that we've ended up with in pkg-go and pkg-protobuf:

artifact	current location	notes
debug & release binaries	artifacts/build	def feels like it should be artifacts/go/build
release archives	artifacts/archive	I can see these benefiting from cross-language/package support
code coverage output	artifacts/coverage	early on we deliberately made this the same cross language
touch files	artifacts/go/lint/	only used in one place
third-party Go binaries	artifacts/go/bin	we have no require-dev in Go, so we install tools here via the makefiles
third-party protobuf binaries	artifacts/protobuf/bin	this is language agnostic, as in protoc can be used by multiple languages
"arg" files	artifacts/protobuf/args	the protobuf package builds files containing large lists of cmd line arguments to pass to protoc, basically a cache

So looks like I've already been using language subdirectories in some places. I can't think of / remember any particular reason to require all languages to put their coverage output in the same directory, but then there's nothing human-readable in there for Go devs, which may not be the case for JS?

Side note: pkg-docker no longer uses artifacts at all.

Perhaps rather than specifying an actual structure under the package-specific directory we could just have a few guidelines. For example, naming directories after the phony target that produces them, where applicable.

[ezzatron]

I would have thought the POSIX names were the opposite of esoteric

I guess I'm just in the group of people to which the etc/var/usr/dev POSIX directory names never made any intrinsic sense. If you're a *nix user you tend to learn their meanings by rote over time, but I've never been like "Hmm, what's the most obvious place for config files? Oh, of course! The etc directory!".

I can't think of / remember any particular reason to require all languages to put their coverage output in the same directory

IIRC, when we started out the make files project, we pretty firmly had the idea of making it possible to have a repo with multiple languages co-existing, because we knew for example that we have PHP + JS projects at CWX. It's possible that introducing a hard language split in the artifacts directory would make those kinds of projects more painful to work on. But I'm not sure either way.

but then there's nothing human-readable in there for Go devs, which may not be the case for JS?

Yeah, there are HTML coverage reports in there, but it wouldn't be a massive deal to move them either, since we open them with coverage-open anyway.

Perhaps rather than specifying an actual structure under the package-specific directory we could just have a few guidelines. For example, naming directories after the phony target that produces them, where applicable.

I'm pretty sure that for every integration I've written so far I've tried to use a directory or file named after the tool being integrated. There can be multiple targets associated with one tool, so I'm not sure that naming things after targets makes sense in all cases either.

---

The simplest solution I can think of is to force packages to put all of their artifacts under a directory matching the package. At least then clashes are only a concern within a single package. Potential naming schemes:

• artifacts/pkg/js and artifacts/ext/cwx/pkg/js
• artifacts/pkg-js and artifacts/ext-cwx-pkg-js

But again, I'm not even really convinced it's justified.