PIPELINE-2662: Add basic repository structure

[tomaslink]

Features:

• ✅ Standard Python project structure & packaging.
• ✅ Dependency management with [pip-tools].
• ✅ Tools for quality checks: documentation, [PEP8], typehints, codespell.
• ✅ [Optional] pre-commit hooks to enforce automatic quality checks.
• ✅ Dockerization with focus in image size optimization.
• ✅ Continuous Integration (CI) workflows (GitHub Actions).
• ✅ Continuous Deployment (CI) workflows (Google Cloud Build).
• ✅ Makefile with shortcuts to increase development speed.
• ✅ README badges with project information.
• ✅ Development workflow documentation.
• ✅ Support for [Apache Beam] integrated pipelines.

https://globalfishingwatch.atlassian.net/browse/PIPELINE-2662

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[rdgfuentes]

LGTM. The only comment/doubt I have is about the complexity of this, I'd recommend to validate that with actual end users of this and demo/walk thru the process we aim them to follow when using this template to create a new project and gather their feedback.

[tomaslink]

LGTM. The only comment/doubt I have is about the complexity of this, I'd recommend to validate that with actual end users of this and demo/walk thru the process we aim them to follow when using this template to create a new project and gather their feedback.

I agree. But the first users of this template are us :). So, it should have everything we need, without making the workflow too rigid. For that I documented the pre-commit hooks as optional. Once we merge this basic structure I would have a meeting with research and gather feedback. Then we can add extra things, like a CLI application structure.

[andres-arana]

@tomaslink The only feedback I have here is what we already discussed in the meeting with Jenn and Anna last week: We need to make it a lot more prevalent in the docs which things are optional and which things aren't depending on who's using this. Pytest is not required for researchers working on POC stuff. Code quality stuff in general is optional, which means most likely everything related to github actions won't be used on POCs. Commit hooks is something you only want to enable when you are dealing with a production repository.

Not sure how you want to document that here, but it needs to be clearer.

[tomaslink]

@tomaslink The only feedback I have here is what we already discussed in the meeting with Jenn and Anna last week: We need to make it a lot more prevalent in the docs which things are optional and which things aren't depending on who's using this. Pytest is not required for researchers working on POC stuff. Code quality stuff in general is optional, which means most likely everything related to github actions won't be used on POCs. Commit hooks is something you only want to enable when you are dealing with a production repository.

Not sure how you want to document that here, but it needs to be clearer.

@andres-arana pytest does no harm. If you don't have tests, the CI will just run fine, it will not block the workflow of researchers. I don't think we need to disable tests in the CI. It is better to have everything configured and prepared for whenever you add tests. Did you meant that or just documenting that creating tests is optional?

The same goes for PEP8 check, in that case it will fail, but can merge it anyways bypassing the rules...So it will not block the workflow. Even better, you already will have information about the status of the repository. One of the goals of the template is not having to re-do stuff, boilerplate configuration in every repository. Again, did you meant to remove PEP8 check from CI or document that failing of PEP8 check can be ignored?

Even more, the CI jobs will not be enforced in a PR for merging unless you configure the repo to do so. That's another step we need to document....

The pre-commit quality checks: it is documented that the step for installing pre-commit hooks is optional. Do you think that is not clear enough? Did you read the whole README and Preparing the environment section?

[smpiano]

LGTM. Meet with Tom and Andres to review.