Process Indicators of Quality

[dt-woods]

All processes in eLCI are hard-coded with (5;5)—no documented review and completeness not scored. See #47 for a path forward for completeness scores (e.g., fractions of captured vs expected flows). For the review process score, I assume all have undergone internal review at this point, so they could be elevated from '5' to '4'. ERG (and potentially others) have provided feedback on certain processes, which should elevate those to a '2'. I'm reaching out to capture what I can on review of processes by external parties, and if there are any existing methods to assist with the completeness check.

[m-jamieson]

I think the issue with moving to even a 4 for the review process is the need for documentation of that review, right? I've certainly done some QA checks on the results (several times over), but I don't think anything has been documented in a more formal sense, even to some level below what would be the ISO 14040/14044 standard.

[dt-woods]

Good point. Let me check with our ISO specialist to see what's needed for documentation on an internal review. It may be something we can patch together from existing internal reviews and upload to EDX. It might be worth it to get a bunch of processes out of '5' to '4' in quality.

[bl-young]

Could you add that documentation to the openLCA reviews field? We've done that with some of the newer data in USLCI. Either a description of the review process or a link to where the review doc lives.

[dt-woods]

Should be able to, Ben. From an internal discussion today, I found that ISO does not provide strict requirements on what counts as a "documented" review; it does provide guidance on what counts as a "critical" review and it provides a set of evaluation metrics to include in a quality review. ISO guidelines appear to be more centered around what we consider flow-level quality indicators, and is more vague regarding process-level indicators.

In the past, our team has recorded our internal reviews (e.g., in a checklist) and marked process review quality as a '4'. It was also brought up that the electricity baselines are generated using publicly available Python code (eLCI), which has its own review (the git history, issues, PRs, and discussions) that includes internal and external parties. This could bump the quality indicator to a '3' or even a '2'.

I guess the question is how do we interpret "documentation" for a quality review of these processes to avoid them looking like they have no quality on the Fed Commons (and elsewhere)?

[bl-young]

Yes I think its an interesting question. Given that the results are regularly reviewed by both NETL and ERG, and that the code has undergone significant review from NETL, ERG (and at times EPA), it feels to me that a 2 is justified. I feel that follows both the spirit and the intent of the guidelines - this thing has been heavily reviewed!

[dt-woods]

@m-jamieson, how do you feel about a new hard-coded process-level DQI of (2;5) in place of the current (5;5)?

[m-jamieson]

I'm good with that. Honestly, we could probably upgrade to 4 for the process completeness review as well. "<40% of determined flows have been evaluated and given a value" seems like a pretty low bar.