The current implementation may not give the requested number of ratings per hit because hits may fall into the "accept but not use". In a recent run 34% of hits fell into this bucket. This adds inconsistency to the ratings (not a constant number) and is also inefficient (wasted $). There are at least two solutions:
- Add a real-time failure for the pair comparison environment check which was 72% of failures in a recent run.
- Use the MTurk API to create a controller that monitors and updates the hits to meet the target goals.
The current implementation may not give the requested number of ratings per hit because hits may fall into the "accept but not use". In a recent run 34% of hits fell into this bucket. This adds inconsistency to the ratings (not a constant number) and is also inefficient (wasted $). There are at least two solutions: