We need to inform the community that there was a minor error in our ranking algorithm implementation. This caused 2 apps to receive a ranking lower than they should have. Blockusign was paid at rank 12, even though it should have been 10th. Sundly was paid at rank 13, even though it should have been 12th. This caused these apps to receive less in payment than they deserved.
The technical cause of this mistake is that we calculated the “theta” function twice for each app reviewer’s final score. This is described in step 4 in the paper, titled “Aggregation across the Reviewers”. We should have only calculated the theta function once , but in our mistake, we calculated it twice .
We plan to correct this issue by paying these apps more Bitcoin and adjusting their ranking on the official site to make up for the difference in rewards that they received. We will not ask for payment back from any apps that mistakenly received more Bitcoin than they would have in their correct ranking.
We also are developing a plan to ensure the quality of our rankings before we publish and pay apps. This includes more diligent, double-blind, internal quality assurance, as well as potentially involving community members to participate in the rankings calculations openly. These steps will double and triple check the work, and hopefully mitigate issues like this in the future.
The correct rankings are listed in our public Google sheet, which includes formulas and raw data, and shows how the final rankings are formed.
We have also updated the public rankings on app.co, to reflect the correct rankings and payment amounts.
We launched the program as a pilot to get ahead of and prevent future accidents such as these, and are grateful for your support and feedback to help us improve. We apologize that this issue happened, but glad that it is making the process more rigorous. We know that app mining is built on trust, and without that trust, app mining is less useful. We will continue to be transparent in our ranking process, and will continue to share raw data in future rankings.
Thank you for participating in app mining.