< Return to the Challenge Details.
Judging Criteria
Final judging will be completed in one (1) round, however, judges may provide feedback prior to the close of the Submission Period in order for Contestants to upgrade and improve their Submissions. Any feedback provided is not an indication of Challenge status. All Submissions will be prescreened to insure they have correctly met the Submission Guidelines and Content Restrictions and meet with Sponsor’s general standards and practices prior to any judging. Please see the detailed rules.
After the Submission Deadline, all valid Submissions will be reviewed and scored by our panel of qualified judges (listed below), taking into consideration the following criteria:
Public Contributions to the Challenge Community
- Publication of the Submission prior to or on the early publication date of Oct 14, 2016 as set forth in the Timeline, if this possibly non-final version of the Submission already includes what ends up being the Submission’s substantial, novel, and/or potentially influential contributions.
- The Submission’s actual positive influence upon submissions by others.
- Efficient reuse of contributions from other contestants, with due acknowledgements. While such reuse is encouraged, we’ll favor submissions that bring their own substance to the Challenge and only reuse others’ ideas on top of that.
Quality, Completeness, and Further Potential
- Testability of the Submission, including self-testing capabilities.
- Inclusion of the Zcash API.
- Ease of integration in zcashd.
- Variety of systems the Submission can build/run on, such as other/older Linux distros (in particular, CentOS 6, which is still commonly found on servers) and non-Linux systems.
- SIMD instruction sets the Submission can target, if any (including SSE2 through AVX2, AVX-512, MIC, and non-x86 SIMD extensions).
- Range of Equihash parameters supported (including through easy compile-time settings). The Submission must work for n=200, k=9, but preferably also for other parameters, especially n=144, k=5.
- Tunability of the throughput/memory tradeoff. Support of multiple tradeoff ratios and whether they are optimal for different users.
- Scope, accuracy, and clarity of documentation included with the Submission (e.g., description of its design, rationale for design decisions and tradeoffs taken, limitations and shortcomings, expected performance).
- Evidence of the Submission’s further potential, including accurate documentation of such, continued commits to the repository (post submission deadline), and receipt of outside input on the Submission (such as through existing or attempted integrations and others’ use of the submitted code).
Performance
- Throughput (solution rate) of the Submission achieved under full or optimal utilization of the target device (e.g., when running the maximum number of concurrent threads supported on a CPU in hardware, unless peak cumulative throughput is reached at a lower number or some limitation or shortcoming prevents this code from easily running this many threads for a given Submission). RAM usage will not be considered as part of this criterion so long as RAM usage is under 28 GiB.
- Average and peak RAM usage of the Submission, as well as a cumulative throughput and memory usage figure such as solutions per GiB*s.
- For GPU implementations, the above applies separately both to host system RAM and to GPU card’s RAM (“global memory”) usage of the Submission.
- Duration and consistency of time to find a set of solutions. In a hypothetical massively parallel implementation processing a large number of inputs at once, this time might approach or exceed Zcash’s target block interval, thereby making the submission unsuitable for Zcash regardless of how high its throughput might be.
- Percentage of all possible solutions that the Submission actually finds. This will commonly be almost or exactly 100%, but it doesn’t strictly have to be (and the documentation should state what it is and include the rationale).
- CPU/GPU utilization of the Submission. Lower utilization of the target device commonly indicates the submission is currently suboptimal, but also suggests there’s potential for improvement, or in other cases it may correspond to an energy-efficient point.
While the performance criteria are listed last, they are nevertheless crucial and we hope it’s the performance criteria that will decide the winner(s) among the community-friendly, high-quality submissions.
The performance will be tested (and the above criteria applied) with Equihash parameters n=200, k=9, and, if supported, also with n=144, k=5. Submissions that perform well at both of these parameter sets will be favored over those that perform well at only one of them.
Testing will be done on a variety of systems. The reference system for testing of CPU implementations will likely be Core i7-4770K, 32 GB RAM (dual-channel DDR3-1600, 4x 8 GB modules), CentOS 7. As portability and capabilities of the submissions permit, they may also be tested on other/dissimilar systems including dual-socket server platforms and Xeon Phi.
GPU implementations will be tested on a variety of GPUs, both NVIDIA and AMD, in a variety of systems, so portability of the corresponding host code (also to be submitted) across at least a wide variety of Linux distros is highly desirable.