2. HPL and HPCG (15 points)
The proposal should include descriptions of the software environment (operating system, complier, math library, MPI software, software version, etc.), the testing method, performance optimization methods, performance estimation, problem and solution analysis, etc. In-depth analysis on HPL, HPCG algorithm and the source code would be a plus.
Download the HPL software at: http://www.netlib.org/benchmark/hpl/.
Download the HPCG software at: https://github.com/hpcg-benchmark/hpcg
It is recommended to run verification and optimization of HPL and HPCG on x86 Xeon CPU or Tesla GPU platforms. Furthermore, teams that have to use other hardware platforms are welcome to submit their analysis and results if there is a reasonable performance.

3. Language Exam (LE) Challenge (30 points)

The Task

Teaching machines to understand human language documents is one of the most elusive and long-standing challenges in artificial intelligence [1]. In order to do so, different tasks in various aspects should be solved like part-of speech tagging, named entity recognition, syntactic parsing, coreference resolution et al.

Since the success of deep learning in other areas like computer vision, designing a deep neural network to understand human language is also a long been concerned approach. Bert, a transformer [3] style architecture deep neural network released in 2018, is the first language model that successfully applied to variety of language tasks such as sentimental analysis, question answering, named entity recognition et al. Since then, some other transformer style models, like XLNet[4], transformer-xl[5], RoBERTa[6], GPT2[7] also published in the recent year.

There are variety of tasks and datasets designed to evaluate how well the deep neural network understand the human language. Most of the tasks, like SQuAD [8], are either crowd-sourced or automatically-generated, bringing a significant amount of noises in the datasets and limits the ceiling performance by domain experts. English language exam, is a comprehensive task designed by teacher to assess the level of the student of the understanding about the English. Commonly used tasks in the exam including listening comprehension, cloze, reading comprehension and writing. Using these teacher-designed human-oriented tasks to evaluate the performance of neural networks is straightforward and also challengeable.

In the preliminary round of ASC20, a cloze style dataset is provided. The dataset is collected from internet and contains multi-level English language exams in China including high school level and college entrance exams, CET4 (College English Test 4), CET6 (College English Test 6) and NETEM (National Entrance Test of English for MA/MS Candidates). Part of the data comes from public CLOTH dataset [9]. Among the dataset, there are 4603 passages and 83395 questions in the training set, 400 passages and 7798 questions in the dev set and 400 passages and 7829 questions in the test set. The participants should design and train a neural network to achieve the best performance on the test set.

Dataset

Download dataset at Baidu: https://pan.baidu.com/s/1t7miv2h2MyEmY191y6lXOA (password: fhd4) or Microsoft OneDrive https://1drv.ms/u/s!Ar_0HIDyftZTsBiXPJUtGfiMmTom?e=w7VPAx.
Below is a sample of the training set. Each data is organized into a json file, the json file contain 3 elements: “article”, “options” and “answers”. “_” is used as the placeholder in the article.

The only difference between train/evaluation datasets and test dataset is that the test dataset only contains 2 elements: article and options. The answers of the test dataset are reserved by the committee for scoring.

Result Submission

The participant should organize all its results into a single json file and follow the format below:

For both preliminary and final rounds, each team should also submit a folder that contains source code and model that could reproduce the test results. The folder structure should be like:

Evaluation
In the preliminary, the score is calculated based on the formula below:

•  is the score based on the test set, and  is the highest score on the test set among all valid submissions.
• 60 is used as the baseline score as it is the most commonly used score to measure whether the student passes the exam.
•  should be higher than 60, otherwise the score  will be 0.

Training Framework and baseline code

The participants must use PyTorch framework (https://pytorch.org/) for this task. The submission using or depending on any other deep learning framework will be forfeited.

The committee do not supply any baseline code for this task, the participants should design their deep learning network based on public resources. The participants should also consider the training performance of their neural network. Using distributed training strategies like data-parallelism and model-parallelism to accelerate the training process is encouraged, as the training performance on the participant-designed HPC cluster will be used as one of the metrics for scoring in the final.

Hardware requirement

It is highly recommended to run the training code on a GPU platform.

Reference

• Danqi Chen, Neural reading comprehension and beyond. https://purl.stanford.edu/gd576xb1833
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://arxiv.org/abs/1810.04805
• Attention Is All You Need https://arxiv.org/abs/1706.03762
• XLNet: Generalized Autoregressive Pretraining for Language Understanding https://arxiv.org/abs/1906.08237
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context https://arxiv.org/abs/1901.02860
• RoBERTa: A Robustly Optimized BERT Pretraining Approach https://arxiv.org/abs/1907.11692
• Better Language Models and Their Implications https://openai.com/blog/better-language-models/
• https://rajpurkar.github.io/SQuAD-explorer/
• Large-scale Cloze Test Dataset Created by Teachers  https://arxiv.org/abs/1711.03225

4. The QuEST Challenge (30 points)

The Task

Because of the huge cost of computational resource and elapsed time, numerous problems, such as cryptology, many body quantum mechanics and quantum machine learning, still can’t be solved effectively even utilizing those powerful supercomputers. However, the quantum computer, which is based on the quantum mechanics, shows its great advantage in these fields benefiting from the suitable quantum algorithms. Among those quantum algorithms, the Shor large number decomposition algorithm as well as the Grover quantum search algorithm are the two most famous algorithms, which gain intense interest from the scientists and inspire the development of quantum computing and quantum information. Unfortunately, the achievement of the quantum computer hardware is so slow that there is no actual quantum computer which can be used for solving the science or cryptology issues. Therefore, so far, the quantum system is still costly simulated by using the classical computers.

QuEST is the first open source, hybrid multithreaded and distributed, GPU accelerated simulator of universal quantum circuits. QuEST is capable of simulating generic quantum circuits of general one and two-qubits gates and multi-qubit controlled gates, on pure and mixed states, represented as state-vectors and density matrices, and under the presence of decoherence.

Describing the state of an n-bit classical register requires n bits, but describing the state of an n-qubit quantum register requires 2n complex numbers. Consequently, simulating a quantum computer using a classical machine is believed to be exponentially costly with respect to the number of qubits. Despite this, classical simulation of quantum computation is vital for the study of new algorithms and architectures.

The mathematical principle and algorithm for QuEST have been fully described in Reference [1]. We strongly recommend using the stable version of QuEST_2.1.0 and the corresponding source code is available in Reference [2]. More information about installation and usage of QuEST can be found from Reference [3].

In the preliminary round of ASC20, all teams are encouraged to complete the simulation of quantum circuits composed of 30 qubits by using the provided quantum random circuit (random.c) and the quantum Fourier transform circuits (GHZ_QFT.c). The memory requirement is at least 16GB. Actually, such circuits are intended for cracking the RSA encryption algorithm. In this challenge, you should obtain the right results and make efforts to reduce the computational needs, both in time and resources. The proposal document should include descriptions of the software environment (operating system, complier, math library, MPI software, and QuEST version, etc.), the testing method, performance optimization methods, performance estimation, problem and solution analysis, etc. In-depth analysis into QuEST's algorithm and source code is highly encouraged. The detailed tasks and requirements of this challenge are listed below.

Compile and install QuEST and run the program against the given data according to the instructions.

In order to compile and install QuEST, you may refer to the following steps:

The source code of QuEST can be obtained by:
The QuEST is most easily downloaded by using apt, git and GNU make, which can first be obtained with “sudo apt install git make (Ubuntu) or yum install git make (Redhat)”; then QuEST can be downloaded to the current directory (path/ to/QuEST) with “git clone https://github.com/QuEST-Kit/QuEST.git”.
Or you can directly download the compressed file of QuEST (.tar.gz) at https://github.com/QuEST-Kit/QuEST/releases.

Install QuEST and run the challenge tests
Three necessary files, including the mytimer.hpp, random.c and GHZ_QFT.c, are provided and can be downloaded through Baidu SkyDrive or Microsoft OneDrive, same as the Language exam (LE) challenge listed above. In this challenge, two workloads should be completed: for the first workload named random circuit, mytimer.hpp, random.c and the source code of QuEST will be used, while for the other workload named GHZ_QFT, the mytimer.hpp, GHZ_QFT.c and the source code of QuEST will be used. Due to the independence of the two workloads, QuEST should be installed separately to run each workload.

For the workload named random circuit:
“cp random.c tutorial_example.c”
“cd build”
“cmake ..”
“make –j4”
“./demo”
Note this is an example to show how to install and run the tests. The participants can compile the code by using their optimizing strategies or run the test with MPI or OpenMP.
The other workload (GHZ_QFT.c) can be installed the same way mentioned above, except for “cp GHC_QHF.c tutorial_example.c” (instead of “cp random.c tutorial_example.c”).

Result Submission
Please submit all the requested files of each workload in following formats:

In the proposal, please describe test platform include hardware configuration and architecture, and run time for each step (submission of a log file is needed). Also the compiling process of the package and your modifications of the source code, how and why, should be described. Describe the strategies used to optimize the performance in the test process. The modified code should be submitted along with the proposal, which gives strong support for verifying the correctness of optimizing strategies.

Evaluation

In each workload, two output files will be generated: probs.dat and stateVector.dat. The former describes the probability of every qubit that equals to 1, while the latter provides the amplitude of the first ten state vectors, which are complex numbers including real and imagine parts. The evaluation follows the rules as below:
1. The output files in each workload should be theexactly same as the given references of probs.dat and stateVector.dat. The corresponding references can be obtained through Baidu SkyDrive or Microsoft OneDrive as listed above;
2. Then the execution time will be evaluated by the number listed in the screen output file *.log on the ground that 1st condition is fulfilled. However, the computing platforms that used by participants have great influence on the execution time, such influence will be considered when giving the score.
3. The score of two workloads is equal because of the independence of them.
4. Proposals with clarity as well as rigorous descriptions are highly appreciated and beneficial for higher scores.

Hardware requirement

In these two contest cases, if GPU accelerating option is used, the results may be not correct. If teams decide to accelerate the task using GPU, please check the output files and be sure it is the exactly same as the given references of probs.dat and stateVector.dat.  

Reference
[1]. QuEST description:
Quest and high performance simulation of quantum computers. Jones T, Brown A, Bush I, et al. Scientific reports, 2019, 9(1): 1-11.
[2]. QuEST_v2.1.0 source code:
https://github.com/QuEST-Kit/QuEST/releases
[3]. QuEST userguide:
https://quest.qtechtheory.org/docs/

For any further questions, please contact techsupport@asc-events.org