Grading Methodology for Run-Time Errors

A score of 0 is given when the error was not detected.

A score of 1 is given for error messages with the correct error name.

A score of 2 is given for error messages with the correct error name and line number where the error occurred but not the file name where the error occurred.

A score of 3 is given for error messages with the correct error name, line number and the name of the file where the error occurred.

A score of 4 is given for error messages which contain the information for a score of 3 but less information than needed for a score of 5.

A score of 5 is given in all cases when the error message contains all the information needed for the quick fixing of the error.


Results

You can sort the tables below by clicking on the table headers with one click sorting from best to worst and another click to sort from worst to best.

FORTRAN results

OS/OpenMP Compiler/ Runtime Tool Deadlock Race Conditions Environment and Clause Errors Wrong Order of OpenMP Directives Uninitialized Shared and Private Variables Wrong Usage of OpenMP Runtime Library Routines Implementation Dependent Errors Average
IBM AIX/ IBM XLF 0.24 0.00 0.27 0.52 0.02 0.00 0.77 0.26
Unicos/ PGI pgf90 0.00 0.00 0.40 0.00 0.02 0.04 0.70 0.17
Unicos/ GNU gfortran 0.03 0.00 0.40 0.19 0.01 0.00 0.74 0.20
Unicos/ Pathscale pathf90 0.00 0.00 0.21 0.00 0.17 0.19 0.37 0.13
Linux/ Pathscale pathf90 0.00 0.00 0.21 0.00 0.14 0.19 0.37 0.13
UNICOS/ Cray Fortran 0.02 0.02 0.00 0.00 0.03 0.31 1.20 0.23
UNICOS/ Cray Fortran 0.02 0.00 0.00 0.00 0.00 0.31 1.25 0.23
IRIX64/ MIPSpro Fortran 0.18 0.00 0.15 0.39 0.19 0.29 0.53 0.25
Solaris/i86/ Sun Fortran95 0.00 0.00 0.00 0.00 0.04 0.00 0.09 0.02
Solaris/i86/ Sun Fortran95/ Thread Analyzer 0.00 2.67 0.00 0.00 0.04 0.00 0.10 0.40
Linux/ Intel Fortran 9.0 0.00 0.01 0.00 0.00 0.03 0.24 0.65 0.13
Linux/ Intel Fortran 9.0/ Thread Checker 0.00 1.09 0.40 0.00 0.30 0.40 0.77 0.42

C results

OS/OpenMP Compiler/ Runtime Tool Deadlock Race Conditions Environment and Clause Errors Wrong Order of OpenMP Directives Uninitialized Shared and Private Variables Wrong Usage of OpenMP Runtime Library Routines Implementation Dependent Errors Average
IBM AIX/ IBM XLC 0.24 0.00 0.15 0.39 0.04 0.00 0.82 0.23
Unicos/ PGI pgcc 0.00 0.00 0.43 0.05 0.05 0.04 0.76 0.19
Unicos/ GNU gcc 0.03 0.00 0.36 0.18 0.02 0.00 0.72 0.19
Unicos/ Pathscale pathcc 0.00 0.00 0.31 0.11 0.23 0.22 0.39 0.18
Linux/ Pathscale pathcc 0.00 0.00 0.31 0.13 0.31 0.22 0.39 0.19
UNICOS/ Cray C++ 0.03 0.00 0.00 0.23 0.07 0.31 1.43 0.30
UNICOS/ Cray C++ 0.03 0.00 0.00 0.00 0.00 0.31 1.43 0.25
IRIX64/ MIPSpro C 0.24 0.00 0.21 0.50 0.30 0.27 0.54 0.29
Solaris/i86/ Sun C 0.00 0.00 0.00 0.11 0.06 0.00 0.00 0.02
Solaris/i86/ Sun C/ Thread Analyzer 0.00 2.54 0.00 0.13 0.05 0.00 0.00 0.39
Linux/ Intel C++ 9.0 0.00 0.00 0.00 0.08 0.05 0.09 0.73 0.14
Linux/ Intel C++ 9.0/ Thread Checker 0.03 1.55 0.12 0.00 0.38 0.13 0.80 0.43

C++ results

OS/OpenMP Compiler/ Runtime Tool Deadlock Race Conditions Environment and Clause Errors Wrong Order of OpenMP Directives Uninitialized Shared and Private Variables Wrong Usage of OpenMP Runtime Library Routines Implementation Dependent Errors Average
IBM AIX/ IBM XLC 0.24 0.00 0.23 0.33 0.05 0.00 1.22 0.30
Unicos/ PGI pgCC 0.00 0.00 0.00 0.04 0.02 0.04 1.35 0.21
Unicos/ GNU g++ 0.03 0.00 0.36 0.40 0.02 0.00 1.08 0.27
Unicos/ Pathscale pathCC 0.00 0.00 0.31 0.19 0.21 0.22 0.55 0.21
Linux/ Pathscale pathCC 0.00 0.00 0.31 0.31 0.29 0.22 0.55 0.24
UNICOS/ Cray C++ 0.03 0.01 0.00 0.28 0.10 0.31 2.40 0.45
UNICOS/ Cray C++ 0.03 0.00 0.00 0.05 0.00 0.31 2.40 0.40
IRIX64/ MIPSpro C++ 0.24 0.00 0.19 0.42 0.31 0.27 0.81 0.32
Solaris/i86/ Sun C++ 0.00 0.00 0.00 0.12 0.08 0.00 0.00 0.03
Solaris/i86/ Sun C++/ Thread Analyzer 0.00 2.55 0.00 0.15 0.08 0.00 0.00 0.40
Linux/ Intel C++ 9.0 0.00 0.00 0.00 0.15 0.07 0.09 1.08 0.20
Linux/ Intel C++ 9.0/ Thread Checker 0.03 1.56 0.12 0.10 0.52 0.13 1.15 0.52

.

.

.

.

.

.

Computer info:
DataStar
IBM p-Series
University of California, San Diego
MPI:
Parallel Environment for AIX 5L V4.1
Compiler:
IBM(R) XL Fortran Enterprise Edition V9.1o
Compiler options used:
-qsuffix=f=f90 -qcheck -qsource
-g -qsigtrap=xl__trce
-qflttrap=enable:zerodivide:invalid:overflow
Environment variables:
MP_EUIDEVELOP=yes
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
pgf90 7.0-6 64-bit target on x86-64 Linux
Compiler options used:
-mp -Mconcur=nonuma -O0 -C -g -Ktrap=fp -Mbounds -Mchkfpstk -Mchkstk -Mchkptr
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
gfortran V. 4.2.0
Compiler options used:
-fopenmp -frange-check -O0 -fbounds-check -g -ffpe-trap=invalid,zero,overflow -fdiagnostics-show-location=every-line -fcray-pointer
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
pathf90 V. 3.0
Compiler options used:
-mp -O0 -g -trapuv -TENV:X=0 -Wdiv-by-zero -C
Computer info:
Lightning - Linux AMD Opteron Cluster
Linux 2.6.9-34.ELsmp
Compiler:
pathf90 V. 2.4
Compiler options used:
-mp -O0 -g -trapuv -TENV:X=0 -Wdiv-by-zero -C
Computer info:
Cray X1
UNICOS/mp 3.1.24
Compiler:
Cray Fortran : Version 5.6.0.0.28
Compiler options used:
-O1 -O task1 -m2 -rl -Rabcs
Computer info:
Cray X2
UNICOS/mp 3.1.24
Compiler:
Cray Fortran : Version 6.0.0.1.2
Compiler options used:
-O1 -O task1 -m2 -rl -Rabcs
Computer info:
SGI Origin 2000
IRIX64 6.5
Compiler:
MIPSpro Compilers f90 : Version 7.4.2m
Compiler options used:
-O0 -g -DEBUG:div_check=3:subscript_check=ON:
conform_check=ON:varargs_prototypes=ON:
varargs_interface_check=ON:
verbose_runtime=ON:fullwarn=ON
Computer info:
Opteron
Solaris 10 i86pc
Iowa State University
Compiler:
Sun Fortran 95 8.3 2007/02/08
Compiler options used:
-xopenmp=noopt -g
Computer info:
Opteron
Solaris 10 i86pc
Iowa State University
Compiler:
Sun Fortran 95 8.3 2007/02/08
Compiler options used:
-xinstrument=datarace -xopenmp=noopt -g
Runtime Tool:
Sun Studio 12 - Thread Analyzer
Computer info:
Intel Pentium 4
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) Fortran Compiler Version 9.0
Compiler options used:
-openmp -g -O0
Computer info:
Intel Pentium 4
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) Fortran Compiler Version 9.0
Compiler options used:
-openmp -g -O0
Runtime Tool:
Intel Thread Checker
.
Computer info:
DataStar
IBM p-Series
University of California, San Diego
San Diego Supercomputer Center
Compiler:
IBM(R) XL C/C++ Enterprise Edition V7.0
Compiler options used:
-qheapdebug -qinfo -qcheck -g -qsource
-qflttrap=enable:zerodivide:invalid:overflow
Environment variables:
MP_EUIDEVELOP=yes
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
pgcc 7.0-6 64-bit target on x86-64 Linux
Compiler options used:
-mp=nonuma -c99 -O0 -C -g -Mdepchk -Ktrap=fp -Mbounds -Mchkfpstk -Mchkstk -Msignextend
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
pathcc V. 3.0
Compiler options used:
-mp -O0 -g -trapuv -TENV:X=0 -Wchar-subscripts -Wdiv-by-zero -std=c99
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
gcc V. 4.2.0
Compiler options used:
-fopenmp -std=c99 -O0 -fbounds-check -g -ftrapping-math -fsignaling-nans -fcx-limited-range
Computer info:
Lightning - Linux AMD Opteron Cluster
Linux 2.6.9-34.ELsmp
Compiler:
pathcc V. 2.4
Compiler options used:
-mp -O0 -g -trapuv -TENV:X=0 -Wdiv-by-zero -C
Computer info:
Cray X1
UNICOS/mp 3.1.24
Compiler:
Cray C++ : Version 5.6.0.0.89
Compiler options used:
-O1 -h omp -h task1 -h bounds -h matherror=abort -h c99 -h exceptions
Computer info:
Cray X2
UNICOS/mp 3.1.24
Compiler:
Cray C++ : Version 6.0.0.1.4
Compiler options used:
-O1 -h omp -h task1 -h bounds -h matherror=abort -h c99 -h exceptions
Computer info:
SGI Origin 2000
IRIX64 6.5
Compiler:
MIPSpro Compilers cc: Version 7.4.2m
Compiler options used:
-O0 -g
Computer info:
Opteron
Solaris 10 i86pc
Iowa State University
Compiler:
Sun C 5.9 2007/02/08
Compiler options used:
-xopenmp=noopt -g
Computer info:
Opteron
Solaris 10 i86pc
Iowa State University
Compiler:
Sun C 5.9 2007/02/08
Compiler options used:
-xinstrument=datarace -xopenmp=noopt -g
Runtime Tool:
Sun Studio 12 - Thread Analyzer
Computer info:
Intel Pentium 4
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-openmp -g -O0
Computer info:
Intel Pentium 4
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-openmp -g -O0
Runtime Tool:
Intel Thread Checker
Computer info:
DataStar
IBM p-Series
University of California, San Diego
San Diego Supercomputer Center
Compiler:
IBM(R) XL C/C++ Enterprise Edition V7.0
Compiler options used:
-cpp -qheapdebug -qinfo -qrtti
-qcheck -g -qsource
-qflttrap=enable:zerodivide:invalid:overflow
Environment variables:
MP_EUIDEVELOP=yes
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
pgCC 7.0-6 64-bit target on x86-64 Linux
Compiler options used:
-mp=nonuma -O0 -C -g -Mdepchk -Ktrap=fp -Mbounds -Mchkfpstk -Mchkstk -Msignextend --exceptions
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
pathCC V. 3.0
Compiler options used:
-mp -O0 -g -trapuv -TENV:X=0 -Wchar-subscripts -Wdiv-by-zero
Computer info:
Cray XT4 (Opteron)
Unicos/LC
Compiler:
g++ V. 4.2.0
Compiler options used:
-fopenmp -O0 -fbounds-check -g -ftrapping-math -fsignaling-nans -fcx-limited-range
Computer info:
Lightning - Linux AMD Opteron Cluster
Linux 2.6.9-34.ELsmp
Compiler:
pathCC V. 2.4
Compiler options used:
-mp -O0 -g -trapuv -TENV:X=0 -Wdiv-by-zero -C
Computer info:
Cray X1
UNICOS/mp 3.1.24
Compiler:
Cray C++ : Version 5.6.0.0.89
Compiler options used:
-O1 -h omp -h task1 -h bounds -h matherror=abort -h exceptions -h scalar1
Computer info:
Cray X2
UNICOS/mp 3.1.24
Compiler:
Cray C++ : Version 6.0.0.1.4
Compiler options used:
-O1 -h omp -h task1 -h bounds -h matherror=abort -h exceptions -h scalar1
Computer info:
SGI Origin 2000
IRIX64 6.5
Compiler:
MIPSpro Compilers CC: Version 7.4.2m
Compiler options used:
-O0 -g
Computer info:
Opteron
Solaris 10 i86pc
Iowa State University
Compiler:
Sun C++ 5.9 2007/02/08
Compiler options used:
-xopenmp=noopt -g
Computer info:
Opteron
Solaris 10 i86pc
Iowa State University
Compiler:
Sun C++ 5.9 2007/02/08
Compiler options used:
-xinstrument=datarace -xopenmp=noopt -g
Runtime Tool:
Sun Studio 12 - Thread Analyzer
Computer info:
Intel Pentium 4
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-openmp -g -O0
Computer info:
Intel Pentium 4
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-openmp -g -O0
Runtime Tool:
Intel Thread Checker