Adding tests in DIRAC¶
The runtest_dirac.py is an extension of the runtest library. The runtest library is a low-level program-independent library that provides infrastructure for running calculations and extracting and comparing numbers against reference outputs.
Reference outputs¶
Reference outputs are placed in directory “result/”.
Easy example¶
Let us look at an easy example.
First we load some standard modules and import functionality from the library (highlighted lines). This part is generic to all DIRAC tests.
#!/usr/bin/env python
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from runtest_dirac import Filter, TestRun
test = TestRun(__file__, sys.argv)
f = Filter()
f.add(from_string = '@ Elements of the electric dipole',
to_string = '@ anisotropy',
rel_tolerance = 1.0e-5)
f.add(from_string = '************ Expectation values',
to_string = 's0 = T : Expectation value',
rel_tolerance = 1.0e-5)
test.run(['PBE0gracLB94.inp', 'GLLBsaopLBalpha.inp'], ['Ne.mol'], f)
sys.exit(test.return_code)
Then we construct the filter object. It consists of two filter tasks. We can construct as many filter objects as we like and each can consist of as many tasks as we like.
#!/usr/bin/env python
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from runtest_dirac import Filter, TestRun
test = TestRun(__file__, sys.argv)
f = Filter()
f.add(from_string = '@ Elements of the electric dipole',
to_string = '@ anisotropy',
rel_tolerance = 1.0e-5)
f.add(from_string = '************ Expectation values',
to_string = 's0 = T : Expectation value',
rel_tolerance = 1.0e-5)
test.run(['PBE0gracLB94.inp', 'GLLBsaopLBalpha.inp'], ['Ne.mol'], f)
sys.exit(test.return_code)
With test.run we really run the job. Note how we pass the filter object. If we omit to pass it, then the calculations will be run but not verified. This is useful for multi-step jobs. Also observe how we give a list of input files and molecule files (in this case two input files and one molecule file). The test library will run and test all input/molecule file combinations (in this case two). We could have executed them separately in two lines. This would make no difference, just more typing.
#!/usr/bin/env python
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from runtest_dirac import Filter, TestRun
test = TestRun(__file__, sys.argv)
f = Filter()
f.add(from_string = '@ Elements of the electric dipole',
to_string = '@ anisotropy',
rel_tolerance = 1.0e-5)
f.add(from_string = '************ Expectation values',
to_string = 's0 = T : Expectation value',
rel_tolerance = 1.0e-5)
test.run(['PBE0gracLB94.inp', 'GLLBsaopLBalpha.inp'], ['Ne.mol'], f)
sys.exit(test.return_code)
Finally on the last line we exit with test.return_code. This is important. The integer test.return_code equals the number of failed test runs. It is zero if the test is successful.
Multi-step tests¶
Here is an example for a multi-step test. Note how only every second run is actually verified by passing the filter object.
#!/usr/bin/env python
import os
import sys
import shutil
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from runtest_dirac import Filter, TestRun
test = TestRun(__file__, sys.argv)
f = Filter()
f.add(from_string = 'Energy at final geometry is',
num_lines = 3,
rel_tolerance= 1.0e-4)
test.run(['O.inp'], ['O.mol'], args='--get=DFCOEF')
shutil.copy('DFCOEF', 'DFPROJ')
test.run(['H2O.inp'], ['H2O.mol'], f, args='--copy=DFPROJ')
test.run(['O.2c_iotc.inp'], ['O.mol'], args='--get=DFCOEF')
shutil.copy('DFCOEF', 'DFPROJ')
test.run(['H2O.2c_iotc.inp'], ['H2O.mol'], f, args='--copy=DFPROJ')
test.run(['O.2c_iotc_noamfi.inp'], ['O.mol'], args='--get=DFCOEF')
shutil.copy('DFCOEF', 'DFPROJ')
test.run(['H2O.2c_iotc_noamfi.inp'], ['H2O.mol'], f, args='--copy=DFPROJ')
# cleanup
os.unlink('DFCOEF')
os.unlink('DFPROJ')
sys.exit(test.return_code)
The other runs only serve to prepare files and are not checked (no filter passed as argument).
#!/usr/bin/env python
import os
import sys
import shutil
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from runtest_dirac import Filter, TestRun
test = TestRun(__file__, sys.argv)
f = Filter()
f.add(from_string = 'Energy at final geometry is',
num_lines = 3,
rel_tolerance = 1.0e-4)
test.run(['O.inp'], ['O.mol'], args='--get=DFCOEF')
shutil.copy('DFCOEF', 'DFPROJ')
test.run(['H2O.inp'], ['H2O.mol'], f, args='--copy=DFPROJ')
test.run(['O.2c_iotc.inp'], ['O.mol'], args='--get=DFCOEF')
shutil.copy('DFCOEF', 'DFPROJ')
test.run(['H2O.2c_iotc.inp'], ['H2O.mol'], f, args='--copy=DFPROJ')
test.run(['O.2c_iotc_noamfi.inp'], ['O.mol'], args='--get=DFCOEF')
shutil.copy('DFCOEF', 'DFPROJ')
test.run(['H2O.2c_iotc_noamfi.inp'], ['H2O.mol'], f, args='--copy=DFPROJ')
# cleanup
os.unlink('DFCOEF')
os.unlink('DFPROJ')
sys.exit(test.return_code)
Passing arguments to pam¶
This can be done with args. Example:
test.run(['O.inp'], ['O.mol'], args='--get=DFCOEF')
Catching expected errors¶
We have tests which fail with MPI or integer(4) compilation in a predictable and controlled way. In this case we don’t want to see the test failing, but we want it to pass.
Example:
#!/usr/bin/env python
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from runtest_dirac import Filter, TestRun
test = TestRun(__file__, sys.argv)
f = Filter()
f.add(string = 'Number of determinants/combinations')
f.add(string = ' Final energy')
test.run(['He.inp'],
['He.mol'],
f,
accepted_errors=['memory off-set too big for INTEGER*4',
'FATAL ERROR for LUCITA runs: memory offset (dynamic memory - static memory) is too big for i*4'])
sys.exit(test.return_code)
When we run this test as separate script, we see:
$ ./test -b ~/dirac/build/
running test: He He
found error which is expected/accepted: FATAL ERROR for LUCITA runs: memory offset (dynamic memory - static memory) is too big for i*4