C96b4 test suite
Home Up C94.00 Other versions Revision history The future etcetera Acknowledgements Contacts, Mailing list Links Site map, search




Reliability in the face of complexity is the real challenge of software development.  I discuss this in this paper.

Check that the code is set up properly

Run the simple test

Execute the code with the single command line


To do this create a file ("test.in") and enter this one line as the first and only line in the file.  Then execute the code with
cloudy.exe < test.in > test.out
assuming the executable was named cloudy.exe..

Now examine the last line of output in the file that was produced.  This test does many internal sanity checks. If the output says that Cloudy ended OK then things are set up OK. The following shows the last line of output produced in a successful run:

 [Stop in maincl, Cloudy exited OK]

Check that it crashes on errors

It would be good if the code would crash on divide by zero, overflow, using NaN, and (for the debug compile) a failed assert.  Compile as directed above, and run the following four tests:

title confirm it crashes on overflow
crash overflow

title confirm it crashes on divide by zero
crash zero

title confirm it crashes when using NaN
crash NaN

title confirm it crashes with a failed assert
crash assert

Note that the failed assert will only occur when the code is compiled in debug mode - asserts do not exist in optimized code.  If the code does not crash on each of these tests, then your system is quite happy with bad floating point math or failed asserts.

It is quite possible that you will not be able to get your system to crash on floating point errors.  (This seems to get a bit harder every year.)  This is not a major problem but is something that should be kept in mind.

Download and explode the test cases

The current version of the test suite can be downloaded from here.in unix, and here in windows format.   Explode this file into its own directory, perhaps one called tests.  Individual test cases can be studied to see how to set up the code. 

The readme_tests.htm file describes several perl scripts that are included in the documentation.

The assert commands within the test suite

The code uses extensive self-checking during a calculation to insure that the results are valid.  All of the test cases include assert commands in the input stream.  These have nothing to do with the simulation or the astrophysics, but instead provide a way for the code to do extensive automatic self-checks.  The perl script tests_remove_asserts.pl will remove all the asserts if you wish to resuse these scripts for other purposes.

A large code must be completely checked every time anything is changed.  The assert commands tell the code what answer to expect, based either on analytical or previous results. They allow the code to confirm that it has found the correct answer, and a distinctive string is printed if the right answer is not obtained.  The perl script checkall.pl will examine all output files to search for failed asserts, which indicated a problem.

The asserts should not be included in actual calculations.  They are only here for verification.

Run the test cases 

The test suite includes Perl scripts to run all the test cases and then check for problems.  These are files that have names  *.pl.  

The runall.pl script will run all the input files.  You will need to edit runall.pl so that the string "$exe" points to your executable for Cloudy.  This test suite takes about seven hours on my 2GHz mhz pc, so you might want to do this overnight.

Check the results

The code uses extensive self-checking to insure that the results are valid.  Many of these tests include assert commands in the input stream which allow the code to determine whether it has found the correct answer.  If anything goes wrong the code will announce this at the end of the calculation by printing a line that includes the string "botched asserts".

The test distribution includes the Perl script checkall.pl that will check for problems once the test cases have been computed.  

The tests atlas.in costar.in costarhalo.in rauch.in and werner.in require the compiled stellar atmosphere files.  These are not expected to execute properly if you choose not to set up the stellar atmospheres.

Test cases output

These files contain the results of the test suite. Results are available for the following platforms,
PC (XP, Visual Studio .net 7, PC end of lines - the rest of the tests have Unix eol),
Linux, gcc AMD, computed by Peter van Hoof
Alpha ccc,
SGI cc, (not yet)
Sun Solaris computed by Peter van Hoof

Note that the code that generated this output did not have the hot fixes included. 

Next step, apply any hot fixes.

Hit Counter
Last changed 03/31/03.
Return to the Cloudy Home Page.
Copyright 1978-2003 Gary J. Ferland