Testing Apache with the Perl Test Harness
These two modules must first be installed;
You'll need to install the CPAN modules listed in: Apache-Test/lib/Bundle/ApacheTest.pm All you have to do to install them all in one shot is: perl -MCPAN -e 'install Bundle::ApacheTest'
Which are also available in one tarball here: http://perl.apache.org/~dougm/httpd-test-bundle-0.02.tar.gz
Note: Crypt::SSLeay requires OpenSSL to be installed (only required for t/TEST -ssl): http://www.openssl.org/ More accurate results may be obtained by using the same openssl command line and libraries as consumed by APR-util and mod_ssl, due to X509 formatting behavior differences.
For an extensive documentation see http://perl.apache.org/docs/general/testing/testing.html or http://svn.apache.org/viewvc/perl/modperl/docs/trunk/src/docs/general/testing/testing.pod
To run the tests for all Apache web server modules, some additional CPAN modules will be required. If the tests don't work, make sure that you have up to date versions of each of these perl modules:
cpan App::cpanminus
cpanm Bundle::ApacheTest \
HTTP::DAV DateTime Time::HiRes \
Test::Harness Crypt::SSLeay Net::SSLeay IO::Socket::SSL \
IO::Socket::IP IO::Select LWP::Protocol::https AnyEvent \
AnyEvent::WebSocket::Client LWP::Protocol::AnyEvent::http FCGI
If you don't care how it works and just want to run the tests, here's how you go about doing that.
The test harness will run every .t file under the t/ directory. So let's say you only want to run the tests for PHP. Do this: t/TEST -httpd /path/to/apache/bin/httpd t/php
That will start the test server, run the .t tests under t/php and shut down the test server. You can also control each of these steps.
This will start the test server: t/TEST -httpd /path/to/apache/bin/httpd -start
This will run the PHP tests in the test environment: t/TEST t/php
This will stop the test server: t/TEST -stop
This will run the server under gdb (using -X): t/TEST -d gdb
Note: At this point, you have a working test environment. You can look in t/conf for the test server configuration files. These are generated by the test harness. Once you have a working test environment, you do not need to specify 'httpd' on the t/TEST command line. For instance, to start the server up again, the command t/TEST -start would be sufficient.
For a full regression test, you should have all modules loaded. Build the server with configure --enable-modules=reallyall --enable-load-all-modules ... among other things. Edit the generated httpd.conf and comment all mpm modules that you do not want. Run "t/TEST -clean" again.
You will see some skipped: cannot find module 'XXX' as not all modules are in every apache release (but the tests run for all).
All in all, some >4k tests will run and the result needs to be: PASS
If you have a "PASS" at the end of "t/TEST", congratulations! If not, this sections gives some advise in order to find the cause. Feel free to expand this to make life easier for others.
t/TEST -clean PATH=:$PATH t/TEST If a lot of ssl tests fail, check in the error log for the presence of a certificate validation error. If you find it, check the expiration date of the TLS/SSL certificates used by the tests, they might be expired. Running TEST -clean should delete the old ssl certificates, so they'll be regenerated during the next run.
Sometimes it's possible that the test is passing properly for the first time, when it's run for the first time in the thread. But when you run it again, the test might fail. It's important to run the repetition smoke testing. For example to repeat the tests 5 times you can run:
t/SMOKE -times=5
It's also possible that a test will pass when it's run after a particular test, but if moved to run after a different state it may fail. For this reason by default the tests run in random order.
Since it's important to be able to reproduce the problem with the random testing, whenever -order=random is used, the used seed is printed to STDERR. Which can be then fed into the future tests with: via APACHE_TEST_SEED environment variable.
By adding the option -order=repeat, the tests will be run in alphabetical order.
Combining these two important smoke testing techniques, one can run tests with:
t/SMOKE -times=N -order=(repeat|random)
For example, to run the mod_rewrite tests 5 times, one would:
t/SMOKE -times=5 -verbose t/modules/rewrite.t
So the tests can be repeated N times, and run in the following three modes:
For configuration options and default settings run:
t/SMOKE -help
For more information refer to the Apache::TestSmoke manpage.
The test server is configured with conf files like any normal Apache server. The tricky part is those conf files are generated by the harness just prior to starting the server. t/conf/httpd.conf is generated by t/conf/httpd.conf.in. If that does not exist, the harness will generate a working configuration and will include LoadModule (and AddModule for Apache 1.3) directives from the httpd.conf associated with the httpd binary you are using for testing. If t/conf/extra.conf.in exists, t/conf/extra.conf will be generated from that, and an Include directive for that file will be put in the generated t/conf/httpd.conf. t/conf/apache_test_config.pm is generated from the test configuration. It contains all the information about the configuration of your test server. You can access this information in test scripts by: my $env = Apache::TestConfig->thaw; Apache::TestConfig access apache_test_config.pm and returns a hash reference with all the information. Look through apache_test_config.pm, it's a lot of stuff. Once these conf files are generated, you have a working test environment, and they must be 'cleaned' if you wish to make changes to them. To clean the environment: t/TEST -clean (Now you will have to specify your httpd binary when starting back up again.)
For more information on using the test harness and writing tests, see the README in Apache-Test and the examples in Apache-Test/t.
The test harness was originally written by Doug MacEachern and is discussed on the httpd dev mailing list ([email protected]).
It is also included in modperl-2.0 source along with tests for modperl-2.0.