Package com.smartgwt.client.docs

# Interface TestRunner

public interface TestRunner

### TestRunner

The Smart GWT TestRunner is a system for running a suite of Selenium tests on a periodic basis, comparing the results to previous results, and generating email alerts reporting on new test failures or fixes to tests that were previously failing.

TestRunner is a key piece of implementing the Continuous Integration methodology, whereby continuous testing is applied so that regressions are caught immediately. This allows a product or application to be kept continuously at a very high level of quality, allowing for more frequent and predictable releases.

### Database Setup

Each time TestRunner executes it by default stores results to a SQL database via two SQLDataSources:

- batchRun.ds.xml: stores global information about the run as a whole: an ID for the run, when it started and ended, and optional data to be used in emails generated by the system.
- testResult.ds.xml: stores the result of each individual test, including when it started and ended, and information about errors that occurred, if any

These DataSources are present in the `selenium/` directory of your SDK. If you choose to move them elsewhere, simply update the DataSources location (configured by `project.datasources` in `server.properties`).

These DataSources behave just like other SQLDataSources:

- they are compatible with all the database types that Smart GWT supports
- they will use the default database configured for your project, or you can set `DataSource.dbName` in the .ds.xml file to use a second database instead
- you can setup the database connection and generate SQL tables using the `AdminConsole`
- you can build your own UI for viewing test results, by loading the `batchRun` and `testResult` DataSources like any other SQLDataSource and binding components such as ListGrids to them.
- if you deploy an application that includes these DataSources, third-party tools can access these DataSources via the RESTHandler servlet

If needed, the IDs of these DataSources can be configured via the `server.properties` settings `autotest.batchRunDS` and `autotest.testResultDS`.

Note: If you use the default server.properties shipped with the SDK, TestRunner and the SDK web server will share a common SQL database, so that the web server and TestRunner cannot both run at once. This means that you must point TestRunner at the web server of a separate SDK installation - on a separate matchine or in a separate filesystem location.

### Adding Test Files

TestRunner currently supports tests written in Selenese, Selenium's HTML-based format for recording automated tests. The Selenium IDE can be used to record tests and save them in Selenese format. For more background on the Selenium IDE, Smart GWT's extensions, and the use of Selenium WebDriver, see the `Automated Testing Overview`.

Test files should be saved with the extension .rctest.html. They should all appear under a common root directory (called `testRoot`), but any level of nesting is allowed, and any other files that appear under `testRoot` will be ignored; only .rctest.html files will be processed.

The `testRoot` directory is passed to TestRunner when you execute it. In the database and in emails, test are identified by their directory path relative to `testRoot`.

Adding a test to the test suite is as simple as placing the .rctest.html file somewhere under the `testRoot` directory; on the next TestRunner execution, TestRunner will notice the new test and start reporting results for it (including reporting it as a failure if it fails in its very first run).

The included test result viewing application (see below) also provides an interface to upload tests if you prefer not to allow direct filesystem access to the machine where TestRunner executes.

### Running Test Runner

TestRunner is an ordinary Java class - com.isomorphic.autotest.TestRunner - and can be run from the command line in the usual fashion, or run programmatically from within a Java application using the wrapper class com.isomorphic.autotest.TestRunnerDriver. We also provide convenience scripts to run the TestRunner Java class in the SDK root directory.

Minimally, TestRunner needs to know the base directory of a set of test files. All files saved anywhere under this base directory which end in the extension .rctest.html will be assumed to be Selenese test files and executed.

As is standard for Selenese test files, the first command in the file is typically an "open" command with the URL of the application which should be opened in a browser so that subsequent commands can be run.

The assumption is that the application that will be tested is already deployed by the time TestRunner is run; how to automate building and deployment of applications is outside of the scope of this document, however, the recommendation is that a Revision-Control System (such as SVN, git or CVS) is used, and that every time a developer "checks in" or "commits" changes, the application being tested is built and deployed to a test server, then TestRunner is run. Continuous Build Servers (such as Hudson, Bamboo or CruiseControl) may help automate the step of building from source control and deploying to a test server, then such a Build Server can typically be configured to trigger TestRunner.

TestRunner requires several resources to be in expected default locations from the current directory unless you provide overrides via the command line or `server.properties`. Some of the required resources are:

- the batch report email template, by default at mailTemplats/batchReport.template
- the selenium test template and batchRun/testResult dataSource XML files, by default in tools/selenium
- the dataSource XML schema files, by default in isomorphic/system/schema

#### Command-line Examples

The following command-line would run TestRunner, execute all tests under the default testRoot directory "tests", and commit the results:

      java com.isomorphic.autotest.TestRunner

This assumes your classpath environment has been set to include the isomorphic SDK JARs; you may invoke the convenience script test_runner.sh|.bat|.command in the SDK root directory to run the TestRunner Java class without having to set the classpath.

The following command-line would execute TestRunner as above, but run all tests under the directory "foo/bar" relative to the current directory, and email a report of the results:

      java com.isomorphic.autotest.TestRunner -tr foo/bar -e user@company.com

To do the same, but only run a particular test, you can use the files option (-f):

      java com.isomorphic.autotest.TestRunner -tr foo/bar -c -f test1.rctest.html -e user@company.com

Note that when a file is specified, the default is not to commit the results unless requested via the commit option (-c).

#### Java API

The following Java code would do the same as the last command-line example:

          TestRunnerDriver driver = new TestRunnerDriver();
          driver.setTestRoot("foo/bar");
          driver.setBatchCommit(true);
          driver.setFiles(new String[] { "test1.rctest.html" });
          driver.setAlertEmail("user@company.com");
          driver.run();

#### TestRunner Configuration

TestRunner supports several more command-line options, or equivalent settings that can be applied via Java. The following table summarizes the command-line options, equivalent Java Setter in the DriverConfiguration interface, and it's behavior (including default behavior).

Command-line Option | Java Setter | `server.properties` Name | Behavior
--- | --- | --- | ---
-b <browser> | setBrowser | N/A | Specifies the browser string passed to Selenium. Default is *firefox See `UsingSelenium`
-br <branch> | setBranch | autotest.branch | Specifies a branch for the batch, used in the batch run record and email notification. Default is MAIN.Test result comparison occurs per branch.
-c/-nc | setBatchCommit | N/A | This pair of argumentless options allows you to force the batch results to be committed (-c) or not committed (-nc). This is useful to override the default of committing (or of not committing if the -f option has been passed).
-cs | setCaptureScreenshot | N/A | Configures TestRunner to capture a PNG screenshot of the browser if a Selenium test fails, adding the image to the test result record.
-f <files> | setFiles | N/A | Specifies a file or list of files to run. This option can restrict which Selenium scripts under testRoot get run. Relative paths from the testRoot or bare filenames may be provided. When present, this option also disables the default behavior of committing test results.
-fr <path> | setFileRoot | N/A | Sets the root directory for all other file system paths. If not set, defaults to current working directory where Java was launched, or the web server root if TestRunner is run in a servlet.
-h | N/A | N/A | Lists available command-line options.
-hp <port> | setHttpPort | N/A | Sets the web server port Selenium should use to run the tests. Default is 8080
-ht <host/IP> | setHttpTarget | N/A | Sets the target web server Selenium should use to run the tests. Default is localhost
-lg <message> | setBatchLog | N/A | Provides a log message to include in the record for this batch run. (No Default)
-lp | N/A | N/A | Informs TestRunner that a message or file has been piped to STDIN as the batch log message.
-sm | setSaveMessages | N/A | Configures TestRunner to save the client log messages for each test to the associated test record if the test fails.
-sr <path> | setServerFileRoot | N/A | Sets the serverFileRoot directory. Default is /. Selenium scripts executing open() commands on the httpTarget server will by use this default path.
-t <timestamp> | setTimestamp | N/A | Forces comparison of the batch results to be against the batch run with a timestamp closest to that provided, rather than the most recent batch run. Format is "2012-12-31 23:59:59" in the local time zone.
-tr <path> | setTestRoot | autotest.testRoot | Sets the testRoot directory relative to the current directory. By default, its value is tests, and all Selenium scripts under the testRoot will be executed by TestRunner. If TestRunner is being run in s servlet container, then this path must use Unix-style path separators (forward slashes) and be absolute (starting with a separator). In the host filesystem, the path will be interpreted relative to the root container directory.
-un <userName> | setUserName | autotest.userName | Specifies a user name for the batch run record. (No Default)
-vm <mode> | setServerLogMode | N/A | Configures TestRunner to collect the server log messages for each test if it fails. Legal modes are "none", "some", or "all". Default is some.
-vo <out> | setServerLogOutputMethod | N/A | Configures the output method that TestRunner will use to report or persist any server log messages; only has an effect for server log modes of "some" or "all". Legal values are "email", "datasource", or "both". Default is email.
-x/-nx | setMaximizeBrowser | N/A | Sets whether to maximize the browser for Selenium tests. If not explicitly set via this call, the browser will be maximized if and only if screenshots are being taken.

### Email Notifications

At completion of the batch of tests, TestRunner can automatically send out an email notification summarizing the results of the test run, including error messages for any newly failing tests. A velocity template file is used to control its format; see `Velocity Support`. The following velocity variables are available:

- $firstBatchFound. Whether baseline batch was found with which to compare
- $fixed. A list of the test results for tests fixed in this batch run
- $regression. A list of the test results for tests broken in this batch run
- $totalTestFiles. The total number of tests run in this batch run
- $passedTestFiles. The number of tests that passed in this batch run
- $batchStartTime. Timestamp associated with start of this batch run
- $batchLog. Log message, if any was provided for this batch run

A sample/default template is provided as the file mailTemplates/batchReport.template.

The following options govern the Email Notifications:

Command-line Option | Java Setter | `server.properties` Name | Behavior
--- | --- | --- | ---
-cc <recipient> | setCcEmail | autotest.ccEmail | Sets the recipient email address for batch report email. This recipient will always be cc'd a copy of the batch report email, regardless of whether fixes or regressions are present. (No Default)
-e <recipient> | setAlertEmail | autotest.alertEmail | Sets the recipient email address for batch report email. If the "repeat email" recipient address has also been set via -re, this address will only be sent "alert email" reports where fixes or regressions are present. Otherwise, it will receive all batch report email. (No Default)
-m <mailHost> | setMailHost | autotest.mailHost | Specifies what mail host to use to send mail. If not provided, your mail software will decide what host to use.
-ms <subject> | setMailSubject | autotest.mailSubject | Sets subject line base to use when sending the email reporting batch results. Regressions and fixes info will be appended to the provided subject content.
-mt <file> | setMailTemplate | autotest.mailTemplate | Specifies what velocity template file to use when generating the batch report email for this batch run. Default is mailTemplates/batchReport.template
-ne | setNoEmail | N/A | Disables sending any email for the batch run. If recipient email addresses have not been set through the command line, Java setters, or server.properties, it's not needed. However, it may be useful in suppressing email in cases where they have been set.
-re <recipient> | setRepeatEmail | autotest.repeatEmail | Sets the recipient email address for batch report email. If the "alert email" recipient address has also been set via -e, this address will only be sent "repeat email" reports where no fixes or regressions are present. Otherwise, it will receive all batch report email. (No Default)
-se <sender> | setSenderEmail | autotest.senderEmail | Sets the sender email address for batch report email. Only needed if there is a problem sending email using the sender address generated by default.

Note: If you choose not to have any email sent upon completion of a batch run, and decide not to commit the results to the DataSources, the results of each batch run can still be determined by examining the Java console log, which captures the output of each RC test script.

### Result Viewer

TestRunner comes with a very very simple application for interactively viewing and searching test results, implemented in SmartClient technology. This application is meant as a starting point for building your own application for interactive viewing of test results, if you prefer to go beyond email notifications.

The source code for this application is just a single testResultViewer.jsp file in the "selenium" directory in the SDK; copy it anywhere under `webroot` in a project that includes the Smart GWT Server and it will function.

The result viewing application also includes the ability to upload new test files to `testRoot` as an alternative to providing testers with direct access to the filesystem for the machine where TestRunner executes.

### Getting Started FAQ

Q: When I run TestRunner, I want to target the SGWT showcase, but TestRunner fails due to HSQLDB reporting a locked database.
A: By default, TestRunner uses the HSQLDB associated with the SGWT showcase when run from the SGWT ZIP root directory. Therefore, if samples/showcase/war has been deployed to a webserver, you must stop it before running TestRunner. One simple alternative is to deploy the file showcase.war from the SGWT ZIP root, which has a separate copy of the HSQLDB. You may also simply install another copy of the SGWT ZIP in a different location on the same machine, or point TestRunner at a different machine using the -ht command-line option.

Q: When I run TestRunner, TestRunner fails reporting that DataSource BatchRun or TestResult cannot be found.
A: These DataSources must be imported into the default HSQLDB before TestRunner can be used. Use the "import" option of showcase/tools/adminConsole.jsp under the deployed SGWT showcase to select and import the BatchRun and TestResult DataSources prior to running TestRunner.