Recently I have been asked to speed up our Selenium regression test suite (currently at 2 hours) by running tests in parallel. We are using Jenkins to kick off the ant task that will run the test suite that is configured in Junit.
There are 2 parts to getting this to work:
- The Selenium Grid2 Hub & Node
- Junit's parallel thread process
As usual my first task is to get a prototype running locally. I started by reading the
Selenium Grid2 documentation which is pretty solid.
I downloaded the
selenium-server-standalone-2.25.0.jar and set up a couple of bat scripts to start the grid.
- startNode.bat
java -jar selenium-server-standalone-2.25.0.jar -role node -hub http://localhost:4444/grid/register -browser browserName=firefox,maxInstances=5 -browser "browserName=internet explorer,maxInstances=5" -maxSession 5
Place these in the dir with the jar, run the hub, run the node and you should be ready to rock. The console can be viewed at: http://localhost:4444/grid/console
Lets look at the code required to send the tests to the grid using the remote web driver. For completeness I've also included the code for setting up a proxy server and a regular web driver if the hub is not running.
/**
* Returns an instance of the configured remote web driver pointing to a local web hub on port 4444
*
* This is for use in stage. If the hub is not running a regular web driver is returned
*
* @return WebDriver hub
*/
private WebDriver getWebDriver() throws UnknownHostException {
try {
return new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), getDesiredCapabilities());
//if the remote selenium hub is unavailable return a regular web driver
} catch (MalformedURLException e) {
return getLocalWebDriver();
} catch (UnreachableBrowserException e) {
return getLocalWebDriver();
}
}
/**
* Returns an instance of the configured web driver.
*
* This is for local testing
*
* @return WebDriver
*/
private WebDriver getLocalWebDriver() throws UnknownHostException {
if (useFirefox()) {
return new FirefoxDriver(getDesiredCapabilities());
} else {
return new InternetExplorerDriver(getDesiredCapabilities());
}
}
/**
* Returns a DesiredCapabilities for either IE or Firefox.
*
* @return DesiredCapabilities for either IE or Firefox
* @throws UnknownHostException
*/
private DesiredCapabilities getDesiredCapabilities() throws UnknownHostException {
if (useFirefox()) { //firefox
DesiredCapabilities dc = DesiredCapabilities.firefox();
FirefoxProfile profile = new FirefoxProfile();
if (useProxy())
dc.setCapability(CapabilityType.PROXY, getSeleniumProxy());
if (getLocale() != null)
profile.setPreference("intl.accept_languages", getLocale().toString());
dc.setCapability(FirefoxDriver.PROFILE, profile);
return dc;
} else { //internet explorer
DesiredCapabilities dc = DesiredCapabilities.internetExplorer();
if (useProxy())
dc.setCapability(CapabilityType.PROXY, getSeleniumProxy());
return dc;
}
}
/**
* This will return a Selenium configured proxy pointing to a local proxy
* on port 6666
*
* @param server
* @return
* @throws UnknownHostException
*/
private Proxy getSeleniumProxy() throws UnknownHostException {
Proxy proxy = new Proxy();
proxy.setProxyType(Proxy.ProxyType.MANUAL);
String proxyStr = String.format("%s:%d", InetAddress.getLocalHost().getCanonicalHostName(), proxyPort);
proxy.setHttpProxy(proxyStr);
proxy.setSslProxy(proxyStr);
proxy.setNoProxy("someaddress");
return proxy;
}
The next step is to set up our build.xml ant task to kick of the tests in parallel.
Thanks to code cop I've come up with this partial
build.xml:
<target name="run-tests-dev-parallel4" depends="compile">
<parallel threadcount="4">
<antcall target="-junitThread">
<param name="junit.division.total" value="4" />
<param name="junit.division.num" value="1" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="4" />
<param name="junit.division.num" value="2" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="4" />
<param name="junit.division.num" value="3" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="4" />
<param name="junit.division.num" value="4" />
</antcall>
</parallel>
</target>
<target name="run-tests-dev-parallel8" depends="compile">
<parallel threadcount="8">
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="1" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="2" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="3" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="4" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="5" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="6" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="7" />
</antcall>
<antcall target="-junitThread">
<param name="junit.division.total" value="8" />
<param name="junit.division.num" value="8" />
</antcall>
</parallel>
</target>
<target name="-junitThread">
<echo message="started thread ${junit.division.num} of ${junit.division.total}" />
<junit fork="true" forkmode="perBatch" printsummary="on" haltonfailure="false" failureproperty="tests.failed" showoutput="true">
<classpath refid="dev-classpath" />
<formatter type="xml" />
<batchtest todir="${dev.test.results.dir}">
<fileset dir="${dev.bin.dir}">
<include name="**/*Test.class" />
<custom classname="fully.qualified.DividingSelector" classpath="bin">
<param name="divisor" value="${junit.division.total}" />
<param name="part" value="${junit.division.num}" />
</custom>
</fileset>
</batchtest>
</junit>
<echo message="ended thread ${junit.division.num} of ${junit.division.total}" />
<fail if="tests.failed">
tests.failed=${tests.failed}
***********************************************************
***********************************************************
**** One or more tests failed! Check the output ... ****
***********************************************************
***********************************************************
</fail>
</target>
The
DividingSelector.java is as follows:
import java.io.File;
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.types.Parameter;
import org.apache.tools.ant.types.selectors.BaseExtendSelector;
/**
* Split a fileset into the given number of parts.
*
*/
public final class DividingSelector extends BaseExtendSelector {
/**
* The num of the current test class we are processing, 1 to totalTests
*/
private int testNum;
/**
* Number of total threads to divide the tests between, 1 to totalThreads
*/
private int totalThreads;
/**
* Current thread to run a test
*/
private int curThread;
public void setParameters(Parameter[] pParameters) {
super.setParameters(pParameters);
for (int j = 0; j < pParameters.length; j++) {
Parameter p = pParameters[j];
if ("divisor".equalsIgnoreCase(p.getName())) {
totalThreads = Integer.parseInt(p.getValue());
} else if ("part".equalsIgnoreCase(p.getName())) {
curThread = Integer.parseInt(p.getValue());
} else {
throw new BuildException("unsupported parameter " + p.getName() + " with value " + p.getValue());
}
}
}
public void verifySettings() {
super.verifySettings();
if (totalThreads <= 0 || curThread <= 0) {
throw new BuildException("parameter part or divisor not set");
}
if (curThread > totalThreads) {
throw new BuildException("parameter part must be <= divisor");
}
}
public boolean isSelected(File pBaseDir, String pRelative, File pFullPath) {
//increment the test counter
testNum++;
//this is the thread that will run the current test
int threadNum = testNum % totalThreads + 1;
boolean result = threadNum == curThread;
if (result)
System.out.println("thread: " + curThread + " ,running: " + pRelative + " , testnum: " + testNum);
return result;
}
}
Once you have the targets working you can configure it in Jenkins to run the
run-tests-dev-parallel8 or
run-tests-dev-parallel4 target depending on what works better for your build server.
When you run the build you'll see in the console output junit spawning the 8 threads and sending them to the hub, the hub will then handle the distribution to the nodes running the actual tests.
run-tests-dev-parallel8:
-junitThread:
[echo] started thread 6 of 8
-junitThread:
[echo] started thread 7 of 8
-junitThread:
[echo] started thread 5 of 8
-junitThread:
[echo] started thread 1 of 8
-junitThread:
[echo] started thread 8 of 8
-junitThread:
[echo] started thread 2 of 8
-junitThread:
[echo] started thread 3 of 8
-junitThread:
[echo] started thread 4 of 8
It will then pre divide the tests between the threads and will printout which thread has been allocated to which test. This can cause problems of a long running thread which is left after everything else is finished. This can be build into the DividingSelector but in not a prefect solution.
Next we will look at running the hub and node
as a windows service on the build server.
References:
http://blog.code-cop.org/2009/09/parallel-junit.html
