In the process of using Q7 to create test projects, we came to the conclusion that some contexts should be executed before each test cases. For example, when testing JDT before each test should be cleared Workspace and Working Set, and terminated all active launches. Initially for these purpose Group Context is used, this context should be added to each test case in Q7 project. But this practice was uncomfortable, because at the addition of a new test case, it was necessary not to forget to add the Default Group Context. Since version 1.3 of Q7 introduced the concept of the Default Contexts list.

At the creation new Q7 Project the “Project Settings” file is created automatically:

Description of project can be given in “Description” section of the “Project Settings”. Default Contexts can be set in “Default Contexts” section. Contexts to default contexts list can be added from this Q7 project and also from related Q7 projects.

These default contexts are displayed in test cases. Sequence contexts defined in the Default Contexts  list can not be changed in the test case and custom contexts can be added to test case only after default contexts. “Project Settings” can be opened from test case by clicking on “Configure Project Default Contexts” link.

At the replaying test case the default contexts are displayed in “Execution View”. Custom contexts are executed after default contexts execution.

More information about the creation and application of Q7 contexts you can get from the Q7 demos - http://q7.xored.com/demos.

In this post I am going to illustrate Q7′s built-in support for rulers and describe how to create a couple test cases for various rulers. As an example we are going to use Java editor and a simple “Hello, world!” Java class, like on a screenshot above. A sample Q7 Project with tests is available at Github – https://github.com/xored/q7.examples.rulers

Let’s start with a test case, which checks that if we double-click a ruler, breakpoint will be added and a tooltip will display a method name. To do this:

1. Start recording
2. Double-click a ruler on line 9
3. Move a mouse over breakpoint so that tooltip appears.
4. Switch to Q7 assertion mode by pressing Alt+Shift+7 hotkey, so that tooltip won’t disappear.
5. Click on a tooltip, check ‘text’ field in Assertion Window and click “Add”

That’s it, stop recording and save a test case, now Control Panel should display a text like this:

with [get-editor "Program.java" | get-left-ruler] {
    get-ruler-column AnnotationColumn | double-click-ruler -line 9
    hover-ruler -line 9
}
get-window -from "Information Window" | get-editbox | get-property text 
    | equals "Line breakpoint:Program [line: 9] - main(String[])" 
    | verify-true

In a similar way a test case for asserting folded content can be added – note that in the beginning of a test case we also expanding all folded blocks via context menu in rulers:

with [get-editor "Program.java" | get-left-ruler] {
    get-menu "Folding/Expand All" | click
    get-ruler-column ProjectionRulerColumn | click-ruler -line 5
    get-ruler-column ProjectionRulerColumn | hover-ruler -line 6
}
get-window -from "SourceViewerInformationControl.()" | get-editbox | 
    get-property text | equals "/**\n"
                             + "\t * @param args\n"
                             + "\t */" | verify-true

A sample Q7 Project with these test cases and a couple more is available at Github – https://github.com/xored/q7.examples.rulers. To run these tests locally without Q7 installation and zero-configuration, simply run these commands in your console (Windows, Linux, Mac OS X):

  $ git clone git://github.com/xored/q7.examples.rulers.git
  $ cd q7.examples.rulers
  $ mvn clean install

There are two bugs now which which might prevent recorded tests from executing, so manual modification might be required in Q7 up to 1.3.2:

1. QS-2624 – Tooltip diappears too quickly after hover-ruler command.
   Q7 implementation of hover-ruler is pretty straightforward – based on given line and ruler column it calculates mouse coordinates and sends two mouse events – MouseMove and MouseHover. But for some reason tooltips for some sort of markers (for instance, TODO markers and Error markers) disappear almost immediately, thus a next command asserting a tooltip cannot find floating window. The workaround is simple and works on all platforms, but totally unexpected and so far we have no idea why does it work – but adding a hover-text command on an editor fixes an issue and tooltips stop disappearing:

    with [get-editor "Program.java"] {
        get-text-viewer | hover-text 1 1 // Workaround for QS-2624
        get-left-ruler | get-ruler-column ProjectionRulerColumn | hover-ruler -line 6
    }
       

2. QS-2625 – Q7 does not capture ruler column when recording hover-ruler. This issue is trivial and will be fixed in 1.3.2 and has a simple workaround too. Moving a mouse over ruler’s marker records command like get-left-ruler | hover-ruler 4. The fix is to insert missing get-ruler-column between get-ruler and hover-ruler. The simplest way to discover ruler column name is to click on it, but just in case here are the names of ruler columns available:
   • AnnotationColumn – shows breakpoints, TODO items, error markers
   • ProjectionRulerColumn – shows expand/collapse items
   • LineNumberColumn – shows line numbers and/or version control system’s blame annotations

One of the greatest advantages of Q7, when it comes to continuous integration, is its ability to ‘install’ plugins from given p2 repository into some RCP before running UI tests. Besides making a build cross-platform and reducing build time, this feature allows to test product on several versions of Eclipse.

To illustrate this, we created a sample project on GitHub – http://github.com/xored/q7.examples.multirun. It contains a simple plugin (based on built-in Eclipse plugin template with a sample view) and few Q7 tests, but the real magic happens when you run these two commands in your shell on Windows, Mac OS X or Linux:

$ git clone git://github.com/xored/q7.examples.multirun
$ mvn clean install

With a zero configuration, this simple command will build an update site with a plugin and run the same project with Q7 tests on four versions of Eclipse – 3.5, 3.6, 3.7, 4.2. When this project is added as a build job on Hudson, it displays nice results with a total of 12 tests:

At first, let’s record an initial script which launches new class wizard, sets package and class name and asserts an error message:

get-view "Package Explorer" | get-tree | select example | get-menu "New/Class" | click
with [get-window "New Java Class"] {
    get-editbox -after [get-label "Package:"] | set-text "org.example"
    get-editbox -after [get-label "Name:"] | set-text Class1
    get-editbox -after [get-label "Java Class"] | get-property text |
       equals "Type already exists." | verify-true
}

get-window "New Java Class" | get-button Cancel | click

Now, what we really want is to type various values into Package and Name fields and make sure that error message changes accordingly. As a next step, let’s parameterize a script using get-param ande set-param commands:

get-view "Package Explorer" | get-tree | select example | get-menu "New/Class" | click

set-param package "org.example"
set-param class "Class1"
set-param message "Type already exists."

with [get-window "New Java Class"] {
    get-editbox -after [get-label "Package:"] | set-text [get-param package]
    get-editbox -after [get-label "Name:"] | set-text [get-param class]
    get-editbox -after [get-label "Java Class"] | get-property text |
       equals [get-param message] | verify-true
}
get-window "New Java Class" | get-button Cancel | click

So there are two logical blocks now: the first one sets parameters, the second one performs parameterized actions. So ideally we want to have a several “parameter” blocks and execute an “action” block after each of them. In ECL we can do that using two commands – emit and repeat-with. The first one just takes an arbitrary number of arguments of any type and writes them into an output pipe, so we can pass several “parameter” scripts to it, and it will write them into output.

The repeat-with is more tricky and does the following:

1. accepts a script from an argument and reads scripts from input pipe
2. for each script from input pipe, executes an input script followed by the argument script
3. writes aggregated output

Therefore, we can emit multiple parameter blocks and repeat them with a given action block:

emit {
   set-param package "org.example"
   set-param class "Class1"
   set-param message "Type already exists."
} {
   set-param package ""
   set-param class "Class1"
   set-param message "The use of the default package is discouraged."
} | repeat-with {
   with [get-window "New Java Class"] {
      get-editbox -after [get-label "Package:"] | set-text [get-param package]
      get-editbox -after [get-label "Name:"] | set-text [get-param class]
      get-editbox -after [get-label "Java Class"] | get-property text |
       equals [get-param message] | verify-true
   }
}
get-window "New Java Class" | get-button Cancel | click

The example above uses only two paramerer blocks, but there can be as many such blocks as necessary, take a look at this video:

During test case recording Q7 assertion mode allows to select a UI element in application under test and generate assertion lines in a script for selected properties. For instance, to check that “Save” button on Eclipse toolbar is disabled, we can switch to recording mode, select a button, check “enablement” property and click “Add”:

As a result, the following line will be appended to the script:

get-button "Save (M1+S)" | get-property enablement | equals false |
   verify-true

And in case this button is enabled during replay, test case will fail with a message: Assertion of enablement failed: expected:<false> but was:<true>

Yesterday one of our customers asked how to assert properties of context menus, as there’s a problem – once user switches to assertion mode, he cannot invoke context menu, as all user operations are blocked. And if context menu is open and then user goes to Control panel, context menu looses focus and closes. So, what can we do here to assert them?

The solution is to manually write ECL commands to perform these assertions. As an example, suppose we need to assert the following statements about context menu of a Java project in Package explorer:

1. The menu item New has children
2. The menu item Close Project is enabled
3. The menu item Compare With / Each Other is disabled
4. The menu item Open Project does not exist

At first, let’s take a look at what is being recorded when we click on some item in context menu:

get-view "Package Explorer" | get-tree |
 select sample | get-menu "New/Class" | click

So the sctipt takes a package explorer, finds a tree in it, selects an item “sample”, gets context menu item “New -> Class” and clicks on it. Comparing this script with an assertion example above, We can replace a click with a series of commands for assertion like this:

get-view "Package Explorer" | get-tree | select sample | 
 get-menu "New/Class" | get-property enablement | equals true |
 verify-true

Here is a list of properties on menu items which can be used for assertion:

• enablement – whether an item is enabled
• accelerator – assigned hotkey
• cascade – whether an item has children
• caption – menu item text

Given that, it is easy to implement three first assertions, but how to make sure that an item does not exist? This can be done by using verify-error command, which accepts a script as a parameter and passes if script fails, and fails otherwise. Therefore, we can write the following script:

with [get-view "Package Explorer" | get-tree | select sample] {
    get-menu "New" | get-property cascade | equals true | verify-true
    get-menu "Close Project" | get-property enablement | equals true |
       verify-true
    get-menu "Compare With/Each Other" | get-property enablement | 
       equals false | verify-true
    verify-error {
        get-menu "Open Project"
    }
}

get-preferences-menu | click
with [get-window Preferences] {
    get-tree | select "Java/Compiler"
    get-group "JDK Compliance" | get-combo -after [get-label 
      "Compiler compliance level:"] | set-text "1.6"
}

Here we record a set-text command to set a value in combobox, and this command works fine independently whether the combobox is readonly or not.

However, recently we’ve got an interesting use case from a customer. The application under test has a combobox with values varying on each launch, so line like this is being recorded and cannot be replayed on consequent executions:

get-combo -after [get-label "Value:"] | set-text "Value #123232"

Of course, we filed (and by the time of writing this post already fixed) an issue, but it is an interesting to find a workaround for this issue using current release of Q7. Suppose we want to always select the first value from a combobox. As a reference example we will use a Compiler compliance level combobox in JDT preferences.

Let’s start recording, open a preference dialog and switch to assertion mode to get an information about combobox:

Advanced properties of assertion dialog shows all properties of selected widget via reflection, so we can find all items available in given combo. If we check the first item in combobox and click Add, the following line is added to recorded script:

get-window Preferences | get-group "JDK Compliance" | get-combo -after 
   [get-label "Compiler compliance level:"]  | 
   get-property "getItems().String[0]" | 
   equals "1.3" | verify-true

In fact, we don’t need this assertion, but rather we need to get this value and use it as an argument to set-text command. Without extra argument get-property returns a special assertion handle object, which is later being used to build a descriptive error message in case of assertion failure, and in order to extract actual property value, we need to add -raw argument to it.

So, going back to the snippet in the beginning of the post and using with command to shorten the script, we can get the following:

get-preferences-menu | click

with [get-window Preferences] {
    get-tree | select "Java/Compiler"

    with [get-group "JDK Compliance" | get-combo -after 
        [get-label "Compiler compliance level:"]] {
        set-text [get-property "getItems().String[0]" -raw]
    }
}

As a result, this script will always select the first value in this combo:

“We believe new Q7 pricing model will be very fair-minded”, said Andrey Platov, Xored’s president. “Testing process generate value only when tests are executed. So we will charge our users for test executions only. Everything else such as tools and infrastructure will be free.”

Q7 Cloud

Q7 Cloud is the next big move, helping teams to be agile. Instead of building expensive infrastructure to support Continuous Integration and Delivery, teams can forward UI testing phase to the Cloud and back with zero-configuration efforts, gaining unprecedented ROI — just paying a fraction of a cent per test executed without any investment in software, hardware, and administration.

“Tools like Q7 are changing GUI testing landscape. Small teams or individual developers are able to maintain huge GUI testbases, but another problem arises: GUI tests are slow, and execution time grows to tens of hours on a single box”, said Ivan Inozemtsev, Xored’s VP of Research & Development. “Now you can run your tests in the Q7 Cloud, distributing tests execution among dozens of machines.”

Tests are developed to run them, people are investing in tests development, continuous integration, and tooling for only one reason: the ability to run tests in exchange for information about quality and localisation of a problem as a return of investment in testing. New Q7′s pricing model clearly reflects this fact, as well as cuts your investments in software, hardware, and test lab administration down to zero.

Pricing & Availability

New pricing plans available right now for eligible persons and companies. Request invite at http://www.eclipse-testing.com

• FREE: up to 5,000 test cases executed/month for FREE
• Starter $99 / month: Includes 10,000 test executions; $0.0099 per test execution
• Standard $399 / month: Includes 50,000 test executions; $0.0079 per test execution
• Professional $699 / month: Includes 100,000 test executions; $0.0069 per test execution
• Enterprise $1,999 / month: Includes 500,000 test executions; $0.0039 per test execution

Unit Testing and Code Coverage

Code Coverage is definitively important metric when you are doing Unit Testing. Since the ultimate goal of Unit Testing is to ensure all your code (units, classes) is working properly (as designed) — Code Coverage level reflects your level of assurance – a percentage of your code, which is behave as expected.

This is natural mapping between unit tests and corresponding pieces of code which is covered by tests. So having 10% of code covered with unit tests you may worry about your codebase quality, but having 90% of code covered with unit tests you may be much more relaxed knowing that major part of your codebase is working as expected.

Functional Testing and Functionality Coverage

Nevertheless functional testing looks quite similar to unit testing, they sits on another level. The goal of Functional Testing is to ensure all the application functionality delivered to the user, and those functions works as expected. Funny thing that any code coverage measurements would not help you to understand amount of planned functionality has been implemented and which amount is working properly.

What you definitely need to know when working on functional tests is Functionality Coverage in some form (like User Stories or Software Requirements). There is natural mapping between functional tests and User Stories, which are covered by tests. So when it comes to functional testing the definitive goal of QA/Dev team should be 100% Functionality Coverage, which has no connection to Code Coverage.

Call for Comments

All of the above is just only our current opinion, and we’d be happy to hear any contras/ideas. Please share your thoughts.

Kind Regards,
Q7 Development Team