Experiencing SilkTest for Eclipse RCP Testing

Comparisons, Q7, 18.08.2011 by Andrey Platov View Comments

Recently we did a quick evaluation of Borland SilkTest to understand how well it works for testing of SWT and Eclipse RCP applications. This is not comprehensive evaluation, so we’d like to mention that Silk left good overall impression: it looks like a solid, well documented, well packaged product. This is product anyone definitely should consider when looking for general purpose UI testing tool dealing well with broad set of application technologies including AJAX, Java, and .NET. However we’re interested in testing of Eclipse applications here, so our evaluation will be focused on Eclipse applications only. For this evaluation we were using enterprise-class Business Intelligence tool, which is an RCP application. It should be easy for any tool since the application we did choose has very straightforward implementation using standard SWT controls very heavily, at the same time the application does not use any diagramming (GEF, GMF), multithreading, etc, etc. So any tool without specific eclipse and Java knowledge should do this job pretty easy. Silk installation and Application Under Test configuration went smoothly, so I’d like to skip this aspect, let’s describe our experience in creation of our first test case. Test case we want to create was about to open a file with application specific multi-page editor, and do some simple operations within editor like changing editor page, click on the “open file” button within that page, and select file from the filesystem. Silk has Capture and Playback functionality so we did use it to record our simple test case very fast (though we’re newbies in Silk). Silk records (captures) end-user actions and generates Java code, which can be use later as JUnit test. Example of the code below:

@Test
public void addSupportFile() {
  desktop.find("/Shell[@caption='Plug-in Development - Eclipse SDK']")
    .setActive();
  desktop.find("/Shell[@caption='Plug-in Development - Eclipse SDK']//SWTTree[2]")
    .expand("/ssp");
  desktop.find("/Shell[@caption='Plug-in Development - Eclipse SDK']//SWTTree[2]")
    .expand("/ssp/ebl");
  desktop.find("/Shell[@caption='Plug-in Development - Eclipse SDK']//SWTTree[2]")
    .select("/ssp/ebl/start.ebl");
  desktop.find("/Shell[@caption='Plug-in Development - Eclipse SDK']//SWTTree[2]")
    .doubleClick(1, new Point(93, 278), ModifierKey.None);
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']")
    .setActive();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//CTabFolder[1]")
    .selectList();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//MenuItem[@caption='Support files' and @className='MenuItem']")
    .select();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//TextField[2]")
    .setPosition(new TextPosition(0, 0));
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//TextField[2]")
    .setText("sample");
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//TextField[3]")
    .setText("sample description");
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//ComboBox")
    .select("TEXT");
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//PushButton[@caption='Browse']")
    .select();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//Dialog[@caption='Support file']//PushToolItem[2]")
    .select();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//Dialog[@caption='Support file']//ListView")
    .doubleSelect("workspace", ModifierKey.None);
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//Dialog[@caption='Support file']//PushToolItem[2]")
    .select();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//Dialog[@caption='Support file']//ListView")
    .doubleSelect("eclipse", ModifierKey.None);
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//Dialog[@caption='Support file']//ListView")
    .doubleSelect("notice", ModifierKey.None);
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//Dialog[@caption='Support file']")
    .releaseMouse(1, new Point(141, 281), ModifierKey.None);
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//PushButton[@caption='Add']")
    .select();
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//SWTTable")
    .selectRow("sample", "Name");
}
So far so good, and we tried to replay (execute) generated test case, which should replay recorded user actions. However we did meet the first problem immediately: test did not replayed, failing with error at line 13. The problem is typical for most of UI testing tools, which we described here: Wait Operations. In our particular test case the application spends some time while opening editor, but test case code assumes it would appear immediately. Quick and dirty resolution would be to wait in test until application opening an editor. So after browsing Silk documentation for a while we did add following code, which resolves problem:
  desktop.waitForObject("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']", 30000);
Trying again and the next problem arises: Open File Dialog remember it’s previous state, so our test case was recoded with one initial folder in the Open File Dialog, but it was different when we were trying to execute test case, and test failed. After understanding what happens and how Silk identifies UI Elements we did replaced lines 29-37 with line below to set absolute path in the dialog and workaround problem:
  desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//TextField[4]")
    .setText("C:\work\e2open\eclipse\notice.html");
Next time everything went nice and test case executed properly replaying our use case, but it’s still fragile. We did resized Application Under Test full screen and test failed. The problem is out application have dozen pages in multi-page editor. When all these pages can’t fit on the screen, Eclipse editor let user to choose some from drop-down menu. If there are more space — editor move some pages from menu onto screen. Technically this pages with page captions are elements of different classes: they are MenuItems when they are in the menu, and they are CTabItems when they are within editor. Our test assume that editor page we’re looking for should be accessible from drop-down menu, which is not always true and depends on screen size, workbench layout, and other pages within editor as well as length of their captions (which also may vary on different operating systems and because of internationalization). So to avoid this issue we added following code, replacing line 17:
  String supportFilesMenuItem = "/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//MenuItem[@caption='Support files' and @className='MenuItem']";
  if(desktop.exists(supportFilesMenuItem)) {
    desktop.find(supportFilesMenuItem).select();
  } else {
    desktop.find("/Shell[@caption='Plug-in Development - ssp/ebl/start.ebl - Eclipse SDK']//CTabItem[@caption='Support files']").select();
  }
OK, we’re done for now and test case works. However it’s still fragile for many reasons, one of them is obvious from the Java code: Silk is considering our application as a flat-set of Operating System controls. There are some hierarchy defined by child shells and dialogs, but at least workbench is flat, see following for example:
  desktop.find("/Shell[@caption='Plug-in Development - Eclipse SDK']//SWTTree[2]")
All such lines are very dangerous: any change of initial workbench state such as Perspective change will break the test since it assumes that AUT should be always in Plug-in Development perspective. Any new view within workbench or some redesign of existing views will break the test, once Tree widget within Explorer view would stop to be a Tree widget number two. BTW I do not know which tree is a tree number one.

Some kudos to Q7

Of course we did try to record exactly the same test case using Q7. It has been done within a blink of an eye without any waiting problems because of Q7′s Runtime Intelligence, as well as without other issues because Q7 is a tool handcrafted for Eclipse thus Q7 knows a lot about Eclipse RCP internals and workbench elements, and generate ECL code which is much less fragile and do not require any post-production. As fans of Behavior Driven Development, we were trying to make test case code understandable by all project participants: QA engineers, developers, managers. If you have a time, please spend a few minutes understanding what both test cases supposed to do, reverse engineering original scenario using SilkTest/Java code above and Q7/ECL code below:
  with [get-editor "E2open E2NA Modeling Tool 7.4"] {
    get-tab-folder | get-tab-item "Support files" | click
     with [get-section "Support file details"] {
       get-editbox -after [get-label Name] | set-text sample
       get-editbox -after [get-label Description] | set-text "sample description"
       get-combo -after [get-label "File type"] | set-text TEXT
     }
  }
  return-from-os-dialog "FILE_SELECTOR" "platform:/resource/extra-files/support.txt"
  with [get-editor "E2open E2NA Modeling Tool 7.4"] {
    get-section "Support file details" | get-button Browse | click
    get-section "Support files" | get-button Add | click
  }

  get-editor "E2open E2NA Modeling Tool 7.4" | get-section "Support files" | get-table | get-item sample | click
Thank you very much! P.S. As Q7 developers we’re always hunting for challenges and use cases/scenarios, which Q7 can not record and replay. If you have ones, please share with us at info@xored.com

blog comments powered by Disqus

Functional and UI Testing for Eclipse-based Applications

Test automation solution, handcrafted for Eclipse. Modern and easy to learn IDE.

Product Health Analytics and Support Automation Platform

End-user and product analytics solution that gives you rich insights into your application health & performance.

Eclipse-based IDE for the Fantom Programming Language

Full-featured Integrated Development Environment for emerging Fantom programming language.