Semantic Automation from Screen Capture

This project proposes a novel paradigm for SW automation. It utilizes a record/replay methodology, in which the system records a sequence of actions to create an automation script and then, the script can be replayed again and again to test the application over its lifetime. Record/replay engines usually rely on having deep knowledge about the technology internals of the application, which complicates the development of generic SW automation tools. The system analyzes captured screen images from the application GUI along with the relevant user actions (e.g. mouse and keyboard) to infer the local representation of UI controls. From that representation it is possible to gain a comparable level of semantic information on the application as if the automation client was deeply integrated inside the software.

The project led to a successful transfer of the technology into HPSW product. It is now part of the HP functional testing proposal as part of QuickTest SW testing product.


The system combines technologies from computer vision and from formal languages to generate GUI representation. It is modeled as a compiler, having a visual lexer module that transforms images and user actions into abstract atoms using computer vision approaches to segment and classify GUI building blocks. These atoms are then used by a spatial parser to construct meaningful GUI construct. The parser uses Visibly Pushdown Languages(VPL) to define GUI constructs using a novel template based configuration language.

  look and feel
HP Labs - semantic autoamtion client

  Publications and Reports
  • Semantic Automation from Screen Capture Barkol, Omer; Bergman, Ruth; Pnueli, Ayelet; Schein, Sagi, HPL-2009-161. HP Labs Technical Report.
  • Visibly Pushdown Languages for a GUI Parsing Application with Probabilistic Lexer , David Lehavi, Omer Barkol and Sagi Schein ICSC 2011.
  • Efficient and robust image descriptor for GUI object classification , Anastasia Dubrovina, Pavel Kisilev, Daniel Freedman, Sagi Schein and Ruth Bergman, ICPR 2012
      [Tech Report]