October 12, 2017

Demand for the screen scraping software automation stays with us from the days when only a small number of the software solutions were designed with possible integrations in mind. Legacy enterprise solutions often do not provide a quick and reliable native way of the data transfer.

Modern enterprises require a wide variety of business applications to support their operations (ERP, CRM, HRMS apps like SAP or Microsoft Dynamics). When deciding what application/product to use all the current business needs are taken into consideration. However, businesses change over time and so too do their needs. So at some point, the chosen application no longer supports all the company’s needs. When this happens, there are a few options:

  1. Extend the current application so it supports all the current needs of the business. This option is available if an application has some extensibility possibilities via API or custom modules or something else. But this is not always the case, and if not, then there are only next two other options available.
  2. Implement a helper application that will extract and manipulate data in your current application and support new usage scenarios. This is the easiest option as well as the most cost and time effective.
  3. Migrate to the entirely new system. This scenario means all the data will have to be shifted across to a new system. Unfortunately, transferring across isn’t always easily achieved if there aren’t existing APIs to support the transfer.


Luckily even if an application does not offer any good APIs for scraping screen data (mostly it is screen scraping software for capturing the text) there are still some options available.

OCR (Optical Character Recognition)

Even though we talked about this option first – in most cases, it is used only as a last (fallback) method if nothing else works. The reason for this is that OCR cannot guarantee 100% accuracy even with advanced training, which is not an acceptable solution if you are transferring sensitive information, e.g., accounting or medical information. Unfortunately, there are some applications like this out there.

System APIs Interception

This option is better than OCR because it can guarantee that you capture text from a desktop screen with 100% accuracy. The idea of this method is that the application will inject its code into all the running applications and intercept system API calls. Whatever UI framework the target application uses (WPF, WinForms, QT or let’s say MFC) and whatever code one writes to add text label to some window – under the hood, in all of these cases, a very few common system functions are called (e.g., TextOut, DrawText, some GDI+ methods, etc.). So when intercepting these methods, you can get all the text that is shown on the screen regardless of what UI framework is used, what font is used etc.

The only disadvantage to this approach is that it uses some Windows undocumented APIs and with each security update it becomes harder and harder to do it. However, it still works well on legacy systems.

Custom Mirror Driver or Accessibility Driver

The so-called Mirror Drivers (starting from Windows 8 Accessibility drivers) are virtual device drivers that mirror all the drawing operations (including text drawing operations we are interested in) that happen on your screen onto a virtual screen. This is a pretty good option that gives a high level of accuracy for the screen scraping software but is also the most complicated (and therefore time and cost consuming) solution.

Using Standard APIs

For a normal desktop application, it is possible to use a set of standard APIs to scrap the text out of them. For edit and richedit controls, there are some window messages that can be sent to a desktop app and get cursor, selection, and text information. For other applications, difference accessibility APIs can be used to allow your business application integration. Examples of such APIs are: MSAA (IAccessible), IA2 (IAccessible2), UIA (UI Automation) and so on. For each specific application, different APIs are applicable because it depends how that specific application is implemented (might be compatible with some of the mentioned APIs but incompatible with other ones).

When it comes to the screen scraping software automation from the office applications (Microsoft Office, LibreOffice, OpenOffice, etc.), these products offer their own APIs (Microsoft Office Interop UNO, etc.) which are pretty advanced and allow you to do screen scraping in the most convenient for you way. Also, they support extensions and macros so integrating with them is pretty straightforward.

Browser Extensions/Plugins for the Screen Scraping

Even though some browsers support some of the standard APIs – this support is quite limited. So if you want to scrape text or any other data from an advanced web application (let’s say Salesforce) – the best option is to do screen scraping through a browser extension. Such an extension acts as a “proxy.” On the one hand, it injects JavaScript code into your application pages, which allows for the capture and manipulation of data in the application. On the other hand, it communicates either with your desktop or web application and feeds data back to it. Since extension is only a proxy, developing the extension itself is pretty easy; the complicated part is interaction with a web application itself, especially if it uses lots of non-standard controls and complicated object hierarchy that can affect the screen scraping software performance.

Java Applications

Java is not very popular these days Still, many Java-based applications like for instance Oracle E-Business Suite (Oracle EBS) are Java-based. What if you need to capture text data from such an application? In this case, not many options for the screen scraping are available. Luckily there is such a thing as Java Access Bridge, a custom accessibility API that allows data extraction and manipulation in Java applications. It is quite limited but in combination with, for example, OCR, it can give pretty good results.

Screen Scraping Software Use Case

Now, let’s have a look at the example of the of the screen scraping automation developed by Existek for one of our clients operating in the Healthcare field. In this case, we’ve had an ordinary data transfer from the legacy desktop CRM to the web-based CRM solution. Considering that all the healthcare records are extremely sensitive we’ve had to develop screen scraping software automation sequence that ensures one hundred percent accuracy.

As it has been mentioned before, OCR recognition works pretty well but can’t ensure that the data from the text and number fields will be scraped without any flaws. So, we’ve had to refuse this option to secure project’s success.

Legacy CRM does not provide any standard API for such kind of the operations like the screen scraping is neither web-interface, so we weren’t able to use API integration method nor browser extension or plugins. The same can be said about custom mirror driver or accessibility driver because this approach would be way too time-consuming even despite its high accuracy level for the screen scraping software automation. Also, the host CRM application wasn’t Java-based so we weren’t able to use standard Java Access Bridge approach in this particular case. Obviously, the only option that is left is System APIs Interception. Luckily, it also provides 100% of the screen scraping software automation accuracy which is vital for any software project in healthcare. The only disadvantage of this method is the fact that there are plenty of undocumented APIs and this requires an involvement of the highly qualified and experienced software engineers who understand these systems deep enough to work without straight documentation and can improvise when needed. Since Existek team has 5+ years of experience with Microsoft technologies on average, this obstacle was barely noticeable.

Host CRM has been build on the pretty standard WPF framework and the engineering team has built the screen scraping application that was able to inject its code into the CRM and read all the APIs calls. As the result, we got the screen scraping software that was able to accurately recognize the CRM fields and text in these fields ignoring any possible distortions resulted, for example, by different text fonts. Speaking shortly, it didn’t require training your OCR engine separately for any language that might be used in your text data. Tests and later the client reported an absolute accuracy of the selected method for building screen scraping software automation.

There is always a way to integrate with any application. Integration options need to be chosen based on wide range of factors to achieve the best possible speed and accuracy.

What’s your experience with the approaches to text scraping mentioned in this article? What challenges did you met during the data transfer from the applications that do not provide API? Just leave your thought in the comment section below and let’s start the discussion!

If you struggle to extract any data including text from some application or you want to automate some processes there – we will be happy to find the best possible solution for you, just drop us a line and we will get back to you as soon as possible or visit our Services Page to learn more.