ScreenTrack: Using a Visual History of a Computer Screen to Retrieve Documents and Web Pages

Computers are used for various purposes and frequent context switch is inevitable. In this setting, retrieving the documents, files, and web pages that have been used for a task can be a challenge. While modern applications provide a history of recent documents for users to resume work, this is not sufficient to retrieve all the digital resources relevant to a given primary document. The histories currently available - file names, web page titles, or URLs - does not take into account the complex dependencies that exist among resources across applications. To address this problem, we tested the idea of using a visual history of a computer screen to retrieve digital resources within a few days through the development of ScreenTrack. ScreenTrack is software that captures screenshots of a computer at regular intervals. It then generates a time-lapse video from the captured screenshots and lets users retrieve a recently opened document or web page from a screenshot that they recognize from its visuals. Through a controlled user study, it was found that participants were able to retrieve requested information more quickly with ScreenTrack than under the control condition. A follow-up study showed that the participants used ScreenTrack to retrieve previously used resources, in order to resume interrupted tasks.