The ISRI OCR Performance Toolkit

The OCR Performance Toolkit (OCRtk) is software for measuring the accuracy of optical character recognition (OCR) output combined with a large and diverse corpus of scanned page images with corresponding ground-truth text. Both were developed at ISRI throughout the middle 1990s as part of a program to compare leading OCR technologies.

News

• 28 May 2007: The Analytic Tools are now distributed under the terms of the Apache License, Version 2.0.
• 28 Mar 2005: Obviously, CAN-SPAM has not stopped email address scraping from web pages (surprise). Apologies to those who subscribed to the announcements list only to find a new source of spam. Since the list is announce-only, posting has been restricted to a person at ISRI.
• 17 Feb 2005: Thanks to Andre Steenveld for noticing that greyscale zone files were missing from the M and N collections. If you have already downloaded Ng.tgz and/or Mg-part?.tgz, you do not need to download them again. Packages containing only the missing zones files are provided below.
• 1 Mar 2005: We have created a mailing list for announcements.

Analytic Tools

The ISRI Analytic Tools are 19 programs designed to measure the accuracy of and experiment with OCR output. The following metrics are available:

• character accuracy
• accuracy by character class
• word accuracy
• non-stopword accuracy
• cost of zoning errors

along with aggregation, confidence intervals, etc. The programs are written in C and should compile on most platforms without changes. Pre-compiled binaries of most of the Analytic Tools are included in the Frontiers Toolkit below.

Download

• User Guide (295k PDF)
• Source Distribution (221k Gzipped Tar file)
• Stephen V. Rice's dissertation (Rice96): Measuring the Accuracy of Page-Reading Systems, (497k PDF)

Images and Ground Truth

The table below shows the nine datasets, along with the numbers of pages and characters, and the ISRI Annual tests that used it. Since image quality is a major factor controlling recognition accuracy, most datasets were manually scanned multiple times to produce bitonal images at 200, 300, and 400dpi, and 300dpi greyscale. The B and L sets also have Standard- and Fine-mode fax images.

Sample		Number of		Bitonal			Grey	Fax		Annual
Name	Description	# Pages	# Chars	200	300	400	300	Fine	Std	Test(s)
2	DOE Sample 2	460	817,946							1993, 94
M	Magazine Sample	200	666,134							1994, 95
N	Newspaper Sample	200	492,080							1995
B	Business Letter Sample	200	319,756							1995, 96
L	Legal Document Sample	300	372,098							1996
S	Spanish Newspaper Sample	144	348,091							1995, 96
3	DOE Sample 3	785	1,463,512							1995, 96
R	Annual Report Sample	300	892,266							1996
Z	Magazine Sample 2	300	1,244,171							1996
TOTAL		2,889	6,616,054	2,429	2,889	2,429	2,429	500	500
				# Pages

For each page, manually-keyed ground-truth is provided, along with manually-determined zone information (for each resolution).

Naming Conventions

Each page has an id of the form DDDD_PPP, where DDDD indicates the document number and PPP the page number within the document. A given page has multiple TIFF image files:

Fax images were made by a Xerox 7024 fax machine, all others were made by a Fujitsu M3096G scanner.

Each image has a corresponding zone file, e.g., DDDD_PPP.2BZ, DDDD_PPP.3BZ, ... For 3A files, use the 3BZ zone file. The format of the zone file is one zone per line, each having the five columns "left", "top", "width", "height", and "type".

For each zone on each page, there is a corresponding ground-truth text file, DDDD_PPP.Z01, DDDD_PPP.Z02, etc.

Finally, each sample has a PAGES file, containing a list of all page ids in the sample; PAGES_1, ..., PAGES_5, which define Page Quality Groups, and a ZTYPES file which lists all zone types used. For the S sample, there are also PAGES.ARG, PAGES.MEX, and PAGES.SPA, which indicate whether the source document originated in Argentina, Mexico, or Spain.

Download

Because of their size, greyscale images have been pulled out into separate packages.

Bitonal images, zone files, ground-truth packages:

Greyscale image packages:

Missing greyscale zones (if you downloaded Ng or Mg before 17 Feb 2005):

ISRI Annual Tests of OCR Accuracy

In 1992-1996, ISRI conducted 5 tests of the performance of leading OCR systems. There was some attempt to compare products to stimulate competition and to advance the technology, but the overall goal was to establish what performance range was provided by contemporary systems and to identify where improvements were most needed. All of the annual tests are provided here.

Download

• "A Report on the Accuracy of OCR Devices", 1992 (384k PDF)
• "An Evaluation of OCR Accuracy", 1993 (160k PDF)
• "The Third Annual Test of OCR Accuracy", 1994 (412kb PDF)
• "The Fourth Annual Test of OCR Accuracy", 1995 (3.6MB PDF)
• "The Fifth Annual Test of OCR Accuracy", 1996 (2.1MB PDF)

OCR Frontiers Toolkit

The Frontiers Toolkit is a companion to the book Optical Character Recognition: An Illustrated Guide to the Frontier, by Rice, Nagy, and Nartker (RiceEtal99). It includes all page images (280) used in the book, manually-determined zone information, ground-truth text, and OCR output from three leading OCR systems.

ftk-1.0

bin precompiled binary versions of the analytic tools

Irix

Linux

Solaris

Win32

data

2.1-5.7 coresponding to section numbers from the Frontiers book

*_3.tif sample pages: bitonal, 300 dpi, CCITT Group 4 compressed

*_3.sdf bounding box for the snippet used in the book

*_3.zon bounding boxes for all (manually-determed) zones

*.txt ground-truth text for all zones

*.z01, *.z02,... ground-truth text for individual zones

man

man1 Unix-style man pages for analytic tools

samples sample OCR output for 3 leading OCR systems

ocr1

ocr2

ocr3

ftk.pdf main documentation for the analytic tools

README.txt

Download

• User Guide (77kb PDF)
• Source Distribution (25MB Gzipped Tar file)

Tesseract OCR

An ideal complement to the OCRtk is the open-source Tesseract OCR engine. Tesseract OCR was developed at HP Labs until 1995 and was released to the open-source community in 2005. It ranked very highly in The Fourth Annual Test of OCR Accuracy.

Mailing Lists

We currently have an announcements-only mailing list, ocrtk-announce@isri.unlv.edu. If you wish to subscribe, send a message to ocrtk-announce-request@isri.unlv.edu (note: this is a human).

References

Contact

Information / questions / help: support@isri.unlv.edu