PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Automatic Alphabet Recognition
Maayan Geffet, Yair Wiseman and Dror G. Feitelson
Kluwer Journal of Information Retrieval 2004.

Abstract

The last step of the Information Retrieval process is to display the found documents to the user. However, some difficulties might occur at that point. English texts are usually written in the ASCII standard. Unlike the English language, many other languages which have other characters sets, don't have one standard. This plurality of standards causes problems, especially in a web environment, where one may download a document in an unknown standard. This paper suggests a purely automatic way of finding the standard which was used by the document writer, based on the statistical letters distribution in the language. We developed a vector-space-based method that creates frequencies vectors for each letter of the language and then matches a new document's vectors to the pre-computed templates. The algorithm was applied on various types of corpora in Hebrew, Russian and English and provides an efficient solution to the stated problem in most cases.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Article
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Natural Language Processing
Information Retrieval & Textual Information Access
ID Code:378
Deposited By:Maayan Geffet
Deposited On:18 November 2004