especially vis-à-vis the position of blank spaces; (ii) the specific
Every indexed document
Browse the released records by type and release. Open any document to read its text and jump to the official source.
69,179 documents
when constructing our US versus British English corpora
we took the more commonly used subject heading and discarded the rest.
92.4% of the books had at least one BISAC subject assignment. In cases where there were
such as to determine how books are shelved in stores. For
as well as statistical language determination using the Popat algorithm (Ref S3). The algorithm
we rely on metadata derived from our 100 bibliographic
books that trigger both the OCR quality filter and
the statistics presented below are influenced by the
usually under 2% of the original number of books.
000) of the books were filtered out in this way. Table S1 lists the fraction removed at
while retaining the vast majority of the books in the original corpus.
in order to maximize the number of books included (note that
we filtered out all volumes whose quality score was lower than
we excluded volumes with poor OCR quality. For the languages that
the Popat model is also capable of computing the probability that the text is
every volume has associated with it a probabilistic OCR quality score (quantized to an integer
including the entire text of a volume. In this
one in each of the 32 supported languages. From the per-glyph probability one can
to compute for each glyph (character) the probability that it is anomalous given other nearby