3) Quality of OCR. This varies from corpus to corpus as described above. For English, we spent a gre
Epstein Suite indexes the text; the original document lives at its official source. We don't host the original file — view it on the official release to read it in full.
View the original on the official releaseDocument text
Text is machine OCR and may contain errors. Confirm against the original source above.
3) Quality of OCR. This varies from corpus to corpus as described above. For English, we spent a great
deal of time examining the data by hand as an additional check on its reliability. The other corpora may
not be as reliable.
4) Quality of Metadata. Again, the English language corpus was checked very carefully and
systematically on multiple occasions, as described above and in the following sections. The metadata for
the other corpora may not be equally reliable for all periods. In particular, the Hebrew corpus during the
19th century is composed largely of reprinted works, whose original publication dates farpredate the
metadata date for the publication of the particular edition in question. This must be borne in mind for
researchers intent on working with that corpus.
In addition to these four general issues, we note that earlier portions of the Hebrew corpus contain a large
quantity of Aramaic text written in Hebrew script. As these texts often oscillate back and forth between
Hebrew and Aramaic, they are particularly hard to accurately classify.
All the above issues will likely improve in the years to come. In the meanwhile, users must use extra
caution in interpreting the results of culturomic analyses, especially those based on the various non-
English corpora. Nevertheless, as illustrated in the main text, these corpora already contain a great
treasury of useful material, and we have therefore made them available to the scientific community
without delay. We have no doubt that they will enable many more fascinating discoveries.
III.0.2 On the number of books published
In the text, we report that our corpus contains about 4% of all books ever published. Obtaining this
estimate relies on knowing how many books are in the corpus (5,195,769) and estimating the total
number of books ever published. The latter quantity is extremely difficult to estimate, because the record
of published books is fragmentary and incomplete, and because the definition of book is _ itself
ambiguous.
One way of estimating the number of books ever published is to calculate the number of editions in the
comprehensive catalog of books which was described in Section | of the supplemental materials. This
produces an estimate of 129 million book editions. However, this estimate must be regarded with great
caution: it is conservative, and the choice of parameters for the clustering algorithm can lead to significant
variation in the results. More details are provided in Ref $1.
Another independent estimate we obtained in the study "How Much Information? (2003)" conducted at
Berkeley (Ref S6). That study also produced a very rough estimate of the number of books ever
published and concluded that it was between 74 million and 175 million.
The results of both estimates are in general agreement. If the actual number is closer to the low end of
the Berkeley range, then our 5 million book corpus encompasses a little more than 5% of all books ever
published; if it is at the high end, then our corpus would constitute a little less than 3%. We report an
approximate value (about 4%) in the text; it is clear that, in the coming years, more precise estimates of
the denominator will become available.
III.1. Generation of timeline plots
III.1A. Single Query
The timeline plots shown in the paper are created by taking the number of appearances of an n-gram ina
given year in the specified corpus and dividing by the total number of words in the corpus in that year.
This yields a raw frequency value. Results are smoothed using a three year window; i.e., the frequency of
13
HOUSE_OVERSIGHT_017021
Have a question about what this document contains?
Ask the documents