Document

3) Quality of OCR. This varies from corpus to corpus as described above. For English, we spent a gre

Ref IMAGES-004-HOUSE_OVERSIGHT_017021.txt Release House Oversight Committee — Epstein Estate Records (Nov 2025) 1 pages

Epstein Suite indexes the text; the original document lives at its official source. We don't host the original file — view it on the official release to read it in full.

View the original on the official release

Document text

Text is machine OCR and may contain errors. Confirm against the original source above.

3) Quality of OCR. This varies from corpus to corpus as described above. For English, we spent a great deal of time examining the data by hand as an additional check on its reliability. The other corpora may not be as reliable. 4) Quality of Metadata. Again, the English language corpus was checked very carefully and systematically on multiple occasions, as described above and in the following sections. The metadata for the other corpora may not be equally reliable for all periods. In particular, the Hebrew corpus during the 19th century is composed largely of reprinted works, whose original publication dates farpredate the metadata date for the publication of the particular edition in question. This must be borne in mind for researchers intent on working with that corpus. In addition to these four general issues, we note that earlier portions of the Hebrew corpus contain a large quantity of Aramaic text written in Hebrew script. As these texts often oscillate back and forth between Hebrew and Aramaic, they are particularly hard to accurately classify. All the above issues will likely improve in the years to come. In the meanwhile, users must use extra caution in interpreting the results of culturomic analyses, especially those based on the various non- English corpora. Nevertheless, as illustrated in the main text, these corpora already contain a great treasury of useful material, and we have therefore made them available to the scientific community without delay. We have no doubt that they will enable many more fascinating discoveries. III.0.2 On the number of books published In the text, we report that our corpus contains about 4% of all books ever published. Obtaining this estimate relies on knowing how many books are in the corpus (5,195,769) and estimating the total number of books ever published. The latter quantity is extremely difficult to estimate, because the record of published books is fragmentary and incomplete, and because the definition of book is _ itself ambiguous. One way of estimating the number of books ever published is to calculate the number of editions in the comprehensive catalog of books which was described in Section | of the supplemental materials. This produces an estimate of 129 million book editions. However, this estimate must be regarded with great caution: it is conservative, and the choice of parameters for the clustering algorithm can lead to significant variation in the results. More details are provided in Ref $1. Another independent estimate we obtained in the study "How Much Information? (2003)" conducted at Berkeley (Ref S6). That study also produced a very rough estimate of the number of books ever published and concluded that it was between 74 million and 175 million. The results of both estimates are in general agreement. If the actual number is closer to the low end of the Berkeley range, then our 5 million book corpus encompasses a little more than 5% of all books ever published; if it is at the high end, then our corpus would constitute a little less than 3%. We report an approximate value (about 4%) in the text; it is clear that, in the coming years, more precise estimates of the denominator will become available. III.1. Generation of timeline plots III.1A. Single Query The timeline plots shown in the paper are created by taking the number of appearances of an n-gram ina given year in the specified corpus and dividing by the total number of words in the corpus in that year. This yields a raw frequency value. Results are smoothed using a three year window; i.e., the frequency of 13 HOUSE_OVERSIGHT_017021

Have a question about what this document contains?

Ask the documents