Arrowsmith Home

Word and Text Similarity Metrics

In this webpage, we present several different ways of representing words, multi-word phrases, abbreviations, and textual passages as low-dimensional vectors, and employ the vectors to create several different ways of computing term-term and text-text similarity metrics. Some of these metrics are new (devised by us) and some are derived from previous research by others. PubMed titles and abstracts have been employed as a corpus, and we hope that the resulting datasets and query interfaces should be valuable both for biomedical investigators and for text mining groups. Click HERE for a brief summary of the vector representations and similarity metrics. See the accompanying reference (below) for full details.

Term Vector Representations and Similarity Metrics

Basic Model

basic_model.csv.gz (~1.4MB uncompressed, ~557K compressed)
basic_implicit_vectors.csv.gz (~2.6GB uncompressed, ~1.1GB compresssed)
basic_word_similarity_metrics.csv.gz (~3.8GB uncompressed, ~1.5GB compressed)

Full Model (includes bigrams, trigrams and abbreviations)

full_model.csv.gz (~3.9MB uncompressed, 1.6MB compressed)
full_implicit_vectors.csv.gz (~6.5GB uncompressed, ~2.5GB compresssed)
full_word_similarity_metrics.csv.gz (~14.6GB uncompressed, ~5.2GB compressed)

Text Vector Representations and Similarity Metrics

Implicit vectors based on the full model have been computed for the entire PubMed corpus of articles (1966-2016) published in English (or with titles and abstracts translated into English) and whose title+abstract contained a minimum of 25 words. The data is available for download at:

imp_vector_download.cgi

Pvtopic vectors have been computed for the entire PubMed corpus of articles (1966-2016) that contained abstracts and whose title+abstract contained a minimum of 25 words, and the data can be downloaded using the links below.

pvtopic_pmid_index.csv.gz (~291MB uncompressed, ~88MB compresssed)
pvtopic_vectors.csv.gz (~52.1GB uncompressed, ~19.8GB compressed)

Updated Pvtopic vectors have been computed for the entire PubMed corpus of articles (1966-2020) whose title+abstract contained a minimum of 25 words. New vectors have been computed with the gensim python library to allow vectors for new articles to be inferred without remodeling the entire corpus. The data can be downloaded using the links below.

pvtopic2020_vectors.tsv.gz (~69GB uncompressed, ~28GB compresssed)
pvtopic2020_wordvectors.tsv.gz (~1.8GB uncompressed, ~.7GB compresssed)
PVTopicReadme.txt (1K uncompressed)

Reference

Smalheiser NR, Bonifield G. (2018) Unsupervised Low-Dimensional Vector Representations for Words, Phrases and Text that are Transparent, Scalable, and produce Similarity Metrics that are Complementary to Neural Embeddings. Preprint deposited in arXiv on January 9, 2018. http://arxiv.org/abs/1801.01884

Supplemental files.

These data are being released under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike CC BY-NC-SA International Public License 4.0.

Last modified: January 15, 2018