Deduplicator

About Deduplicator

The deduplicator is based on the algorithm described in the reference below. The input is a set of files, each one a list of records taken from a different bibliographic database (PubMed, EMBASE, Psychinfo, or others). They are cross-examined so a record that is present in more than one database is removed (the PubMed record is preferentially retained and there is a hierarchy of preference after that). The reason this task is non-trivial is that the same article may be transcribed and indexed in different ways in the different databases. The output is a single list of records in a defined format.