More than 120 journals are read (see Journals Reviewed List) to identify articles with the strongest methods for clinical attention. Journals are selected based on suggestions by librarians, clinicians, editors, and editorial staff; Science Citation Index (SCI) impact factors; systematic examination of the contents of each selected journal for at least 6 months; and by ongoing yield of articles that meet basic Inclusion Criteria for assessing the quality of studies concerning the cause, course, prediction, diagnosis, prognosis, prevention, and treatment of medical disorders. Access to foreign-language journals is provided through the systematic reviews we read, especially those in the Cochrane Database of Systematic Reviews, which summarizes articles taken from over 800 journals in many languages.

More than 400 journal titles have been assessed since 1991 and, based on the number of articles meeting criteria, the top journals form the core set reviewed. Journal yield is reviewed annually and journals with very low yield (<1-2 articles/year) are dropped. New journals are added if nominated or if needed to expand the scope of disciplines covered and if they contain articles that meet our criteria based on review of at least 6 journal issues.

Research associates trained in health research methodology read all original and review articles in the journals on the Journals Reviewed List and apply the Inclusion Criteria relevant to the purpose category (i.e., therapeutics, diagnosis, prognosis, etc). Articles read include a broad range of medical disciplines including primary care, emergency medicine, internal medicine and its subspecialties, 14 surgery disciplines, obstetrics, gynecology, pediatrics, psychiatry, nursing, rehabilitation, and pharmacy. Articles that pass criteria are tagged for entry in the Critical Appraisal Process (CAP) system.

The process of applying criteria to identify high-quality articles yields relatively few that make it through the methods filter. For example, in 2009 of about 25,000 original and review articles published in 120 journals read, we identified 3728 articles (15%) for clinical attention.

After articles are identified as passing criteria, they are entered into the Critical Appraisal Process (CAP) system. PubMed numbers are used as unique identifiers to ensure items are entered into the process only once. Using the PubMed number, the CAP system automatically pulls the article title, authors, and journal citation.

Research associates add the Article Type (i.e., Original or Review), one or more of ten Purpose Categories (i.e., primary prevention, treatment, diagnosis, prognosis, etiology, quality improvement, economics, clinical prediction guide, differential diagnosis, and qualitative), and one or more Patient Populations (i.e., neonates, pediatrics, adolescents, adults, geriatrics, all, and indiscernible). For specific purpose categories, additional data is required. For example, for therapy articles, the follow-up rate is included. Clinical disciplines/specialties are added depending on the target audience (e.g., for an article of interest to both physicians and nurses, research associates have a selection of 50 and 61 disciplines/specialties from which to choose, respectively). Most articles are of interest to two or more disciplines/specialties.

Indexers add index terms to the article, which are used by various end products.

After the data in Step 3 are added, the articles are then passed to the editor stage where one or more clinicians checks the data added by the research associates and indexers.

Articles that pass the filter process (Step 2) and the checking process (Step 4) are transferred to the McMaster Online Rating of Evidence (MORE) system. Using the PubMed number, the MORE system pulls in additional article information from PubMed such as the abstract and major and minor MeSH headings.

For a detailed description of the MORE step, see the MORE website at http://hiru.mcmaster.ca/more.

The MORE system has a database of > 4000 physician raters. Physician raters in MORE are MDs or equivalent, have completed formal training, and are in independent clinical practice at least part time in internal medicine or its subspecialties, general practice/family medicine, obstetrics/gynecology, oncology, pediatrics, or surgery. In all, there are more than 60 physician disciplines in MORE.

The MORE EBN (Evidence-Based Nursing) system has >2000 nurse raters who are in independent clinical practice at least part time (or supervise or manage others with direct patient contact) in >60 specialties.

MORE Rehab raters are currently practicing as an occupational or physical therapist in a clinical setting, or as a clinical practice leader or manager directly supervising clinical care.

MORE raters complete a profile page indicating their Patient Populations, Discipline/Specialty selections, and the number of articles they would like to rate each month.

We continue to recruit raters for all disciplines/specialties for all the MORE systems. If you would like to join us as a MORE rater, please contact us at more@mcmaster.ca (physicians), moreebn@mcmasterhkr.com (nurses), or rehab@mcmasterhkr.com (rehab specialists).

Articles in MORE are matched to rater selections and rating requests are sent to at least four raters for each discipline/specialty. For example, the article "Primary angioplasty vs. early routine post-fibrinolysis angioplasty for acute myocardial infarction with ST-segment elevation: the Gracia 2 non-inferiority, randomized, controlled trial," which is of interest to both physicians and nurses, was sent to 4 raters each for Cardiology, Internal Medicine, and Emergency Medicine for physicians, and to 4 raters each for Cardiology and Emergency/Trauma for nurses.

Ratings in MORE are on two 7-point scales (highest score, 7). The first scale, for Relevance, is for the extent to which the article was pertinent to practice in the rater’s clinical discipline/specialty. If Relevance is rated at least 4, the rater completes a second 7-point scale, on the extent to which the article’s content represented news or something that clinicians in the rater’s discipline were unlikely to know (which we labeled newsworthiness). Optionally, raters also provide up to 1000 characters of freetext comments (sample Rating Form).

When at least three ratings are received for a discipline, the totals are averaged and articles scoring ≥ 4 of 7 for both Relevance and Newsworthiness for at least one discipline/specialty are transferred to a permanent database, Premium LiteratUre Service (PLUS). Articles not scoring at least 4 of 7 for any discipline/specialty are transferred to a quarantine database.

PLUS ratings are used to help select content for ACP Journal Club*,evidence-based summary texts , BMJ Clinical Evidence, eTherapeutics+, FirstConsult), and alerting services (EvidenceUpdates, OBESITY+, STAT!Ref Evidence Alerts, NURSING+, Rehab+, KT+), and national health library services (Norwegian Electronic Health Library). Additional ratings are provided for DynaMed for articles meeting the stringent selection criteria of both PLUS and DynaMed.

*Terms used in ACP Journal Club can be found at Glossary.