NGC - Expert Resources - Expert Commentary

Expert Commentary

Perspective

User Perspectives: An Overview of the Clearinghouse Survey Results

By: The National Guideline and National Quality Measures Clearinghouse Management Team

Each year the National Guideline Clearinghouse (NGC) and the National Quality Measures Clearinghouse (NQMC) conduct a user survey on their respective Web Sites to determine how well we are fulfilling our mission of providing an accessible mechanism for obtaining objective, detailed information on clinical practice guidelines and quality measures and to furthering their dissemination, implementation, and use in order to inform health care decisions. In the past the data has been used internally and has guided decisions regarding enhancements to content and the Web Sites. This year, we thought it would be interesting to share the Survey results with our users and examine the differences between the two clearinghouses.

The 2008 NGC/NQMC User Satisfaction Survey was completed in the fall of 2008. This was the 8th such survey of NGC and the 4th of NQMC. Overall, a total of 9,945 NGC users and 722 NQMC users provided feedback to the Agency for Healthcare Research and Quality (AHRQ) and ECRI Institute teams. Surveys like these allow AHRQ and the NGC/NQMC teams to keep abreast of who is using the databases, how they are being used, how satisfied users are, and what can be done to ensure continued usefulness and satisfaction with these critical resources. Below, we describe general findings from the 2008 survey.

Who is Using These Clearinghouses?

For NGC, about two-thirds of users have direct clinical responsibilities: physicians, physician assistants (P.A.), nurses, nurse practitioners, pharmacists, dentists, or other clinical professionals. By contrast, such individuals comprise only one-fourth of NQMC users. Other prevalent occupations in the NQMC user base are Quality Manager (20%), Administrator or Manager of Hospital, Health Plan, or Medical Group (13%), and Health Services/Clinical Researcher or Measure Developer (10%). Figure 1 below provides a more detailed breakdown of both clearinghouse audiences.

Figure 1: Distributions of Job Titles for NGC and NQMC Respondents

[Figure 1 Text Description]

Many current users are new to the clearinghouses. In NGC, 56% of those who filled out the survey did so at their very first visit to the Web site. The percentage for NQMC was similarly high (49%). We also noticed that the NGC population is more international: 35% of NGC users were from outside the U.S., as compared to only 12% of the NQMC population. Both sites have similar proportions of content from non-US developers; however, NGC has a greater diversity of developers representing the non-US guidelines, and this may help to explain why NGC appears to receive more visits from non-US users. In NGC 29% of the content (714 out of 2441 guidelines) was produced by 44 different non-US developers. In NQMC, 24% of the content (415 out of 1754 measures) was produced by 4 different non-US developers.

How Satisfied are Users?

Overall, 45% of NGC users said they were "Very Satisfied" and another 43% said they were "Fairly Satisfied." Another 9% were "Neutral", and the other 2.5% were either "Fairly Dissatisfied" or "Very Dissatisfied." These responses indicate general satisfaction, but there clearly is room for improvement. Although most of the general comments from NGC users were positive, the most common complaint regarded the NGC search function. Specifically, users found the resulting list of guidelines to be cumbersome, and often irrelevant or incorrect. Several users requested a more sophisticated search engine, and/or more explicit instructions on how new users are to successfully conduct a guideline search. For NQMC, users were asked a different question ("Would you recommend the NQMC Web site to anyone else?"), and virtually all (99%) responded "Yes."

In addition, both groups of users were asked to rate the breadth or comprehensiveness of the clearinghouse on a 1-5 scale (5 being the best), and the average ratings were 4.1 and 4.4 for NGC and NQMC, respectively. Also, ease of navigation was generally favorably assessed, with average ratings of 3.7 and 3.8 for NGC and NQMC, respectively. These ratings, however, suggest there is room for improvement, particularly in ease of navigation. For the latter issue, 37% of NGC users gave ratings of either "Neutral," "Difficult," or "Very Difficult."

Which Parts of the Web Sites are Most Useful?

Each clearinghouse Web site provides many options for accessing content, but which are actually used the most? For NGC, the top three responses were Basic Search (cited by 57% of respondents), Disease/Condition Browse (47%), and Treatment/Intervention Browse (33%). Further, the relevance of results of text keyword searches in NGC was rated an average of 3.9 on a scale from 1-5 (5 being the best). The NQMC users listed the same top three access methods, but the order was different: most often cited was Disease/Condition (45%), followed by Basic Search (40%), and Treatment/Intervention Browse (29%). Like the NGC users, text keyword searches were generally quite relevant, with an average of 3.9 on the 1-5 scale.

How do NGC and NQMC Users Compare?

NQMC users were much more likely (52%) than NGC users to learn about the clearinghouse via a link from another health-related web site, and they were also twice as likely to subscribe to the What's New email (36% for NQMC as compared to 17% for NGC). Regarding the What's New email, both groups reported that the most useful part was the notification of recent updates posted to the Web site.

Overall, Web site ratings were slightly more favorable for NQMC. The difference between the two clearinghouses was 9% of the scale range for breadth/comprehensiveness, 11% for relevance of search results, and 2% for ease of navigation. (For example, if a scale ranges from 0 to 4, then 9% of the scale range is (4-0)*0.09 or 0.36 points, similar to going from 3.00 to 3.36.) One possible reason for this difference is the contrasting user bases between NGC (mostly clinicians) and NQMC (mostly managers/administrators), as noted above.

What are the Trends Over Time?

In all eight NGC surveys, users were asked "Overall, how would you rate your satisfaction with NGC?" (we used a 1-5 scale, with 5 being the best). The average response has been consistently high, with averages ranging from 4.29 to 4.42. There is some suggestion of a small decrement over time in these ratings, with the two highest ratings of 4.42 occurring in 2002 and 2004 and the two lowest ratings of 4.29 and 4.31 occurring in 2008 and 2007. These are quite small differences, however, representing only about 3% of the scale range.

Two other items included in all eight NGC surveys are 1) "How would you rate the breadth and comprehensiveness of NGC's guideline collection?" and 2) the level of agreement with the statement "NGC's guideline summaries present all the important attributes of a guideline, when this information is available." As with overall satisfaction, responses to these items have stayed consistently positive. Figure 2 below provides a historical view of this data.

Figure 2: Trends Over Time in NGC Ratings

[Figure 2 Text Description]

We also performed trend analyses for NQMC and found no changes in users' ratings over the past 4 years (see Figure 3 below). This was true for recommending NQMC to others, ratings of breadth/comprehensives of the quality measures collection, and completeness of the available information on a given quality measure.

Figure 3: Trends Over Time in NQMC Ratings

[Figure 3 Text Description]

Conclusions

Generally, the users of the NGC and NQMC clearinghouses are satisfied with the guidelines and measures included, as well as the content of that material. One area identified for future improvement is ease of navigation. NGC user comments suggest that improvements need to be made to increase relevance of search results. NGC users tend to be more clinically oriented and are more dispersed internationally than NQMC users. Also, both Web sites are attracting a large number of new users, as evidenced by the continued finding that about half of the survey respondents had never previously visited the Web site.