The Cornerstone of Public Health Practice: Public Health Surveillance, 1961--2011

Lisa M. Lee, PhD

Stephen B. Thacker, MD

Office of Surveillance, Epidemiology and Laboratory Services, CDC

Corresponding author: Lisa M. Lee, PhD, Office of Surveillance, Epidemiology, and Laboratory Services, 1600 Clifton Road, N.E., MS E-94, Atlanta, GA 30333; Telephone: 404-498-6010; Fax: 404-498-6365; E-mail: [email protected].

Introduction

The roots of modern public health surveillance took hold in 17th century Europe (1), but the seed for CDC's role as America's national agency for collecting, analyzing, interpreting, and using data to protect the public's health was firmly planted only in 1961, when the Morbidity and Mortality Weekly Report (MMWR) was transferred to what was then the Communicable Disease Center (CDC; now the Centers for Disease Control and Prevention) (2). The advent of MMWR at CDC marked the beginning of CDC's responsibility for aggregating and publishing data weekly on nationally notifiable diseases and publishing the data annually in MMWR's Summary of Notifiable Diseases, United States.

The Beginnings of Modern Public Health Surveillance in the United States

In its earliest incarnation in the United States, surveillance took the form of morbidity reporting. By 1925, the year all states began reporting regularly, the expectations were limited to collecting, compiling, and publishing statistics in weekly reports. By the 1950s, however, simply compiling and reporting statistics clearly was insufficient to alleviate disease threats, and the National Surveillance Program was started. That program and the Malaria Surveillance Program, which had started 2 years earlier, were based on the notion that effective disease control cannot occur without implementing new ideas and expanding use of data collected (3).

Nowhere was the idea of connecting public health surveillance data directly to public health action more successful than during the 13-year global effort to eradicate smallpox. During 1966--1978, the initial tools for eradication were public education and mass vaccination. When the disease returned in some areas thought to have reached elimination, timely, complete surveillance and ring vaccination (i.e., administering vaccine to persons in close contact with an infected patient) enabled the program to turn the corner on eradication (4).

Effective national disease surveillance was an idea that captured the imagination of Alexander D. Langmuir, CDC's chief epidemiologist for 23 years. In 1963, in his sentinel paper published in the New England Journal of Medicine (5), Langmuir separated the discipline of surveillance from the other activities of public health and emphasized the importance of systematic collection of pertinent data, consolidation and analysis of these data into useful information, and dissemination of results to persons who need to know and can take action. These concepts were argued convincingly to the World Health Assembly as the approach for monitoring communicable and noncommunicable health events; subsequently, surveillance systems were developed, and findings from these systems were highlighted in a special issue (volume 5, number 1) of the International Journal of Epidemiology in 1976.

During the 50 years since Langmuir published his concept of public health surveillance, developments in four areas have changed the field: 1) national coordination, 2) technology and informatics, 3) expansion beyond communicable diseases, and 4) methodologic development. Through these, however, the core definition and integrity of surveillance practice have remained unchanged.

National Coordination of Public Health Surveillance

The United States Constitution leaves responsibility for public health practice primarily to the states as part of their police powers (6). The federal government, however, retains important roles. A major role in public health surveillance for CDC is to provide the national epidemiologic profile, through aggregation of surveillance data provided by the states, for the most important diseases and conditions. Having accurate and useful data requires that surveillance methods be coordinated across the 50 states and other independent jurisdictions that conduct data collection. Coordination includes establishment of consistent case definitions, collection methods, and population coverage; it requires that the data be deduplicated to avoid inaccurate counting and that additional case information be matched accurately to avoid data errors.

Recognition of the federal role in surveillance led to considerable work during the 1970s and 1980s, when national coordination became a major emphasis for public health surveillance. CDC and the Council of State and Territorial Epidemiologists (CSTE), initially convened by CDC as the Conference of State and Territorial Epidemiologists in 1952 to bring states together to address shared concerns regarding public health, annually spent hours in consultations and symposia working on ways to coordinate public health surveillance. A report released in 1977 (J.L. Gale, Surveillance data: quality, use and effect on public health divisions in local and state health departments, unpublished report, 1977) called for national surveillance activity coordination at CDC. A year later, in 1978, the Consolidated Surveillance and Communications Activity was established to respond to the recommendations of Gale's report. These activities fostered a new emphasis on the scientific bases of surveillance, including the introduction of new statistical methods (e.g., time-series analysis), formation of the Surveillance Coordination Group that included the major CDC programs and CSTE, and introduction of changes to the MMWR weekly and Annual Summary of Notifiable Diseases. These activities also led to the first comprehensive CDC plan for public health surveillance, which was created in conjunction with state partners and CSTE and appeared in 1985 (3). The plan was designed to be flexible, with quick and easy updating, done simply by the click of a three-ring binder and removal and reinsertion of paper copies of critical sections. This document started with a surveillance definition that expanded the one formulated by Langmuir and was agreed on by leaders of all programs at CDC, both infectious and noninfectious diseases, and by CSTE. The plan emphasized the importance of consistency in the seven steps that are now recognized as part of any surveillance system: 1) system design, 2) data collection, 3) collation, 4) analysis, 5) interpretation, 6) dissemination/communication, and 7) application to program.

National coordination of these steps was implemented in the mid-1980s, when the most complex and well-funded national surveillance system ever created in the United States began to track cases of a new devastating immune-compromising disease, acquired immunodeficiency syndrome (AIDS). What eventually became the National HIV/AIDS Surveillance System (7) began with great forethought and consideration of the utility and applicability of the data collected at the national and state levels. From the start, all cases reported to the system were subject to the same case definition (8), and changes to the case definition (9) went into effect uniformly on the same date in every state. The same data elements were collected on the same case report form in all states and reported by using the same software. A system of deduplication activities to ensure accurate case counting was implemented early and included two key tools. The first tool emanated from a CSTE resolution (10) and permitted cross-state communication of case information among the 50 states allowing public health surveillance personnel to establish whether similar-looking cases were the same individual reported more than once to the system. The second tool was special statistical programming conducted on the national database to search for possible duplicates (11). This coordination continues today in the National HIV/AIDS Surveillance System. Similar coordinated case reporting exists for other nationally notifiable diseases (e.g., tuberculosis).

Today, public health surveillance remains an activity of the states, but CDC continues to carry out its national role by coordinating national public health surveillance activities with the states, CSTE, and other partners, including the Association of State and Territorial Health Officers, the National Association of City and County Health Officers, the Association of Public Health Laboratories, the National Association for Public Health Statistics and Information Systems, and the World Health Organization (WHO). In 2009, these partners came together with CDC to discuss challenges and a new vision for the future of public health surveillance in the 21st century.

Technology, Informatics, and Public Health Surveillance

Technologic advances began to improve the timeliness and accuracy of public health surveillance in 1961 when CDC implemented weekly telegraphic reporting by states for cases of notifiable diseases. This technology remained state of the art until 1975, when telephone reporting of nationally notifiable diseases began. In 1981, in addition to routine postcard reporting, telephone reporting began including interactive data transfer to a computer of the aggregate numbers for publication in MMWR. In 1984, CDC and six states piloted the Epidemiologic Surveillance Project (ESP), which experimented with electronic transfer of individual, de-identified case record data to CDC. By 1989, all 50 states and selected territories were participating in the National Electronic Telecommunications System for Surveillance (NETSS), which still exists for data transfer of the majority of nationally notifiable diseases. This leap forward allowed unprecedented reductions in counting and transcription errors and began the ability to remove human error in several of the ongoing, systematic steps in a surveillance system (Figure).

Today, the role of public health informatics and information technology in public health surveillance is twofold: 1) to improve timeliness and completeness of data collection and analysis and 2) to free human resources to focus on the areas that require the most creative thought and to do the work that technology cannot. The idealized mix of technologic and human inputs into a public health surveillance system are illustrated in this report (Figure). With effective informatics tools, automated data systems can reach into electronic health records and extract data for public health surveillance, relieving the time-consuming and expensive "shoe-leather" data collection of chart reviews, paper forms, and morbidity cards that have characterized traditional reporting. Health information exchanges, which mobilize health information electronically across organizations within a jurisdiction, will provide a timely, efficient, and accurate means of data exchange and are an example of an informatics tool that holds considerable promise for public health.

During spring 1995, the CDC/ATSDR Steering Committee on Public Health Information and Surveillance System Development promulgated a blueprint for the agency's highest priority objective: the creation of integrated public health information and surveillance systems (12). The Steering Committee, comprising representatives from all centers, the institute, and offices at CDC, anticipated the impact of health reform and accompanying data collection and storage reforms and responded with sweeping recommendations for an integrated information and surveillance system. The blueprint envisioned coordinating the disparate and fragmented existing CDC surveillance systems to enhance functionality and efficiency. The purpose was to minimize the need for separate systems while maximizing the analytic value of the data for public health action. However, attaining a meaningful integrated information and surveillance system has proven more challenging than anticipated. Efforts continue to realize a fully functional integrated electronic health information system that begins at the clinical encounter and seamlessly connects through the ongoing activities of public health surveillance, with federal investments in electronic health records (13). Ensuring, through "meaningful use" requirements (14), that public health is at the collective table in formulating the requirements for software development is critical for the future of public health surveillance.

Electronic algorithms that collate data from disparate sources are critical to improving accuracy and timeliness as person-based surveillance records are connected across time. This is especially important in registry-based surveillance systems (e.g., HIV [7] and cancer [15]) where connecting subsequent events to the correct case is essential for accurate analyses. Using consistent statistical programs across jurisdictions and across time allows for timely and comparable analyses, which increasingly are important as the demands on public health surveillance data increase (e.g., distribution of resources according to disease burden, or support of public health program spending based on evidence of outcomes). In addition, new computer programs and applications can help public health programs better disseminate and communicate surveillance results. For example, they can help create understandable and interactive graphical representations of surveillance data that can tell stories to different audiences, including those untrained in health or public health (e.g., policymakers and the general public). Reaching such audiences is a critical step for using surveillance information for action, the last defining step of a public health surveillance system.

Technology assists public health practitioners by spreading information for action quickly and broadly, reaching program partners and others responsible for action. An example occurred at the start of the severe acute respiratory syndrome (SARS) epidemic in 2003, when the need for a practical, consistent case-finding tool quickly became evident. The Milwaukee Health Department was able to adapt an innovative informatics tool called the Regional Emergency Medical Internet (REMI) to help find and triage SARS cases. The tool was implemented rapidly and inexpensively in 27 hospital emergency departments (EDs) within 3 days after pilot-testing in a single Milwaukee hospital (16). REMI had been designed originally to assist EDs communicate when they must divert ambulances and had been adapted by the health department into a multi-ED surveillance system to tackle different syndromic illnesses, from heat-related syndromes to potential biologic terrorism occurrences during international sporting events (17). Another example of rapid, innovative adaptation of surveillance technology occurred during the 2010 Deepwater Horizon oil spill. CDC's BioSense syndromic surveillance system was used to help the five affected Gulf states monitor the health (including mental health) of affected populations after the spill. With a daily report from 86 coastal health-care facilities, BioSense assisted with ongoing, up-to-the-day evaluation of possible health concerns (18).

Continued use of public health informatics promises more efficiencies in public health surveillance. As time and mental energy are freed for the surveillance scientist to focus on developing and improving systems and applying evidence to program implementation, usefulness of public health surveillance will continue to increase.

Expansion of Public Health Surveillance beyond Communicable Diseases

Until 1970, the "CDC" acronym stood for the Communicable Disease Center, indicating the strict focus of CDC on prevention and control of communicable diseases. In 1970, the agency's name was changed to the Center for Disease Control; then in 1980, to the Centers for Disease Control; and finally, in 1992, to the Centers for Disease Control and Prevention. The name change in 1970 signaled an expansion of CDC's mission to include prevention of unnecessary illness and premature death from all causes, infectious and noninfectious. The focus of CDC's activities broadened to include prevention of the major chronic conditions, including heart disease, cancer, stroke, and unintentional injury, and their associated risk behaviors (e.g., smoking, sedentary lifestyle, inadequate nutrition, and use of passenger restraints). In 1984, a total of 15 states and CDC began collecting information monthly about risk behaviors related to the leading causes of death through the Behavioral Risk Factor Surveillance System (19). In addition, CDC and its surveillance partners began communicating findings for action, including descriptions of the new surveillance systems for injury (20), chronic diseases (21), and environmental health tracking (22). MMWR, seeking a way to standardize reporting of data from the increasing number and types of surveillance systems and condition-specific surveillance reports, began publishing a new series called CDC Surveillance Summaries in 1983, which continues today. The first issue of CDC Surveillance Summaries contained reports on multiple topics, including summer mortality from selected cities and counties as reported by medical examiners, temporal trends in malformation incidence reported to the birth defects monitoring program, and psittacosis cases in the United States in 1979 (23).

After the events surrounding September 11, 2001, interest increased in using surveillance methods to detect unusual health events that might indicate public health emergencies: naturally occurring or human-made. Three outgrowths of public health surveillance came from this. The first, syndromic surveillance, is defined as the ongoing, systematic collection, analysis, interpretation, and application of real-time (or near--real-time) indicators for diseases and outbreaks that allow for their detection before public health authorities otherwise note them (24). Syndromic surveillance has been enhanced by new technology and statistical methods that can help identify disease patterns that would not be noted otherwise. The second outgrowth, biosurveillance, stemmed from a 2007 U.S. homeland security presidential directive that addressed activities beyond the scope of public health surveillance to include data collection for event detection, enhanced collection and analysis for event characterization, further data collection for situation awareness, and additional data collection for investigation and recovery activities (25). The third outgrowth was the recognition that, with modern transportation, most of the world's populations live just one incubation period away from other persons on the planet, and the health of one population is related to the health of others. These developments have kept CDC closely involved in international health (see global health article in this issue), including international public health surveillance. In 1992, CDC and WHO sponsored a 3-day international symposium on public health surveillance. Held at the Carter Center in Atlanta, the symposium had three goals: 1) foster an understanding of the role of public health surveillance in reducing morbidity and mortality, 2) identify topics for further development at future meetings, and 3) bring experts together to describe a new global agenda for public health surveillance (26).

A decade later, on the heels of the SARS epidemic (27) and in the midst of threats of influenza pandemics, revision of the International Health Regulations in 2005 (IHR 2005) and their implementation in 2007 were crucial events for international public health surveillance and served as a tool for countries to communicate about possible international epidemics. IHR 2005 replaced the three notifiable diseases or pathogens listed in the original IHR, written in 1969, with a specifically defined "public health emergency of international concern" (28). IHR 2005 requires all member states to report a public health emergency of international concern within 24 hours. It also requires WHO to provide guidance and technical assistance to member states to develop and strengthen public health surveillance and response capacity. CDC participates in similar technical assistance activities, with 35 self-sustaining programs in 20 countries in which field epidemiology and laboratory training programs help educate local public health staff in surveillance methods as part of broader curricula since 1980 (29).

Advancement of Surveillance Methods

Throughout the past 50 years of surveillance activities, public health surveillance scientists have been developing methods to advance the field by coordinating methods among systems, applying advanced technology, and expanding systems to meet the surveillance mission. Methods advancement has occurred across the spectrum of the seven ongoing, systematic activities of a surveillance system (Figure).

In 1986, CDC developed a comprehensive plan for what was then called epidemiologic surveillance. This plan (30), developed by CDC's Surveillance Coordination Group, defined surveillance as follows:

The ongoing, systematic collection, analysis, and interpretation of health data is essential to the planning, implementation, and evaluation of public health practice, closely integrated with the timely dissemination of these data to those who need to know. The final link in the surveillance chain is the application of these data to prevention and control. A surveillance system includes a functional capacity for data collection, analysis, and dissemination linked to public health programs.

The 1986 plan included the first proposed method for evaluating a surveillance system (31), which was the precursor to the more formal Guidelines for Evaluating Surveillance Systems published in MMWR in 1988 (32) and its updated version, Updated Guidelines for Evaluating Public Health Surveillance Systems published in MMWR in 2001 (33).

The definition of public health surveillance has remained stable across time, even as public health experts have debated the purpose and meaning of surveillance. During the 1970s, Langmuir argued that the boundaries of surveillance stopped at "epidemiologic intelligence" and that it did not encompass all of epidemiology (e.g., investigations and research) (34). In 1988, Thacker and Berkelman suggested a new name, public health surveillance (35), to indicate its scope and context. In 2009, approximately 20 years after the last time the definition had been reconsidered, CDC gathered 100 surveillance scientists to discuss special topics in public health surveillance in the 21st century, including its definition. After careful consideration addressing the drivers of health information in the coming century, the group recommended maintaining the existing definition of public health surveillance because it remains applicable and flexible to accommodate public health needs across the spectrum of topic areas. However, the group recommended incorporating explicitly two key principles: 1) the purpose of the activity must be to address a defined public health problem or question and 2) the public health question(s) must exist a priori, that is there must be a planned public health purpose to the collection, storage, and use of the data.

A tenet of modern surveillance is that the utility of surveillance is determined largely by proper analysis of the data. Herman Biggs, the 19th century physician who pioneered public health surveillance in New York City, was known for insisting that collected data be used to improve health, not merely to keep "adding machines" busy (36). To be useful, surveillance data must be converted into information for public health action. Fortunately, the tools used for analysis have improved substantially since 1961. For example, the ability to differentiate "noise" from true aberrations in the data has been a problem keeping surveillance scientists occupied for years (37). This problem plays out in surveillance for influenza, a public health priority since 1918 when a system was established by the U.S. Public Health Service in 50 cities based on death certificates (and is still maintained today by CDC in 122 cities and published weekly in MMWR). Influenza surveillance was a priority for Langmuir, who worked with colleagues Serfling and Sherman to develop a seasonal regression model that could help analyze influenza mortality data more precisely than previous methods based on the moving average (38). In 1979, pneumonia and influenza data were modeled by using time-series analyses to identify aberrations in incidence (39,40); today, other systems (e.g., anthrax [41] and syndromic surveillance [42]) routinely use these methods to model surveillance data. Application of epidemiologic study designs to examine efficacy of different types of surveillance methods and approaches has also been accomplished. In the early 1980s, two innovative randomized clinical trials evaluated active surveillance strategies compared with passive reporting. Both studies, one in Vermont (43) and one in Monroe County, New York (44), demonstrated substantial improvements in completeness using active surveillance strategies for communicable diseases. Differences in improvement were observed by disease and report source, leading to the conclusion that in the analysis of surveillance data, knowing and attending to the local context is desirable. This conclusion remains critically important today.

By the early 1990s, many schools of public health in the United States had begun to focus on the science of public health surveillance, and the lack of a textbook was obvious. Until Public Health Surveillance was published in 1992 (45) and the first edition of Principles and Practice of Public Health Surveillance was published in 1994 (46), surveillance practitioners were able to rely only on journal articles, consultations convened by CDC, and professional exchanges to share methodologic advances and preferred practices. Now the Principles text is in its third edition (47), and additional texts have been published, including one devoted to statistical principles and methods of public health surveillance (48) and another to infectious disease surveillance (49). As the science of public health surveillance continues to evolve and the tools of public health informatics become integral to the work of surveillance practitioners, methods will continue to develop that enable the public health epidemiologist to put data to use in the most effective way.

The Future of Public Health Surveillance

Evidence-based decision making in public health begins with surveillance---and the demands on health data continue to increase. The ways of knowing about the health of a community also continue to evolve as information technology eases the effort to collect, collate, store, analyze, and disseminate data. The integrity of the discipline of public health surveillance has held fast for the past 50 years and most likely will continue for the next 50 and beyond. The tools available to public health surveillance practitioners and scientists will change as technology improves efficiency and frees practitioners to attend to creative problem solving in such critical areas as program planning and applying data to action. CDC will continue to evaluate its efforts and move the field forward, welcoming the opportunities that lie ahead.

References

1. Surveillance [editorial]. Int J Epidemiol 1976;5:3--7.
2. Thacker SB. Historical development [Chapter 1]. In: Lee LM, Teutsch SM, Thacker SB, St Louis ME, eds. Principles and practice of public health surveillance. 3rd ed. New York, NY: Oxford University Press; 2010: 1--17.
3. CDC. Manual of procedures for national morbidity reporting and public health surveillance activities. Atlanta: US Department of Health and Human Services, CDC; 1985.
4. World Health Organization. The global eradication of smallpox: final report of the Global Commission for the Certification of Smallpox Eradication. Geneva, Switzerland: World Health Organization; 1980.
5. Langmuir AD. The surveillance of communicable diseases of national importance. N Engl J Med 1963;268:182--92.
6. Goodman RA, Kocher PL, O'Brien DJ, Alexander FS. The structure of law in public health systems and practice [Chapter 2]. In: Goodman RA, Hoffman RE, Lopez W, Matthews GW, Rothstein MA, Foster KL, eds. Law in public health practice. 2nd ed. New York, NY: Oxford University Press; 2007:45--69.
7. Glynn MK, Lee LM, McKenna MT. The status of national HIV case surveillance, United States 2006. Public Health Rep 2007;122(Suppl 1):63--71.
8. CDC. Update on acquired immune deficiency syndrome (AIDS)---United States. MMWR 1982;31:507--8, 513--4.
9. CDC. Revised surveillance case definitions for HIV infection among adults, adolescents, and children aged <18 months and for HIV infection and AIDS among children aged 18 months to <13 years---United States, 2008. MMWR 2008;57(No. RR-10).
10. Council of State and Territorial Epidemiologists (CSTE), Infectious Disease Committee. HIV case reporting: reciprocal notification. Atlanta, GA: Council of State and Territorial Epidemiologists; 2001. Position statement 2001-ID-04. Available at http://www.cste.org/dnn/AnnualConference/PositionStatements/tabid/191/Default.aspx.
11. Glynn MK, Ling Q, Phelps R, Lee LM. Accurate monitoring of the HIV epidemic in the United States: case duplication in the national HIV/AIDS surveillance system. J Acquir Immune Defic Syndr 2008;47:391--6.
12. CDC. Integrating public health information and surveillance systems. Atlanta, GA: US Department of Health and Human Services, CDC; 1995. Available at http://www.cdc.gov/nedss/Archive/katz.pdf.
13. Cunningham R. Stimulus bill implementation: expanding meaningful use of health IT. NHPF Issue Brief 2009;(834):1--16.
14. Anonymous. Meaningful use---final at last. Final rule adds flexibility to objectives, eases measures. J AHIMA 2010;81(8):18--9.
15. CDC. National Program of Cancer Registries. United States cancer statistics, technical notes. Atlanta, GA: US Department of Health and Human Services, CDC; 2010. Available at http://www.cdc.gov/cancer/npcr/uscs/2006/technical_notes/.
16. Foldy S, Barthell EN, Silva J, et al. SARS surveillance project---Internet-enabled multiregion surveillance for rapidly emerging disease. In: Syndromic surveillance: reports from a national conference, 2003. MMWR 2004;53(Suppl):215--20.
17. Foldy SL. Linking better surveillance to better outcomes. In: Syndromic surveillance: reports from a national conference, 2003. MMWR 2004;53(Suppl):12--7.
18. CDC. Emergency preparedness and response: health surveillance---Gulf oil spill 2010. Atlanta, GA: US Department of Health and Human Services, CDC; 2010. Available at http://www.bt.cdc.gov/gulfoilspill2010/2010gulfoilspill/health_surveillance.asp.
19. CDC, National Center for Chronic Disease Prevention and Health Promotion. Behavioral Risk Factor Surveillance System---BRFSS history. Atlanta, GA: US Department of Health and Human Services, CDC; 2008. Available at http://www.cdc.gov/brfss/history.htm.
20. Graticer PL. The development of state and local injury surveillance systems. J Safety Res 1987;18:191--8.
21. Thacker SB, Stroup DF, Rothenberg RB. Public health surveillance for chronic conditions: a scientific basis for decisions. Stat Med 1995;14:629--41.
22. Thacker SB, Stroup DF, Parrish RG, Anderson HA. Surveillance in environmental public health: issues, systems, and sources. Am J Public Health 1996;86:633--8.
23. CDC. [Multiple reports.] MMWR Surveillance Summaries 1983;32(No. SS-1):1--42.
24. Sosin DM. Syndromic surveillance: the case for skillful investment. Biosecur Bioterror 2003;1:247--53.
25. Sosin DM, Hopkins RS. Public health surveillance for preparedness and emergency response: biosurveillance for human health [Chapter 14]. In: Lee LM, Teutsch SM, Thacker SB, St Louis ME, eds. Principles and practice of public health surveillance. 3rd ed. New York, NY: Oxford University Press; 2010:306--20.
26. CDC. Proceedings of the 1992 International Symposium on Public Health Surveillance. MMWR 1992;41(Suppl):1--218.
27. CDC. Outbreak of severe acute respiratory syndrome---worldwide, 2003. MMWR 2003;52:226--8.
28. World Health Organization (WHO). Communicable disease surveillance and response, international health regulations (2005). Geneva, Switzerland: WHO; 2008. Available at http://www.searo.who.int/en/section10/section369_9695.htm.
29. CDC. Center for Global Health. 2010 Field epidemiology and management capacity building programs [poster]. Atlanta, GA: US Department of Health and Human Services; CDC; 2010. Available at http://www.cdc.gov/globalhealth/FETP/pdf/factoid_poster.pdf.
30. CDC. Comprehensive plan for epidemiologic surveillance. Atlanta GA: US Department of Health and Human Services, CDC; 1986.
31. Thacker SB, Parrish RG, Trowbridge FL. A method for evaluating systems of epidemiological surveillance. World Health Stat Q 1988;41:11--8.
32. Klaucke DN, Buehler JW, Thacker SB, et al., and the Surveillance Coordination Group. Guidelines for evaluating surveillance systems. MMWR 1988;37(No. SS-5).
33. CDC. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR 2001;50(No. RR-13).
34. Langmuir AD. Evolution of the concept of surveillance in the United States. Proc R Soc Med 1971;64:681--4.
35. Thacker SB, Berkelman RL. Public health surveillance in the United States. Epidemiol Rev 1988;10:164--90.
36. Fairchild AL, Bayer R, Colgrove J. Searching eyes: privacy, the state, and disease surveillance in America. Berkeley, CA: University of California Press; 2007.
37. CDC. Proposed changes in format for presentation of notifiable disease report data. MMWR 1989;38:805--9.
38. Serfling RE, Sherman IL, Houseworth WJ. Excess pneumonia-influenza mortality by age and sex in three major influenza A2 epidemics, United States, 1957--58, 1960 and 1963. Am J Epidemiol 1967;86:433--41.
39. Choi K, Thacker SB. An evaluation of influenza mortality surveillance, 1962--1979: I. Time series forecasts of expected pneumonia and influenza deaths. Am J Epidemiol 1981;113:215--26.
40. Choi K, Thacker SB. An evaluation of influenza mortality surveillance, 1962--1979: II. Percentage of pneumonia and influenza deaths as an indicator of influenza activity. Am J Epidemiol 1981;113:227--35.
41. Buckeridge DL, Switzer P, Owens D, Siegrist D, Pavlin J, Musen M. An evaluation model for syndromic surveillance: assessing the performance of a temporal algorithm. In: Syndromic surveillance: reports from a national conference, 2004. MMWR 2005;54(Suppl):109--15.
42. Ozonoff A, Forsberg L, Bonetti M, Pagano M. Research methods bivariate method for spatio-temporal syndromic surveillance. In: Syndromic surveillance: reports from a national conference, 2003. MMWR 2004;53(Suppl):59--66.
43. Vogt RL, LaRue D, Klaucke DN, Jillson DA. Comparison of an active and passive surveillance system of primary care providers for hepatitis, measles, rubella, and salmonellosis in Vermont. Am J Public Health 1983;73:795--7.
44. Thacker SB, Redmond S, Rothenberg RB, Spitz SB, Choi K, White MC. A controlled trial of disease surveillance strategies. Am J Prev Med 1986;2:345--50.
45. Halperin W, Baker EL, eds. Public health surveillance. New York, NY: Van Nostrand Reinhold; 1992.
46. Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance. New York, NY: Oxford University Press; 1994.
47. Lee LM, Teutsch SM, Thacker SB, St Louis ME, eds. Principles and practice of public health surveillance. 3rd ed. New York, NY: Oxford University Press; 2010.
48. Brookmeyer R, Stroup DF, eds. Monitoring the health of populations: statistical principles and methods for public health surveillance. New York, NY: Oxford University Press; 2004.
49. M'ikanatha NM, Lynfield R, Van Beneden CA, de Valk H. Infectious disease surveillance. Malden, MA: Blackwell; 2007.

All MMWR HTML versions of articles are electronic conversions from typeset documents. This conversion might result in character translation or format errors in the HTML version. Users are referred to the electronic PDF version (http://www.cdc.gov/mmwr) and/or the original MMWR paper copy for printable versions of official text, figures, and tables. An original paper copy of this issue can be obtained from the Superintendent of Documents, U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800. Contact GPO for current prices.

**Questions or messages regarding errors in formatting should be addressed to [email protected].