ERS Data Product Quality Reviews

The ERS Data Product Review Council (DPRC) completed its sixth round of data product reviews, and has more reviews planned for Summer-Fall 2017 (see below). The review process is used to evaluate data products and their adherence to the Data Quality Standards using the ERS Data Product Quality Review: Evaluation Checklist.

Completed Reviews

Date: Winter 2016/17

Council members: Mark Denbaly (Chair), Lewrene Glaser, Stephen MacDonald, Mary Maher, Daniel Milkove, Katherine Ralston, Utpal Vasavada, and Pheny Weidman

County Typology Codes classify all U.S. counties according to six mutually exclusive categories of economic dependence and six overlapping categories of policy-relevant themes. They provide a convenient tool to summarize rural economic and social conditions, and are thus used by other Federal agencies and academia.

Review summary: While the Council found the product to be compliant with many of the quality standards, there is room for improvement in a few areas. Information should be provided to users on differences between ERS's codes and Treasury’s Community Development Financial Institutions Fund (CDFI) codes for Persistent Poverty Counties. The Council also recommends improving branding in the Excel files, creating a list of key and representative stakeholders that is available to the Branch Chief, adding a general description of the probability of misclassification to the documentation with links to accuracy measures for the input data sources, and conducting an external review of methods and research to improve methods. The Council notes that 508 testing has not been done on the product.

Dairy Data covers U.S. supply, demand, and trade for milk and major dairy products.

Review summary: The Council recommends a number of improvements to the data product, including more transparent and comprehensive documentation of all tables and data sources. The Council also suggests using web metrics to inform product usage and product refinement, as well as reporting accuracy measures and/or adding links to the documentation to source agency data-quality pages. Review procedures for ERS-generated estimates should be documented and available upon request. The Council would like the Market and Trade Economics Division to consider interagency agreements with the sources of nonpublic estimates wherever appropriate rather than relying solely on informal arrangements.

Season-Average Price Forecasts provides three Excel file spreadsheet models that use futures prices to forecast the U.S. season-average price received and the implied Price Loss Coverage (PLC) payment rate for three major field crops (corn, soybeans, and wheat).

Review summary: The Council found the data product to have good usability and documentation. The Council's top recommendation is to add information on accuracy of the estimates. In addition, the Council suggests documenting and making available on request the procedures for pre-dissemination review and prerelease security because the data product draws on proprietary data. Consultation with Market and Trade Economics Division, Office of the Director is encouraged to address accessibility issues.

Eating and Health Module is a supplement to the Bureau of Labor Statistics American Time Use Survey that captures information on secondary eating and drinking, food shopping and preparation, health, and household income.

Review summary: The Council found the product to be in compliance with most of the data quality standards. The Council recommends adding accuracy measures for all ERS-generated estimates, documenting the review process and making the documentation available on request, providing advance notice of the next release on the product’s overview page, and conducting an independent, external review of methods of ERS-generated estimates.

Price Spreads from Farm to Consumer estimates farm-to-retail price spreads to measure the costs of processing and marketing foods. They are calculated as the difference between the prices farmers receive for the raw commodities they sell and the prices consumers pay for equivalent food products. Estimates are reported for individual foods and food baskets that represent a typical household’s purchases over 1 year.

Review summary: As a premier ERS data product, the product is held to a high standard of data quality. While the product meets many of the key standards, the Council noted potential improvements. The Council recommends usability testing and an external review of quality of communication and methods. The Council also suggests developing a data management plan, providing accuracy measures of reported estimates, and developing procedures to protect proprietary data from involuntary disclosure.

Date: Spring 2016

Council members: Lewrene Glaser (Chair), Mark Denbaly, Hisham El Osta, Karen Hamrick, Stephen MacDonald, Mary Maher, Utpal Vasavada, and Pheny Weidman

Farm Income and Wealth Statistics provides the latest U.S. farm-sector income and wealth statistics data. The data include historical U.S. and State-level farm income and wealth estimates as well as U.S.-level forecasts for the current calendar year.

Review summary: As a premier ERS data product, the product is held to a high standard of data quality. The Council found that nearly all aspects of the data product were in compliance. However, there is room for improvement in a few areas. While the web tables are branded, the Excel summary tables and some application-download formats are not. The Excel tables do not cite sources, and this could be remedied by providing this information.

Food Access Research Atlas provides spatial indicators of food access in low-income and other census tracts, and enables users to create maps depicting food access for selected populations by census tract using different measures and indicators of supermarket accessibility. The data can be downloaded for community planning or research purposes.

Review summary: As a premier ERS data product, the product is held to a high standard of data quality. While the product meets many of the standards, the Council noted a few potential improvements. The map images and spreadsheets can benefit from adding branding (and title and source) information. In addition, the Council recommends that the quarter and year in which the product is targeted for its next release be announced in advance on the ERS calendar.

Food Expenditures annually measures total U.S. food expenditures, including purchases by consumers, governments, businesses, and nonprofit organizations. ERS developed the series in 1987, and data are available from 1929 through 2014.

Review summary: The Council recommends that the Food Economics Division (FED) consider revising the data product to ensure that it satisfies agency data-quality standards while committing resources and economic expertise to sufficiently support the product. A decision by FED to retain the data product should 1) ensure that the objectivity criteria are met; 2) update the data production process; 3) revise overview and documentation to incorporate information on the international component; 4) identify and make available a list of the key stakeholders while enhancing the communication quality of the information; 5) assess and document pre-dissemination review and pre-release security processes; and 6) validate that accessibility elements satisfy the open data requirements.

Meat Price Spreads provides monthly average price values, and the differences among those values, at the farm, wholesale, and retail stages of the production and marketing chain for selected cuts of beef, pork, and broilers.

Review summary: The Council recommends a number of improvements to the product in order to meet ERS’s data quality standards. This includes an external review of methods, using web metrics to inform product usage and product refinement, and adding links to the documentation to source agency data-quality pages. The procedures for storage, security, and processing of nonpublic data and for review and pre-release security should be documented and available on request. This important product has the potential to become a premier product with even further investment by the Market and Trade Economics Division (MTED) and the Data Product Manager.

Organic Prices offers wholesale prices for select organic and conventional fruits and vegetables, first receiver prices for organic poultry and eggs, and f.o.b. and spot market prices for organic grain and feedstuffs.

Review summary: The Council suggests the next release of this data product be branded with complete citation of data sources, and be announced in advance on the ERS calendar. Clarity about the internal and external stakeholders of this product can be enhanced. It is unclear whether the current review process applies to both the methodology and the data. Review procedures for the data should be documented and made available upon request.

Phytosanitary Regulation identifies which countries, under USDA’s Animal and Plant Health Inspection Service phytosanitary rules, are eligible to export to the United States the fresh fruits and vegetables that are most important in the American diet.

Review summary: The Council recommends that MTED evaluate the resource commitment needed to sufficiently support the data product in a way that satisfies ERS data quality standards or archive it. If MTED continues to maintain the product, the Council recommends compiling a list of key and representative stakeholders that is available to the branch chief and division management. The procedures for storage, security, and processing of nonpublic data and for review and pre-release security should be documented and made available upon request. Also, the Council recommends, in the next release of the data product, improving the branding of the data tables with a complete and comprehensive citation of data sources. If the product is not archived, a data management plan to address Federal Open Data Guidelines is needed.

Date: Fall 2015

Council members: Lewrene Glaser (Chair), Mark Denbaly, Hisham El Osta, Karen Hamrick, Stephen MacDonald, Mary Maher, Utpal Vasavada, and Pheny Weidman

Adoption of Genetically Engineered Crops in the U.S. summarizes the adoption of herbicide-tolerant and insect-resistant crops since their introduction in 1996. Data for 2000-15 cover genetically engineered varieties of corn, cotton, and soybeans for the United States and States.

Review summary: While the Council found the data product was generally in compliance, some areas warrant further attention. The Council recommends providing the data in open, machine-readable formats. Although USDA’s National Agricultural Statistics Service (NASS) is cited as the source of the data, the tables are not branded as coming from ERS. Moreover, revisions in the NASS surveys and/or ERS adjustments should be separately documented. Also note that monitoring web stats can help guide improvements to the data product and provide information on usage patterns.

Agricultural Trade Multipliers provide estimates of employment and/or output effects of trade in farm and food products on the U.S. economy. These effects, when expressed as multipliers, reflect the amount of economic activity and/or jobs generated by agricultural exports.

Review summary: To better identify the estimates as coming from ERS, the Council recommends that the ERS Estimates and the Calculator output tables be sourced as coming from ERS. The estimates and calculator output should also be in format that is machine readable. As a premier ERS data product, the product is held to a high standard of data quality, and thus, processes for review and pre-release security need to be documented and available upon request. In the future, should any substantial modifications be made to the model, prior notice on the ERS calendar and on the data product’s overview page is recommended.

Atlas of Rural and Small-Town America assembles statistics on people, jobs, agriculture, and county types. Data have been updated to reflect the latest population, poverty, and American Community Survey statistics.

Review summary: While the Council found nearly all aspects of the data product were in compliance, it found that some areas warrant attention. The Council suggests adding sources to the Read Me tab in the Excel files and including the next update date on the product’s overview page. In addition, the Council recommends elaborating on the statistical methods and operational procedures used to derive the compiled data based on the Resource and Rural Economics Division research program on rural America.

Cost Estimates of Foodborne Illnesses offers detailed data about the costs of major foodborne illnesses in the United States, updating and extending previous ERS research.

Review summary: This is a new data product, and as such, meets most of the standards. However being new, the data product has not needed to meet some of the standards, such as advance notice of substantial changes. The Council encourages keeping these standards in mind going forward. The Council recommends providing data in an open format, undertaking further research on model performance, adding to the documentation a discussion of the measures of statistical accuracy and/or the limitations of the data, and reviewing pre-release security procedures.

Food Price Outlook provides ERS food price forecasts for the short-term period. The Consumer Price Index (CPI) for food is probably the most widely used indicator of changes in retail food prices.

Review summary: As a premier ERS data product, it is held to a high standard of data quality. In nearly all respects, the Council was satisfied with the present system. However, the Council recommends providing the data in open, machine-readable formats. The Council also suggests documenting internal processes for review and pre-release security and the list of stakeholders and ensuring they’re available to others in the branch and division.

Rice Yearbook offers data on U.S. rice production, supply, disappearance, trade, and prices. Includes State acreage, yield, and production data; U.S. and world price series; program statistics; and world supply and use estimates.

Review summary: The Council strongly recommends a separate, comprehensive documentation page, including items such as the methodology and revision history for ERS-generated estimates, differences in ERS data series vs. those published by other agencies, and links to other agencies’ web pages that discuss their data quality procedures. In addition, the data product should cite ERS as the source of the tables. An external review of methods is recommended for ERS-generated estimates. Consideration should be given to releasing data in an open data format (e.g., CSV).

Date: Spring 2015

Council members: Mitch Morehart (Chair), Mark Denbaly, John Dyck, Lewrene Glaser, Karen Hamrick, Mary Maher, Utpal Vasavada, and Pheny Weidman

China Agricultural and Economic Data is a collection of agricultural-related data from official statistical publications of the People's Republic of China. There are 250 data items available at the national level, and 45 crop, livestock, agricultural output, and population items available at the provincial level.

Review summary: Although this is a thoughtful and valuable product, there is no evidence that the product has been updated since 2011 or that there is a plan to update. The Council recommends that management evaluate the resource commitment needed to support the data product or archive it so that it remains accessible to users. If management decides to keep the product, some of the key steps would be to update the data regularly, identify key stakeholders and monitor usage, and carry out an external review for communication quality and a review of methods.

The Food Availability (Per Capita) Data System includes three distinct but related data series on food and nutrient availability for consumption. The data serve as popular proxies for food consumption at the national level. Food availability data are the foundation for loss-adjusted food availability and nutrient availability data.

Review summary: The data product is fundamental to understanding U.S. food consumption. In general, the product is well organized, presented, and documented. As a premier ERS data product, the product is held to a high standard of data quality. In most respects, the Council was satisfied with the present system. However, more work is recommended in formal review and in interaction with users. Providing advanced notice of planned releases and methodology changes on the ERS website and explicit guidance for users on update and revision history should all be undertaken. The current effort to provide an open data format should be expedited.

Western Irrigated Agriculture summarizes the farm-structural characteristics for irrigated farms in the 17 Western States based on USDA's 2008 and 1998 Farm and Ranch Irrigation Surveys.

Review summary: The product summarizes data from USDA’s Farm and Ranch Irrigation Survey organized by ERS farm size typology. The product makes irrigation information by State and farm-size category available to policymakers, researchers, farmers, and the public. The Council recommends an external review by independent experts of data-product methodology and quality of communication for a broader audience beyond irrigation experts. Advanced notice of the next release and provision of the data in an open-data format should be undertaken.

Date: Fall 2014

Council members: Mitch Morehart (Chair), Mark Denbaly, John Dyck, Lewrene Glaser, Karen Hamrick, Mary Maher, Utpal Vasavada, and Pheny Weidman

Feed Grains Database provides statistics on four feed grains (corn, grain sorghum, barley, and oats), foreign coarse grains (feed grains plus rye, millet, and mixed grains), hay, and related items. This includes data published in the monthly Feed Outlook and previously annual Feed Yearbook.

Review summary: The Feed Grains Database is an important component of USDA’s monitoring of the grain, oilseed, and livestock complex. Menus on the Custom Query page illustrate the breadth of the data provided and give users easy access to data of their choice. The Data Product Review Council recommends the documentation for ERS-generated data series being expanded to include complete and thorough information on methods and accuracy measures. The Council also recommends an external review of methods be conducted, and encourages engaging stakeholders and addressing their needs.

Food Dollar Series measures annual expenditures by U.S. consumers on domestically produced food. The series is comprised of three primary components, each showing different ways to split up the same food dollar: the marketing bill, the industry group, and the primary group.

Review summary: Having recently undergone substantial revision, the Food Dollar Series is an important educational and analytical tool that uses data from other agencies. Its design and user menus are effective in achieving educational and analytical goals. While the Council found that this product met most of the quality standards, improvements could be made by providing advance notice of substantial changes, creating an archive of previous releases, and ensuring complete citation of sources in the data tables.

State Fact Sheets provide information on population, income, poverty, food security, education, employment, organic agriculture, farm characteristics, farm financial indicators, top commodities, and exports. Updated items include 2012 Census of Agriculture data and unemployment for 2013.

Review summary: The State Fact Sheets provide a concise service to a wide variety of users and magnify the effectiveness of the data by making them more visible. While the Council found the product to be at near full compliance with the quality standards, it recommends implementing more pre-dissemination review and building the proposed documentation page. In addition, the data product could benefit from communicating upcoming releases in advance and developing an archive of previous releases.

Date: Spring 2014 (pilot round)

Council members: Mitch Morehart (Chair), Patrick Canning, Mark Denbaly, Lewrene Glaser, Mary Maher, Utpal Vasavada, and Pheny Weidman

Agricultural Productivity in the U.S. provides statistics of total factor productivity (TFP), along with estimates of 10 outputs and 12 inputs in the U.S. farm sector for the 1948-2011 time period.

Review summary: While the Data Product Review Council found that the data product met most of the quality standards, concerns were raised about data product documentation. The Council recommended that there was scope to improve the documentation to meet OMB Open Data Guidelines. The Council’s concern will be properly addressed in a revision of U.S. productivity data product website documentation and exploration of alternative dissemination approaches as will other recommendations coming out of an external expert panel review.

Commodity Costs and Returns provide estimates for 12 major crop and livestock products since 1975. ERS has been the sole source of national-level commodity costs and returns—the only set of estimates that is consistently evaluated across the United States and utilizes a sampling structure ensuring that the estimates are representative of commodity production in the Nation and major production regions.

Review summary: While the Council found that the data product met most of the quality standards, attention to OMB’s new Open Data Guidelines is needed as well as evidence of ongoing research into methods and operations. In addition, the Council suggested finding ways in engaging stakeholders to improve the product’s fitness for use.

Food Security in the United States provides annual estimates of food insecurity for the Nation overall and by selected household characteristics, and for States (three years of data are combined for State estimates). ERS is the preeminent source of U.S. food security estimates. Food security for a household means access by all members at all times to enough food for an active, healthy life.

Review summary: The Council found that the product meets all OMB directives and best practices by Federal Principal Statistical Agencies. The Council recommended that OMB’s new Open Data Guidelines be implemented. In addition, the Council suggested finding ways that would make the documentation, the accuracy measures, the revisions and substantial changes, and the archived material more readily accessible on the website.

Upcoming Reviews

Date: Summer-Fall 2017

Council members: Mark Denbaly (Chair), Bryce Cooke, Lewrene Glaser, Mary Maher, Dan Milkove, Katherine Ralston, Utpal Vasavada, and Pheny Weidman

ARMS – Farm Financial: USDA's Agricultural Resource Management Survey (ARMS) is the primary source of information on the financial condition, production practices, resource use, and economic well-being of farm households. The December 8, 2016, release includes information on farm finances from the 2015 ARMS survey, now available through the Farm and Household Finance Tailored Reports.

FoodAPS National Household Food Acquisition and Purchase Survey: USDA's National Household Food Acquisition and Purchase Survey (FoodAPS) is a nationally representative survey of household food purchases and acquisitions.

Food Environment Atlas: The Food Environment Atlas is a web-based mapping tool developed by ERS that allows users to compare U.S. counties in terms of their "food environment" – indicators (most at the county level) that help determine and reflect a community’s access to affordable, healthy food.

International Food Security: This dataset is the basis for the International Food Security Assessment, 2017-27, released in July 2017. It provides annual country-level data on production, consumption, and trade of grains and root and tuber crops, food aid, and macroeconomic variables for 76 countries.

International Agricultural Productivity: This data product provides agricultural output, input, and total factor productivity (TFP) growth rates across the countries and regions of the world in a consistent, comparable way for 1961-2013.

Major Land Uses: The Major Land Uses (MLU) series is the longest running, most comprehensive accounting of all major uses of public and private land in the United States. The MLU series contains acreage estimates of major uses by region and States, beginning in 1945 and published about every 5 years since.

Oil Crops Yearbook: Oilseed, oilmeal, and fats and oils supply and use statistics. Includes oilseed acreage, yield, and production estimates and farm and wholesale price series.