ACS Data Users Group

Hello:

I work for a news organization that wants to write about some of the 5-year ACS demographics, especially poverty and income, for local towns in our area. There are 340 municipalities that we cover. If we look at the main estimates for 2013-17, and compare the numbers with a non-overlapping time period such as 2006-10, we see some striking changes in income and poverty, but the margins of error give us pause.

We believe our readers would be interested in changes in income and poverty in their towns, and we can say that the top-line numbers are estimates, and are subject to margins of error. But let’s take a town in our region, Bellmawr, N.J., which has about 11,500 people. Poverty rose from 10 percent in 2006-10 to 20 percent in 2013-17, or an increase of 10 percentage points. With the error margins, however, that change could be as small as +2 percentage points to +19 on the high end.

So would you stay away from these numbers, publish them with the margins of error, aggregate towns into large sub-county geographies, or do something else? Of course, we would be doing on-the-ground reporting as well.


Thanks,

John

Just a couple of comments.
1. Well, margins of error are usually larger for smaller entities, so unfortunately you are likely to get large MoEs with smaller towns.
2. You wrote "aggregate towns into large sub-county geographies". Aren't towns and cities the largest subcounty geographies? I thought counties were divided into MCDs (minor civil divisions), which are towns and cities.

Gene:

On your second comment, I should have written "aggregate towns into CUSTOM sub-county geographies." I'm considering taking groups of MCDs and creating custom geographies, such as "Lower Bucks County" or "Gloucester County river towns."

John

The first thing I would do is smooth the data for public consumption. Round median income to '00s (e.g. $76,300 rather than $76.342). Do NOT use tenths of percentages.

I also agree with your idea of aggregating MCDs into useful larger geographic areas. It you can get them to areas of, say, 100K or more, the MOEs should go down considerably. It's useful to create such geography and use it consistently over a number of years, so the public gets used to it.

Finally, I would NOT publish the MOEs. They'll just confuse the readers. MOEs are our responsibility to worry about and we can't shove that onto the end users of the data.

Patty Becker

John,

I would include margins of error (MoEs), for the following reasons:

1. One thing you don't want communities to do is to say, well, we are doing better than is community X, because their poverty rate is 23.3% and ours is 21.2%. If the MoEs are largish and they overlap by quite a lot between the two communities, then neither can really say one is doing "better" or "worse" than is the other. If you can give them a bit of education, then hopefully they can better understand what the numbers mean and how they can and cannot use them.

2. I see quite a lot of surveys reported in the news, and often if it's a major news organization and they are reporting from Roper or some main survey group, the story includes margin of error, and sometimes even a brief explanation of what the MoE means. I hope that means that MoE is now enough in the public so that many people understand it to some degree.

Gene
(from home, so I am not representing or speaking for any organization)

Reporting MOEs and other elements of methodology are requirements of the AAPOR (American Association for Public Opinion Research) Transparency Initiative. See www.aapor.org/Transparency_Initiative.htm. Organizations self-select into doing this, and being a member of the TI has non-tangible benefits such as having a prior for smaller polling error by Nate Silver (he uses TI membership as a predictor in his models, and it moves the needle in noticeable ways).

Transparency: sounds good! The other part is writing about it well enough to that it is understandable to the general public. A tough part.

Reporting MOEs for surveys, in an appendix to a report, is a whole different thing than reporting line by line, item by item, MOEs in a presentation of ACS data. The CENSUS BUREAU is doing what AAPOR suggests that all survey producers do. That doesn't mean that intermediary users of the census data need to report MOEs to the general public. We just need to make sure that we're not releasing bad data.