Live After Quit

Our article in the Applied Economic Perspectives and Policy’s “sizing up survey samples” symposium (?) is published online

This paper resulted from the BP/Deepwater Horizon damage assessment conducted for the state of Florida (Huffaker, Clouwer and Larkin 2012). We used the Knowledge Networks (KN) internet panel for the sample (besides me, the authors are Andrew Ropicki, John Loomis, Sherry Larkin, Tim Haab, and Sergio Alvarez). At one point I remember Sherry Larkin asking all of us if it was OK if KN filled out our sample needs (n=2000) with opt-in data (I don’t recall what they called it back then). We all said fine, assuming KN knew what they were doing and it was the gold standard, right? The report sat without a paper carved out of it until I started using opt-in samples, noticed the different samples in the Florida data and compared models using the two different samples (*note). I presented the results at the 2019 Society for Benefit-Cost Analysis Annual Meeting and the paper sat a while longer. We finally submitted it to AEPP for a symposium on samples in stated preference research in the Fall of 2021 and it was accepted last month.

We find that the opt-in sample data leads to higher willingness to pay estimates and statistically insignificant scope effects (**note). The scope elasticity with the probability-based sample is 0.73 and statistically significant (***note). The income elasticity is higher with the probability-based sample too. 

There are three other papers in the symposium (not sure if AEPP is calling it a symposium, but I’m sure that it is a group of articles that are being published together) that are all good reads and should inform future research with internet panel data (which, I think, is the future). While I think opt-in data is inferior to probability-based panels, I’m not sure if the added cost is justified. This is especially true with more modern survey designs, relative to the NOAA panel era split-sample single binary choice scope test (repeated questions as CVM merges with discrete choice experiments), and methods to deal with the lower quality data (e.g., attribute non-attendance). One article in the symposium makes the comparison with this type of data (sans attribute non-attendance models). A third article compares high quality and convenience sample data and the fourth considers bots and other threats to data quality with internet data (Goodrich et al. 2023 — the other two haven’t appeared on AEPP’s website yet). 

Here is the link to our paper: https://doi.org/10.1002/aepp.13352

Notes: 

*A recreation demand paper was previously published out of this report (Whitehead et al. 2018).

**This result might be due to low sample size. The scope elasticity is 0.37 which isn’t crazy low. In other words, a huge sample size would reduce the standard error so that we could claim statistical significance. For example, see the federal study that has a sample size of n>3000 and a modest scope effect (the result becomes statistically insignificant if you randomly choose samples of n=1000, more on this someday). 

***I have another data set where GfK (formerly KN) did this. We are estimating CVM recreation trip models with repeated questions and variations in the choice attributes. I hope to write up the paper and submit it somewhere this summer. 

References:

Bishop, Richard C., Kevin J. Boyle, Richard T. Carson, David Chapman, W. Michael Hanemann, Barbara Kanninen, Raymond J. Kopp et al. “Putting a value on injuries to natural assets: The BP oil spill.” Science 356, no. 6335 (2017): 253-254.

Goodrich, Brittney, Marieke Fenton, Jerrod Penn, John Bovay,and Travis Mountain. 2023.“Battling Bots: Experiences and Strategies to MitigateFraudulent Responses in Online Surveys.”Applied Economic Perspectives and Policy1–23.

Huffaker, R. G., R. L. Clouser, and S. L. Larkin. 2012. Contract for Analytical Services Related to the Deepwater Horizon Disaster: Estimation of Lost Indirect and Passive Use Economic Values to Floridians. Final Report to the Florida Legislature. Gainesville, FL: Food and Resource Economics Department, University of Florida. 

Whitehead, John C., Tim Haab, Sherry L. Larkin, John B. Loomis, Sergio Alvarez, and Andrew Ropicki. “Estimating lost recreational use values of visitors to northwest Florida due to the Deepwater Horizon oil spill using cancelled trip data.” Marine Resource Economics 33, no. 2 (2018): 119-132.