Research Methods

“Designing Adaptive Experiments” With V. hadad, s. athey & D. Karlan. stanford graduate school of business (2021).

Adaptive experiments present a unique opportunity to more rapidly learn which of many treatments work best, evaluate multiple hypotheses, and optimize for several objectives. For example, they can be used to pilot a large number of potential treatments when the researcher does not have strong hypotheses about what works and why; the data can then be used to narrow down a set of alternatives for further development, hypothesis testing, and evaluation. However, adaptive experiments may not be appropriate in all cases, as they create complexity and may require additional resources for implementation. This guide explains what adaptive experiments are, when they can be beneficial, and their limitations. It also offers insights into the questions to ask when considering running adaptive experiments on technology platforms.

“Survey Sampling in the Global South Using Facebook Advertisements” With P. BergQuist, K. Hoffmann Pham, F. Rampazzo & M. Mildenberger. Political science research and Methods (2025).

Survey research in the Global South has traditionally required large budgets and lengthy fieldwork. The expansion of digital connectivity presents an opportunity for researchers to engage global subject pools and study settings where in-person contact is challenging. This paper evaluates Facebook advertisements as a tool to recruit diverse survey samples in the Global South. Using Facebook’s advertising platform, we quota-sample respondents in Mexico, Kenya, and Indonesia and assess how well these samples perform on a range of survey indicators, identify sources of bias, replicate a canonical experiment, and highlight trade-offs for researchers to consider. This method can quickly and cheaply recruit respondents, but these samples tend to be more educated than corresponding national populations. Weighting ameliorates sample imbalances. This method generates comparable data to a commercial online sample for a fraction of the cost. Our analysis demonstrates the potential of Facebook advertisements to cost-effectively conduct research in diverse settings.

“Online Surveys and Digital Demography in the Developing World: Facebook Users in Kenya” with K. Pham & F. RampazZo. [Pre-Analysis plan]

As internet and cell phone use spread, a growing number of citizens in developing countries are getting online and using social media. Can social media platforms provide a cheaper, faster, and reliable mechanism to reach populations that are currently expensive to survey in comparative contexts? How representative of the national population are social media users? We examine these questions in the case of Facebook users in Kenya. First, we obtain the demographic profiles of Facebook users in Kenya using the Facebook advertising platform, which provides data on users’ location and demographic characteristics for marketing purposes, and compare them to the overall population as measured by census and WorldPop data. Second, emulating the sampling strategy used by a well-known nationally representative survey (AfroBarometer), we use Facebook advertisements to recruit online survey respondents. Respondents are asked a series of questions that appear on nationally representative surveys, and are presented with canonical behavioral survey experiments. Fielding our survey on Facebook immediately following the 2019 census and an in-person nationally representative survey allows us to ensure that any observed differences between our survey and nationally representative samples are not due to differences in timing. We examine how closely the social and political attitudes, beliefs, and behaviors of Kenyans recruited on Facebook match those of the general population. In doing so, we evaluate the potential of Facebook to serve as a survey respondent recruitment mechanism in developing countries. We provide practical information about how to conduct stratified sampling with Facebook ads, outlining best practices for paying respondents, and highlighting important limitations to consider when conducting online surveys and experiments in contexts with varying levels of literacy.