Virtual Lab Request a proposal

Population-representative samples,
recruited through ad platforms.

You set the target distribution. We move ad budget between strata, hour by hour, until the achieved sample matches it. The method is published and the code is open.

Request a proposal
841,660 Respondents
17,979,910 Survey responses
41 Countries

Since February 2020.

Trusted by researchers from:

The World Bank
Columbia University
George Washington University
Harvard University
University of Washington
Washington University in St. Louis

We have recruited in 41 countries:

Countries where Virtual Lab has fielded studiesUnited States of America — 103,475 respondentsNigeria — 88,460 respondentsJordan — 79,915 respondentsIraq — 75,209 respondentsBangladesh — 72,201 respondentsLebanon — 49,529 respondentsUnited Arab Emirates — 48,373 respondentsEgypt — 22,786 respondentsPakistan — 18,830 respondentsKenya — 17,226 respondentsHaiti — 16,545 respondentsIsrael — 16,028 respondentsIndonesia — 14,584 respondentsZambia — 11,923 respondentsGhana — 9,307 respondentsLibya — 8,460 respondentsSerbia — 7,669 respondentsBulgaria — 7,563 respondentsKyrgyzstan — 7,235 respondentsHonduras — 6,933 respondentsKuwait — 6,571 respondentsLaos — 6,202 respondentsJamaica — 6,006 respondentsRomania — 5,024 respondentsDjibouti — 4,284 respondentsChad — 4,122 respondentsSaudi Arabia — 3,930 respondentsCameroon — 3,701 respondentsBelize — 3,599 respondentsUkraine — 2,853 respondentsPapua New Guinea — 2,838 respondentsCongo — 2,519 respondentsGambia — 2,274 respondentsIndia — 1,643 respondentsMorocco — 562 respondentsIreland — 206 respondentsGermany — 23 respondents
under 100 100+ 1,000+ 10,000+ 100,000+
Respondents by regionMiddle East & North Africa — 311,363 or more respondents, 42.2% of those attributed to a countrySub-Saharan Africa — 143,816 respondents, 19.5% of those attributed to a countryAmericas — 136,558 respondents, 18.5% of those attributed to a countrySouth & Southeast Asia — 113,460 respondents, 15.4% of those attributed to a countryEurope & Central Asia — 30,573 or more respondents, 4.1% of those attributed to a countryPacific — 2,838 respondents, 0.4% of those attributed to a country311,363MIDDLE EAST & NORTH AFRICA143,816SUB-SAHARAN AFRICA136,558AMERICAS113,460SOUTH & SOUTHEAST ASIA30,573EUROPE & CENTRAL ASIA2,838PACIFIC Respondents by region311,363MIDDLE EAST & NORTH AFRICA143,816SUB-SAHARAN AFRICA136,558AMERICAS113,460SOUTH & SOUTHEAST ASIA30,573EUROPE & CENTRAL ASIA2,838PACIFIC

Ad cost per respondent:

Advertising cost per respondent newly recruited Advertising cost per respondent newly recruited, across 44 studies. The median study pays $1.05. The middle half fall between $0.29 and $1.57. The tenth percentile is $0.09 and the ninetieth $3.67, read against a scale from $0 to $4. The full range runs $0.04 to $8.89. Advertising spend only. This is a distribution across studies, not an uncertainty interval on an estimate. Advertising cost per respondent $0.29 $1.05 $1.57 $0 $2 $4 PER RESPONDENT Advertising spend only — not the incentive, not the survey platform, not our fee. Distribution across 44 studies. Box: 25th to 75th percentile. Whiskers: 10th ($0.09) to 90th ($3.67); the range runs $0.04 to $8.89. Per respondent newly recruited, $166,243 over 163,555 respondents. Advertising cost per respondent newly recruited Advertising cost per respondent newly recruited, across 44 studies. The median study pays $1.05. The middle half fall between $0.29 and $1.57. The tenth percentile is $0.09 and the ninetieth $3.67, read against a scale from $0 to $4. The full range runs $0.04 to $8.89. Advertising spend only. This is a distribution across studies, not an uncertainty interval on an estimate. Advertising cost per respondent $0.29 $1.05 $1.57 $0 $2 $4 PER RESPONDENT Advertising spend only — not the incentive, not the survey platform, not our fee. Distribution across 44 studies. Box: 25th to 75th percentile. Whiskers: 10th ($0.09) to 90th ($3.67); the range runs $0.04 to $8.89. Per respondent newly recruited, $166,243 over 163,555 respondents.

Budget reallocations per study:

Budget reallocations per study Budget reallocations per study, across 109 studies. The median study takes 61. The middle half of studies fall between 28 and 165. The tenth percentile is 13 and the ninetieth is 351, read against a scale running from 0 to 400 reallocations; the longest study ran to 1,308. This is a distribution across studies, not an uncertainty interval on an estimate. Budget reallocations per study 28 61 165 0 200 400 REALLOCATIONS Distribution across 109 studies, each counted once. Box: 25th to 75th percentile. Whiskers: 10th (13) to 90th (351); the longest ran to 1,308. One reallocation is one pass over every stratum, resetting each budget from what the stratum currently costs and how far it is from its target share. Budget reallocations per study Budget reallocations per study, across 109 studies. The median study takes 61. The middle half of studies fall between 28 and 165. The tenth percentile is 13 and the ninetieth is 351, read against a scale running from 0 to 400 reallocations; the longest study ran to 1,308. This is a distribution across studies, not an uncertainty interval on an estimate. Budget reallocations per study 28 61 165 0 200 400 REALLOCATIONS Distribution across 109 studies, each counted once. Box: 25th to 75th percentile. Whiskers: 10th (13) to 90th (351); the longest ran to 1,308. One reallocation is one pass over every stratum, resetting each budget from what the stratum currently costs and how far it is from its target share.

Respondents recruited per day:

Respondents recruited per study on an active day Respondents recruited per study on a day of active recruitment, across 129 studies. The median study recruits 140 on such a day. The middle half of studies fall between 69 and 300. The tenth percentile is 41 and the ninetieth is 531, read against a scale running from 0 to 550 respondents. This is a distribution across studies, not an uncertainty interval on an estimate. Respondents recruited per active day 69 140 300 0 250 500 RESPONDENTS Distribution across 129 studies, each at its own median active day. Box: 25th to 75th percentile. Whiskers: 10th (41) to 90th (531). An active day is a study-day recruiting at least 20 respondents; half of studies have 11 or more. Respondents recruited per study on an active day Respondents recruited per study on a day of active recruitment, across 129 studies. The median study recruits 140 on such a day. The middle half of studies fall between 69 and 300. The tenth percentile is 41 and the ninetieth is 531, read against a scale running from 0 to 550 respondents. This is a distribution across studies, not an uncertainty interval on an estimate. Respondents recruited per active day 69 140 300 0 250 500 RESPONDENTS Distribution across 129 studies, each at its own median active day. Box: 25th to 75th percentile. Whiskers: 10th (41) to 90th (531). An active day is a study-day recruiting at least 20 respondents; half of studies have 11 or more.

What it takes to recruit respondents on social media

  1. 01Strata One ad set per stratum, each targeting a different slice of the population, based on the appropriate stratification variables.
  2. 02Allocation Spend allocated among strata, so every stratum fills toward the share you asked for rather than the share that happens to be cheap.
  3. 03Prices The allocation itself revisited as prices move — prices for each stratum need to be estimated individually, live, to know how to reallocate spend to buy the most precision in your final estimator.
  4. 04Uniqueness Verifying the identity of each respondent— because otherwise everyone on the internet will answer your survey many times over.
  5. 05Incentives Paying an incentive to each respondent, in their own country and in a form they can actually use.
  6. 06Follow-up The same people found again months later for an endline survey, where appropriate.

For ad allocation, we solved an optimization problem

  1. 01Strata
  2. 02Allocation
  3. 03Prices

You are choosing how many respondents to recruit in each stratum. You want the smallest variance on your weighted estimate, and you are bounded by your budget.

argmin n1…nH ∑h Wh2σh2 nh subject to ∑h phnh ≤B

Where Wh is the weight of stratum h, σh its outcome dispersion, ph the price of one more respondent there, and B the budget.

The price per stratum is not known in advance. It has to be learned while the campaign is running, from the campaign itself — and every new estimate changes the allocation, which changes what you learn next. That loop is why this is software and not a spreadsheet.

The method is published, and the validation with it.

Adaptive Survey Sampling via Ad Platforms

Dante Donati · Columbia Business School and CESifo
Nandan Rao · Virtual Lab and Universitat Autònoma de Barcelona

Read the paper on SSRN

For surveying, we built a chatbot

  1. 04Uniqueness
  2. 05Incentives
  3. 06Follow-up

We built a chatbot survey platform to facilitate online recruitment, surveying, and interventions. The questionnaire runs as a conversation inside the messaging app the respondent already uses — Messenger or WhatsApp — one question at a time, in their own language.

It asks, the respondent answers, and the thread stays open. Months later the same conversation reopens for an endline, with no one re-enrolling.

A questionnaire running as a conversation A survey asks a question inside a chat thread. The choices arrive as a numbered list and the respondent replies with a number. A dashed rule marks a pause of months, after which the same thread reopens and asks again. How often do you use a mobile phone? 1 Every day 2 A few times a week 3 Rarely or never 2 MONTHS LATER Welcome back. A few more questions. Has that changed since we last spoke? Yes A questionnaire running as a conversation A survey asks a question inside a chat thread. The choices arrive as a numbered list and the respondent replies with a number. A dashed rule marks a pause of months, after which the same thread reopens and asks again. How often do you use a mobile phone? 1 Every day 2 A few times a week 3 Rarely or never 2 MONTHS LATER Welcome back. A few more questions. Has that changed since we last spoke? Yes
Answers come from an account, not a link
A respondent answers inside the Messenger or WhatsApp account they already use, and a form cannot be entered twice from the same account.
Incentives inside the thread
Mobile airtime, a gift card or a data bundle, paid in the conversation.
Pause and resume
A baseline and an endline in one thread, months apart, without re-enrolling.
Actions recorded, not just answers
A link is sent as a button and the click is recorded. A video plays inside the thread, and play, pause, seek and completion are recorded as events.
Conditional logic
A question can branch on any earlier answer or hidden value, nested as deep as the design needs.
Answer validation
Invalid or out-of-range answers are rejected and the question asked again, in wording set per survey.
Randomized arms
A participant’s arm is a hash of the form and the participant, so assignment reproduces from the exported data.
Photos from respondents
A question can ask for a photo, and the respondent sends one from the phone they are already answering on.

The code is open source

The whole platform is public on GitHub — the optimizer that moves the budget between strata, and the instrument that carries the questionnaire. It self-hosts on Kubernetes with Helm, and you are welcome to run it yourself.

Most people would rather we ran it. Studies we operate run on our own infrastructure in the European Union, encrypted in transit and at rest.

github.com/vlab-research Read the documentation

Tell us what you need to measure

Tell us what you want to measure, as a survey or an impact evaluation, and we will help you come up with a design, a timeline, and a price.

info@vlab.digital