The sampling frame is the whole problem
Every claim about consumers drawn from social data rests on one decision that usually gets no scrutiny at all: how the accounts got into the dataset. Model quality, sample size and analysis sophistication all sit downstream of it. If the accounts are the wrong accounts, a better model just produces a more confident wrong answer.
The default method is keyword or hashtag search. It is fast, it needs no infrastructure, and it is close to the worst available option.
Why search returns the wrong people
Search a city hashtag and you get the accounts posting about that city most, and most findably. Those are travel creators, lifestyle accounts and people whose content strategy is the city. Residents who live there and post about their friends do not appear, because they never tagged anything.
The problem is not that the sample is noisy. Noise averages out. The problem is that the selection is systematic and points in a known direction: toward whoever optimises hardest for reach. You have not sampled a city. You have sampled the people performing that city to an algorithm.
Worse, the bias is uncorrectable. Weighting a biased sample requires knowing the probability each unit had of being selected. With search, that probability is set by a ranking system you cannot observe, that changes without notice, and that is optimising for engagement rather than representativeness. There is no estimator for that.
This is the same failure that makes social listening read the loudest 2% rather than the population, one step earlier in the pipeline.
Chain referral: how survey science reaches people no list contains
There is an established answer to this, and it does not come from tech. Survey methodologists have long needed to study populations that no register lists and no sampling frame covers, groups that can only be found through each other. The method they developed is respondent-driven sampling, or chain referral.
The structure is simple. Start from a small set of confirmed members of the population, the seeds. Move outward through their real social connections to find the next wave. Validate each candidate. Repeat.
What makes it a sampling method rather than a snowball of whoever you happened to find is the bookkeeping. Every account that enters carries a record: which wave it arrived in, which account referred it, how close it sat to the target population. That paper trail is not administrative overhead. It is the thing that makes the sample analysable later, because it reconstructs the selection process that search hides.
How we run it
Our version starts from a small set of confirmed New Yorkers. We read the comment sections under their posts to find the next wave, on the reasoning that people who talk to each other in public are socially connected in a way that hashtags do not capture.
Every candidate is then scored on two surfaces at once: how the account writes, and what its thumbnails look like. An account is admitted if either clears the bar. Two independent surfaces matter because each fails differently. Text alone misses people who post visually and caption in three words. Images alone admit anyone whose photos look right regardless of whether they live there.
Crucially, membership is judged on how an account speaks rather than what it posts about. An account is a New Yorker because it talks like one, not because it posts New York content. That distinction is the entire difference between this and hashtag search: the former is a property of the person, the latter is a property of their content strategy.
The New York run took 876 accounts across 4,464 posts, and produced 832 distinct garments. We have run the same process end to end on Paris and London.
The bias we have not fixed
Chain referral does not eliminate bias. It swaps an uncorrectable bias for a correctable one, and we have not yet done the correcting.
Referral chains over-sample the well connected. Someone with 400 ties is far more likely to be reached than someone with 12, so the sample skews toward the socially central. This is a known, named, well studied property, and the literature comes with estimators that weight by network degree to correct for it.
We have not finished implementing those estimators. That is the honest state of this layer, and it is the item currently in flight on our roadmap.
We state it plainly because the alternative is worse in a specific way. Search gives you an error you cannot name, cannot measure and cannot fix. Chain referral gives you an error with a name, a direction, a literature and a solution you have not applied yet. Those are not the same kind of unknown, and pretending either is zero would be the actual problem.
If you are building this yourself
Four things we would tell someone starting from scratch.
Diversify your seeds harder than feels necessary. Chain referral inherits the neighbourhood of wherever it started. A handful of seeds from one social circle produces a beautifully connected map of one social circle. Seed wide and seed deliberately across the sub-populations you expect to exist.
Go deeper in waves than you think you need. Early waves look like the seeds. The sample only starts behaving like the population once the chains have moved several steps away from where you began.
Record the referral graph from day one. Degree-weighted correction is impossible to retrofit if you did not store who referred whom and in which wave. This is cheap to capture at collection time and unrecoverable afterwards.
Validate on behaviour, not on topic. The moment your inclusion test becomes "posts about the thing," you have rebuilt hashtag search with extra steps and reintroduced exactly the bias you were avoiding.
Why this matters commercially
This looks like a methodology detail. It decides whether the output is worth anything.
A brand planning a range against social data is making real inventory commitments against a picture of an audience. If that picture came from search, it is a picture of content creators in the category, and the range gets planned for people who perform the category rather than people who buy it. The failure is invisible, because the data looks abundant and the charts look the same either way.
Getting the sampling frame right is unglamorous and it is the load-bearing layer. The full four-layer method, with the measured error at every step, is published in mapping what consumers actually wear. What we build on top of it is on what Hugo does.