Inconvenient Samples

14. June 2025 by Maria Eduarda Barbosa

What is a sample and why do we need it?

A sample is a slice of a population. Scientists measure certain variables in a sample in order to guess the measurements of a target population. In other words, scientists ask a question to a sample and, if the sample gives a significantly good answer, they assume that the answer holds for the population –until new evidence suggests the contrary. For example, imagine you have a large pool filled up with sand, each grain of sand is either black or white. Imagine you look at the pool and think “well, it looks a bit too dark.” Hence, you hypothesize that there are more black than white grains. It would be really hard to verify the color of all the approximately 20 trillion grains of sand that would fit in an olympic pool.

We will further explore sample sizes and statistical tests in a future blog post, but by now you probably understand what a sample is.

In scientific research, however, potential human participants are usually not standing still in a pool waiting to be picked and examined. Hence, we need to actively look for them.

What is the main problem with our imaginary experiment?

One assumption is necessary for our imaginary sand experiment: the grains must be well mixed. Similarly, a scientist usually strives for a sample with a good mix of all different types of subjects present in our target population. our figure then could be better construed as follows:

In real life, that nice mix is usually quite hard to achieve. In general, scientists try to select a random sample from the population, with each observation independent of every other observation. In an ideal world, this means every case (e.g.: every grain of sand) is equally likely to be included –which is rarely true.

Imagine you want to conduct a study using a survey. Your target population will be dog owners in your city and you want to know how much money on average they spend with each dog. You then find a few graduate students to collect the survey in ten different dog parks. If less than 30% of the people you and your team approach answer your survey, you have what we call the non-response bias, because people who answer the survey might have a specific characteristic that makes them more likely to do so. For example, they might have more free time and resources and, as a consequence, spend some extra money on their pets. One everyday example of this type of bias is the skew of online reviews towards the negative, since most people who are satisfied with a product do not bother to leave a review.

Back to your dog experiment, people who even take their dogs to dog parks on a regular basis might be of a higher social strata than the average dog owner –but they are easier to find. That means you are working with a convenience sample.

What are the advantages and limitations of convenience samples?

As already mentioned, convenience samples are usually used because they are accessible. However, these samples do not necessarily represent your target population. A paper titled “The weirdest people in the world?” turned undergraduate students –the most common subjects for psychological research– into the classic example for the limitations of convenience samples. In this study, the authors used the word “WEIRD” as an acronym standing for western, educated, industrialized, rich and democratic.^[1] If the target population for these studies were university students, the problem would not be so significant, especially if students from many different universities were included.

However, all these “weird” characteristics impact research questions ranging from moral values to the inheritability of IQ. And the intersection of such traits also makes up for a very small proportion of the population. As a consequence, the results from psychological research using data only from university students does not necessarily apply for the rest of humanity. This is a big problem, since the goal of research is not to discover the characteristics of a sample –these, you, as the researcher, might as well directly measure–, but to infer from the sample the characteristics of the target population.

How to work with what we have?

Does this mean we should get rid of all convenience samples? Not at all! It does mean we should pay attention to the sampling process when analysing the results of research. Namely, we should be mindful of inherent bias in the selection of cases included –because results can be generalized for the population from which cases were actually drawn, not for the population from which the researchers wish they could have drawn. If you include participants admitted to hospitals, your results generalize for participants admitted to hospitals, if you include dog owners who go to dog parks in your city, results should generalize for dog owners who go to dog parks in your city, not necessarily to all dog owners of the world. Often, convenience samples are all we got and we can try to make educated guesses from them, but if we need to extrapolate to a population that is not part of the conveniently accessible population we should be aware that we risk acting as if we knew something we actually do not know.

As a comparison, many studies are conducted on animal models, which are never a perfect recreation of human disease, some models come close, some models are quite far. If a study on an animal model yields a certain result, it is fair to consider that it might be true for humans, but it certainly does not provide conclusive evidence. However, different animal models can yield similar or different results adding more insight into the question. For example, research on covid made use of multiple animal models, including non-human primates, minks, livestock, and genetically engineered mice to enhance our understanding of the new virus.^[2]

Similarly, different convenience samples can provide similar or different results giving insights on the validity of the result and on the importance of the specific characteristics of each sample.