We all encounter research claims every day. We see sensationalized assertions on social media and in the news. The titles of the articles are presented as cut and dry facts. Some of us are faced with the quandary of conducting such research, and ensuring that our results are valid and represent the real world. Regardless of which side of the proverbial lab bench you are on, being able to analyze scientific data is relevant to all of our lives. In this article we will explore the fundamentals of research sampling.
Research sampling, in contrast to census taking, is a systematic attempt to choose a subset of a target population of interest in which to test a specific variable or monitor the effects of a specific condition. Because it is generally impossible to study a particular population in its entirety, sampling is an integral part of the experimental process in much of psychological and social sciences research, as well as other fields. It is critical for researchers to conduct this sampling correctly in order to generate valid data. For data to be deemed valid, accurate measurement is necessary. There are many considerations required to ensure that this is achieved. For our current purposes, we must remember that if a subject pool is not representative of the population, then population validity cannot be attained.
It is just as important for the reader to understand the pros and cons of each type of sampling method so that they can decide for themselves if the data is worth believing or if it is too highly flawed. Both the scientist and the public should understand the basics of these concepts so that the most thorough research can be applied in life. Let's pick an example headline to be critical of:
"Breaking News: People who drink bottled water live longer!"
There are some critical questions not addressed by this "fact." Let's argue that this sample of the population "bottled water drinkers" never drink tap water. To only drink bottled water, these people must have a disposable income that others may not. They also have constant access to a store that sells bottled water and a means to get to this location. These people have money. They probably can also afford health insurance, dental care, a healthy diet, and preventative medicine. They don't represent the population as a whole; they are a subset with high economic status and a confounding medical history. That is to say that there are other variables in this sample population that would cloud research results. This conclusion is likely false and is obviously sensationalized. Admittedly, that is an oversimplified example, but it gives us a foundation upon which to work as we consider what is necessary to understand when cultivating a sample population for scientific research.
The above example has a lot of factors we will consider throughout this article including differences in socioeconomic status, age, gender, medical history, and other potential confounds (aka exclusionary criteria). We will start with some types of research sampling methods and their definitions and then proceed to deconstruct them a bit to look at the strengths and weaknesses of each.
Sampling types
Probabilistic
There is an equal chance for any member of the population to be chosen as part of the sample.
Simple random sample:
Every member of the population has an equal chance of participation and they are randomly selected. Sometimes this is done using a sampling frame, a list of attributes held by the population of interest, this defines this population. Another version of a simple random sample are systematic samples, for example if every fifth subject is chosen from a population. Stratified sampling is another type in which samples are chosen first as groups with a certain trait, and then subjects from each grouping have an equal chance of being selected from their group. A similar approach is cluster sampling in this case the clusters themselves are the units being sampled, but each cluster maintains the same probability of being chosen.
Critical points:
In the case of simple random sampling, in the most general sense, you are ensured that you have a high likelihood, given a large enough sample size, of selecting a ratio of types of people (perhaps men v women) that is also seen in the general population. Systematic sampling is equally unbiased, but there is still a potential that the sample will not be representative of the population. However, if you take the route of stratified sampling, for example, and there are not the same number of people in each subset of the population, this will not be an accurate representation. Because of this, you need accurate population data for stratified sampling to produce a representative subject pool. For example, if there were twice as many yellow people, but only two were selected from each color group, this ratio is discrepant from the actual population you want to generalize your data to. In the case of cluster sampling, in this example, it is of note that one group is not represented at all. If these colors represented school districts, for example, data might be skewed in the interest of some students in particular, and not the population as a whole. Hence, there are issues with how representative the data generated is. This is something to be mindful of regardless of if you are generating data or reading research results.
Non-probabilistic
There are unequal chances of being chosen as part of the sample.
- Convenience (Opportunity) sampling—One of the most common versions of non-probabilistic sampling, participants are taken from a pool of convenient individuals. This is highly common in psychology research, which often capitalizes on the ready pool of undergraduate students in psychology classes. They are often incentivized with extra credit points or participation is made a course requirement. In this case, some members of the overall population have no chance of being sampled, be they computer science students, bankers, or truck drivers.
- Volunteer sampling—Similar to convenience sampling, but in this case, participants may be recruited through message boards at businesses or through media, for example. In these cases, there is generally some form of compensation for participating.
Critical points:
Although this type of sampling has its uses, such as examining correlations between certain behavioral or psychological traits, these are subjects from a "self-selected group." In the case of convenience samples, they are often undergraduates with an interest in psychology, and probability come from a certain demographic (perhaps young upper-middle class single individuals without children). Or, in the case of volunteer studies, they may be individuals that are unemployed and have time in the day to participate in the study and really need the money offered as compensation for participation. According to an article, listed below, from PsyHub, volunteer participants also tend to be younger and extroverted. These extraneous factors can skew the data and therefore alter the conclusions reached. This is why it is important to be mindful of how an experiment is conducted, not only the (often sensationalized) results.
Some other considerations include:
- Sample size—How samples are selected should be given more weight than their final size, but it is important to know that smaller samples tend to have a higher rate of sampling error. This means that the sample failed to accurately represent the population. A well-done study will note its margin of error, generally with the use of error bars above and below the line on a graph, for example. Even excellent studies have some degree of error; this is why we generally accept a 95% confidence level.
- Power—Sample size is often dictated in part by the power of the analysis. Power is the ability to reject the null hypothesis, and you need a certain number of subjects to accurately analyze the potential to achieve this. As stated within this post:
The null hypothesis is essentially the assertion that there is no difference between two populations (or more) that are being examined. In responsible research, scientists do not try to prove their idea, they try to see if they can disprove it, thus they check to see if they can reject the null or not. When the null hypothesis is rejected this means that the results of a particular test are not due to chance, with a probability generally below 0.05%.
A power analysis can be obtained online using G*Power, a commonly accepted statistical tool from the University of Dusseldorf.
Conclusion
These are things to keep in mind as a citizen or a scientist. Scientific literacy is an important part of rational discourse and decision making; not only the interpretation of results or the creation of experimental protocols. Being able to think about scientific data critically and having the foundational knowledge necessary to evaluate the claims of social media "science news" will provide the framework for the creation of new protocols, as well as the evaluation of 'scientific information' you encounter in your everyday life. These points should provide a foundation for ensuring that research samples are representative of the population, allowing for valid data generation. It's not a matter of if research was done, it is a matter of it was done well.