6 million Danes) and still report like confidence intervals-we know the true parameter estimate, there is no uncertainty.Īs an example, let’s look at the current polls for the upcoming German elections, where things look pretty interesting: while CDU/CSU (Christian Democrats) is leading strongly in front of SPD (Social Democrats), the remaining 4 parties are very close together. This is why it is a bit funny if you see papers that observe the whole population (e.g. If you have the whole population, and observe that men are taller than women, it does not matter what the height difference is for it to be “significant”. And the fact that we only study a sample we draw from the population, and not the whole population, is the main reason we use statistics in the first place. The more people we draw, the more likely our sample estimates will match the population estimates (given a number of assumptions). Sampling variability means that when we draw our sample from the population, there will be variability in the parameter estimates based on the number of people we draw. In which case, studies in small samples are problematic. And I would argue that below a certain threshold, if uncertainty is too large to be meaningful, we should not fund research in the first place. But there is sampling variability, and there is accuracy of parameter estimates. And on the long run, aggregating enough studies, these n=20 will contribute to the literature. Small n is fine if you don’t do anything with the results. n=20 might not be great, but it’s all we sometimes get. They are just as part of the population as any other subset.Īnd finally, a point that wasn’t raised in the recent discussion but that regularly comes up in discussions with clinical researchers: it can be very difficult to collect data in clinical populations. I don’t conduct such experimental studies, but I would probably also get upset if reviewers just shot down my paper “because you only have 20 participants”.įourth, the 20 people I draw from a population are part of this population - there is nothing wrong with them. We should not generally assume every n=20 study to be “bad”, and there are more important details to look at. Third, sample size does not equal power, and there are within-subject designs that have many hundred repeated measures (e.g., functional neuroimaging), offering tremendous power. Interestingly, this highlights the important role of publication bias especially when typical samples are small. Second, if there is no publication bias, and studies in a field are all based on small N, 10 studies with n=20 will have the same result as 1 study with n=200. Nothing wrong with n=20 if researchers report margins of errors. The main point is that sample size alone does not make a good or bad study, for many reasons.įirst, the problem is usually inference, but that is an issue of interpretation, not sample size. Let’s start with trying to present the statement that small samples are not inherently problematic in the best possible light, in a way that Richard and others would hopefully agree with 1. I hope this clarifies why - given the way science is currently interpreted - small samples can lead to a lot of trouble. To explain my position, let’s look at situations where small samples are fine, sampling variability, the accuracy of parameter estimates, and the way many clinical studies are interpreted. In the interesting discussion that ensued, I kept thinking about clinical psychology and clinical trials, where I believe that small samples are problematic. A few days ago, Richard Morey started a discussion on Twitter arguing that small samples are not inherently problematic.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |