This article is going to dive into the most effective ways to quantify the precision of a rifle and ammo particularly when it comes to group sizes and dispersion (i.e. how the bullet holes on target are distributed or spread out). We’re going to build on Part 1 that laid the foundation, and Part 2 that looked at quantifying muzzle velocity variance – and in this final article of my Statistics for Shooters series we’re going to apply those concepts in two dimensions to analyze groups.
In Modern Advancements for Long-Range Shooting Vol. 2, researcher Bryan Litz provides a great introduction to this with a central question that all of us shooters are wondering:
“One common question shooters ask about measuring groups is: should I shoot 3-shot groups or 5? Is it necessary to shoot 10 shot groups? There’s no easy answer to this question, but we can say that the more shots you fire, the more decisive and accurate your understanding of precision will be. If you’re just trying to see what your rifle’s 3-shot average group size is, then you just shoot a few 3-shot groups and you know. But if you’re trying to decide if there’s a precision difference between ammo loaded with primer A vs. primer B, then a couple of 3-shot groups probably won’t answer that question. So it very much depends on what question you’re trying to answer when you consider how many shots are necessary.”
“So it very much depends” is right! This article will dive down that rabbit hole, and equip you to answer the following questions for your specific application:
- How many shots per group do I need?
- Is extreme spread the best way to measure my groups?
- Should I exclude fliers?
- What is the most effective and accurate way to compare groups between two loads? (That doesn’t require a math degree!)
- Are there any practical tips or shortcuts that can help with this?
My goal is to help you get the most out of the shots you fire at the range doing load development or even comparing factory ammo, and make more informed decisions that will help us get more rounds on target. As with all the other articles in my Statistics for Shooters series, I spent an absurd amount of time arduously crafting this article and creating visuals so it was approachable by shooters who aren’t math nerds because I firmly believe these concepts can help a TON of people in the long-range community.
Precision vs. Accuracy
Before we dive into the details, we need to clarify two central terms related to this topic. Many shooters use the terms “precision” and “accuracy” interchangeably, but there is an important difference – in both shooting and statistics.
“Note that here we use the specific statistical meaning of these terms, even though in casual usage they are often sloppily interchanged. Accuracy refers to how well shot groups are centered on a target, and is essentially a problem of sighting-in a shooting system. If the shot group doesn’t center on the desired target the assumption is that the weapon merely needs to have its point of aim adjusted. It is implicitly assumed that the aim adjustments can be made finely enough so that any difference between the average POA and POI would be statistically insignificant. Precision describes the spread of individual shots about the center point of a shot group. A precise shooting system will, over many shots to and from the same points, produce a tight shot group with smaller distances between individual shots than a less precise system. However, in contrast to accuracy, precision can’t be “adjusted” into a shooting system by dialing some knob on the weapon.” – Ballistipedia.com
Both accuracy and precision are important to hitting targets, but when we’re shooting groups we’re testing precision. Often that’s also referred to as dispersion, which can be thought of as how the bullet holes on target are distributed or spread out. That is what this article will be focused on.
The Problem with Extreme Spread and Other “Range Statistics”
The most common way to try to quantify the precision or dispersion of a rifle/ammo system is to measure the extreme spread (ES). ES is typically expressed as the center-to-center distance between the two shots that are furthest apart. That is easy to measure by hand and it is a useful stat. However, like any descriptive statistic, we are simplifying multiple data points into a single number – so by definition, we are losing some level of detail, but it does make it much easier to do direct comparisons.
“Measuring the extreme spread of shot groups is quick and easy, but it’s not actually a very good measure of dispersion. What do I mean by a good measure? A good measure should give you useful information, which is information you can use to make good decisions. When you look at the extreme spread of a 5-shot group, that measurement is determined by only 2 out of the 5 shots. In other words, only 40% of the shots are considered in the measurement. Even worse, for a 10-shot group, a center-to-center measurement is only using information from 20% of the total shots. Since the extreme spread, center-to-center measurement, is determined by only a small portion of the total shots available, it’s just sort of an indicator of precision.” – Bryan Litz
Bryan certainly isn’t alone on this view. Ballistipedia groups ES into what it refers to as “Range Statistics,” which also includes measurements like the smallest enclosing circle and others. They say these types of measurements “are more commonly used because they are easier to calculate. But they are statistically far weaker because they virtually ignore inner data points.” They point out that ES and other range statistics will always increase in size as we increase the number of shots. “They are the least efficient statistics, but are also the most commonly used because they are so easy to measure in the field and so familiar to shooters.”
That isn’t to say that ES is completely irrelevant. A stat that is easy to measure at the range, and that everyone is familiar with can be helpful! However, other statistics can help us make better decisions on how a rifle/ammo system will perform in the future, and maximize the benefit from every shot fired.
A Better Measurement: Mean Radius (a.k.a. Average Distance To Center)
“There are many alternative measures of precision,” explains Litz. “You could measure the location of each shot and calculate the vertical and horizontal standard deviation (SD), radial standard deviation (RSD), circular error probable (CEP), etc. You could get really carried away with statistical methods of characterizing precision.” We could really nerd out and talk about Rayleigh distributions and computing the corrected sum of squares – but Bryan is right, it’s easy to get carried away! (Read about these other measurements.)
There is a much simpler approach that allows us to get most of the practical benefits of those more sophisticated methods. Here is the stat Bryan Litz recommends that can provide that goldilocks balance of statistical rigor and insight, without getting carried away (excerpt from Modern Advancements In Long Range Shooting, Vol. 2):
- Mean radius uses information from every shot in a group, not just the two most distant points. Because of this, mean radius can provide a higher confidence measure of precision than extreme spread.
- The more complicated measures involving standard deviation assume a normal, or Gaussian distribution; as in the bell curve. It’s arguable weather or not shot groups follow a Gaussian distribution or not [REF 3], and we don’t need to stir up arguments with statisticians.
- There is a free software program called On Target [REF 8] which automatically calculates mean radius (On Target calls it average to center or ATC). So you don’t have to do any of the tedious math.
- When compared with the more sophisticated methods of measuring precision, mean radius is effectively just as good at resolving relative precision.
There are some criticisms of the mean radius measure; in particular:
- Extreme spread can be measured simply with a ruler or calipers; no computer required. Depending on the projects, people, and equipment involved, it may be better overall to use the simpler measure (extreme spread) to avoid confusion.
- When groups of shots are so tight (bug-holes), you can’t always locate each shot, so a mean radius measurement wouldn’t be possible.
Although mean radius is a little harder to measure than extreme spread, it provides us with a better measurement of actual precision. I’m not claiming that mean radius is technically superior in general, only that it’s a good tool for the job, considering practical application.
Just a few more comments on statistics and then we’ll get into looking at dispersion for various shooting applications.
Circular Error Probable (CEP) is a common metric used to quantify dispersion. CEP is defined as a circle in which 50% of shots will land inside. CEP and mean radius are very similar, within 5% or 6%, but are technically not the same thing.
If you want to know how big your group is likely to be based on the mean radius, you can multiply by 2.1 to get the 95% radius (R95). For example, suppose you shoot a 10-shot group at 1000 yards with a mean radius of 3 inches. 2.1 times 3 inches gives you the radius of the circle in which 95% of your shots can be expected to land, so that’s 6.3 inches. Remember that’s a radius, so the total group size would be around 12.6 inches.
We discussed how your center-to-center group measurements can be expected to grow with more shots. Mean radius will also grow with increasing number of shots, but not as much as center-to-center. In other words, you can expect a 24% difference in group size when going from a 5-shot to a 10-shot group measuring center-to-center. But the mean radius will be closer to the same value for 5, 10, or 20-shots. Basically, this means you converge on the final answer in fewer shots when measuring mean radius. Another way to look at it is, using mean radius, you can resolve smaller differences in precision with fewer shots. So if you’re testing to see if ammo loaded with primer A is more or less precise than ammo loaded with primer B, mean radius or average-to-center is more likely to discover the true answer.
Great explanation, Bryan! But, let’s not just take his word for it. What do other industry experts say?
“The US Military uses the measurement of the mean radius of all the shots in each group as the definition of the angular accuracy of a given firearm, not this ‘extreme shot distance’ measurement,” explains Chris Long, a respected researcher, and writer on precision rifles. Ballistipedia corroborates that saying, “Armed forces also often explicitly uses the more statistically powerful Mean Radius and Circular Error Probable measures.”
Mean radius, also referred to as Average Distance To Center (ATC), is easy enough to understand. By that, I mean that it isn’t some abstract value that is hard to see how it relates to the physical group printed on the target. And it gives us more reliable insight that can lead to better decisions.
While many recreational shooters may not be familiar with mean radius, that isn’t because it is new. It has been used in serious research for years. Here is an excerpt from an old issue of American Rifleman on the subject:
“Mean radius is the mean [or average] distance of bullet impacts from the center of the test group. It is used in government ammunition acceptance because it takes account of every shot and comes close to maximizing the test information. While there is no exact relationship between this measure and the simpler and more convenient group diameter, the 10-shot group diameter averages slightly over 3 times the mean radius.” … “These examples illustrate the sensitiveness of the extreme spread to the number of shots in the group. Indeed, as the table indicates, the measures made to only the outside shots of the group, e.g. the extreme spread, are very sensitive to the number of shots, while the measures made to all the shots, e.g. the mean radius, are far less so. It may be added that the latter measures are also less variable in their representation of the group; they are more efficient. This explains why the target testing of U.S. military rifle ammunition is by mean radius.”
How To Measure/Calculate Mean Radius
Okay, let’s say we just fired a group – how do we calculate the mean radius? While it’s possible to measure and calculate by hand, I wouldn’t suggest it. Virtually nobody does it that way. The easiest way is to snap a photo of the target and use software to analyze it. The software will generate X and Y coordinates for the center of each hole, and then do the measurements and math for us.
Desktop apps like OnTarget will do this (costs $12), and some phone apps allow you to snap a photo of a target and analyze the group, like TargetScan ($18). A few months ago, I collaborated (on a volunteer basis) with the maker of a phone app called Ballistic X ($8) to add more advanced statistics to its analysis, including Mean Radius and some of the other advanced statistics used to quantify dispersion. The photo below is from a prototype of the changes I helped test for them, but these new features haven’t been released in Ballistic X at the time this was written, but I’m told they will be very soon (Update 12/16/2020: The owner of Ballistic X told me, “We are hoping to have a Beta for testing before Christmas. Then a full released after!”) . (Ballistic X for Apple | Ballistic X for Android)
Let me say this as clearly as I can: If you are comparing two loads or two rifles and trying to decide which is superior in terms of precision, comparing the mean radius of the groups fired will lead to more reliable conclusions than comparing extreme spread. If you got lost on the rest of the “math talk,” that should be your key takeaway. You have to use an app to calculate it, but doing that will help you get more benefit from those shots you fired at the range.
Should We Exclude Flyers?
There is no lack of strong opinions on this topic, but I will try to take a practical, yet science-based, approach to thinking through this question. This turns out to be critical when trying to quantify, model, and predict dispersion.
In general, excluding an observation just because it is an outlier is not good science, so most serious researchers don’t believe it’s justifiable to filter out data points. (Good article on this topic.)
“To exclude a shot from a group just because it appears to be a ‘flyer’ is bad measurement technique and would lead one to underestimate long-range dispersion.” – A User’s Guide to Dispersion Analysis by Bruce Winker
“In target shooting a ‘flier’ is a shot that flies wide of the target, or appears to be an outlier. Every shooter experiences fliers. Sometimes the cause of a flier is known or can be found. But in statistics we should be careful to discriminate between ‘outliers’ and ‘fliers,’” Ballistipedia explains. “Not every outlier is a flier.”
The answer to our question often comes down to the difference between an “outlier” and a “called flyer.” (Note: Both flyer and flier are accepted spellings. 😉)
The dispersion of a group is primarily driven by two factors:
- Mechanical precision of the weapon system (i.e. barrel, optics, ammunition, and other parts, combined with how all those things attach and interact under recoil)
- Aiming error of the shooter (i.e. the trigger was pulled when the sights weren’t perfectly centered on target or perhaps your body position or rest wasn’t consistent shot-to-shot causing the weapon to move differently under recoil in a way that skews the bullet’s trajectory)
While our goal is to quantify #1 in complete isolation – with shoulder-fired rifles aimed by a human, #2 is unavoidable to some degree. Often we simply try to minimize it as much as possible. However, if as the shooter breaks a shot they notice their sights were not perfectly centered on the target and they “call” that (i.e. admits it, often verbally) before they looked to see where that bullet fell in their group – that is a called flyer, and there is a valid reason to exclude that shot from the group.
Here’s the catch: If you look at the target and notice that the last shot didn’t fall within the rest of the group, you can’t then say “You know, I must have pulled that one. I’ll exclude it.” Often this comes down to ego vs. good science. Be honest with yourself! If you want to make the best decisions then you’d only exclude called flyers from your groups – and leave any other outliers in the results.
Have you ever thought about why we are so quick to want to exclude an outlier in our groups? Molon suggests that is also related to groups with just a few shots in them: “When we’re firing three or five-shot groups with a flier, it is only natural to assume that it was caused by a flinch or ‘pulling’ the shot. Therefore, since the flier was our own fault, the tendency is to eliminate it from any reporting of group size.” But, if instead of stopping after 3 or 5-shots we continued to fire bullets until we had a 10-shot group, it’s likely the “obvious outlier” might become an obvious part of the group that eventually formed. MacDonald presents this helpful example:
Let’s focus on Group 2 above. If we had just fired Group 2 on a target, it would be natural to assume the highlighted shot that fell out of the group must have been our fault. However, when we look at the overlay of all the groups, we get a clearer picture of the rifle’s true dispersion and we see that shots from other groups also landed in a similar location on the right. While it would be completely natural to blame ourselves for a outlying shot like that and exclude it as a “flier” – it would also be wrong and cause us to underestimate the rifle’s true dispersion.
With practice and practical steps like double-checking your scope’s parallax, using a firm rear bag and solid front support, ensuring your body position and trigger pull is consistent shot-to-shot, etc., most shooters can get to the point where their influence on the rifle can be minimized so that it isn’t one of the largest variables when it comes to dispersion. There will always be some impact from the shooter for any shoulder-fired weapon where a human is pulling the trigger, but if you’re reading this – I’d suspect you are likely one of the highly detailed, educated shooters who should be slower to blame yourself for fliers.
Finally, Ballestipedia.com explains the more technical reason that we shouldn’t exclude outliers: “We have accepted unbounded distributions as models of the shooting process, and so we have to also accept that outliers are part of both the model and the real world and that our model can correctly account for them if they are part of the modeled process. Granted, if I had a rail gun on an indoor range and had triple-checked every component of every round I sent downrange I may not accept an unbounded normal distribution as a model of my shot dispersion. But once we allow for outdoor conditions and normal ammunition, not to mention a shooter operating the gun, then in the normal course of events we will get outliers, and they are representative of the underlying normally-distributed process.” (Note: If you aren’t sure what “normally-distributed” means, read Part 1.)
Said simply, if you exclude anything but shots that were truly “called fliers” resulting from clear shooter error – you will underestimate dispersion. It might make you feel better about your group size, but it’s not going to help you hit targets at long range.
What Is A Good Sample Size?
What is a good sample size? The short answer comes down to how minor of an improvement you are trying to detect and how much confidence you want to have that the results will be a good predictor of future performance – but I’ll try to give you a more direct answer than that, without getting too technical. 😉
Recently I listened to a few of the sessions from the 2020 Modern Day Rifleman Summit, including two lectures by Hornady’s Ballistician, Jayden Quinlan. I enjoyed both of them, but here is an especially relevant excerpt I pulled from the 2nd lecture:
Jayden Quinlan, ballistician at Hornady, says that the very first thing he does in his process of setting up a new rifle is “getting a really good handle on what the system is capable of. I’m glad we talked about this part because I believe it often gets overlooked. One part of that is cost because there is a cost associated every time we pull the trigger, both in barrel life and cost of components. The second part is ego: We don’t want to know how bad our system really is. I have a 1/2 MOA rifle every time I do my part – that cracks me up every time I hear it. So here is my typical response to someone who says “I have a 1/4 MOA or 1/2 MOA rifle.” My first question is, “Over how many shots? 2 or 5 or 10 or 30?” Because that’s important to know. If we’re talking about some kind of dispersion we need to know the sample to quantify if it is good or not. Because a 1/4 MOA group that is 2 shots doesn’t really tell me a whole lot, but a 1/4 MOA group that is 10 shots tells me a whole lot. So how many shots? That’s the first question, and generally, it is a super small sample size and it was done once, and the next group was 1 inch.
The argument is if you truly have a 1/2 MOA rifle at 1000 yards, and it’s a shoulder-fired, hunting-type, lightweight field rifle and it shoots 1/2 MOA at 1000. If it does, then take it to the heavy Benchrest Nationals because you’re probably going to win. You know?! Look at the average group size of those rifles. I’m sorry, I don’t believe you.
Putting ego aside, what is it really capable of? So I shoot no less than 10 shot groups when I’m testing my dispersion. Dispersion is pretty much linear, and it goes in non-linear when you add in velocity and drag and stuff like that. So aiming error and dispersion are kind of linear. So I shoot them at 100 yards, and no less than 10 shots. And I don’t have a big giant ego fit if it is 1 MOA, and I don’t have a big giant ego if it is 3/4 MOA.
We’ve done enough large sample size testing in 50 shot groups to understand that you’re chasing your tail sometimes when you’re trying to squeeze that last little bit out of it, and when you think you have it your sample size isn’t large enough for that to actually be valid. Because when I go to a match, how many shots am I shooting? 200 rounds? 250? So how representative is my sampling of 10 to my capability for 200 or 250? Not as good as you think.
So I shoot no less than 10 for my baseline. Let’s say it is 1 MOA. Okay, then 1 MOA is my baseline. Then I move on to testing my muzzle velocity …”
Well said, Jayden! Lots of wisdom there.
I’ve yet to see any respected expert who is familiar with statistics claim that a 3-shot group is adequate to quantify a rifle’s precision. In fact, after rigorous statistical analysis, Ballistipedia.com says, “If you’re shooting 3-shot groups … then you are wasting bullets.” While many keyboard snipers love to cite how tiny their 3-shot groups are, they often fail to mention that their groups impacted at different spots on the target. The exact center of each bughole shifts around a bit relative to the point of aim. Again, let’s refer to MacDonald’s illustration below and we can see that the 5-shot groups on the left are each smaller than the composite group in the overlay, but we can also notice that the center of each of the 5-shot groups’ shifts around slightly.
This is a big part of why 10-shot groups are a much better indicator of the true dispersion of a rifle/ammunition combination – and a rifle’s true zero. (Great article with more on this: Statistics, Shooting, and the Myth of the 3-Shot Group)
Let’s step away from stats and science for a minute, and simply reflect on our own experience: We’ve all been at the range and had a couple of shots go through the same hole. Why are we so hesitant to send another bullet at that same target? If we’re honest, we’ve all felt that! It’s because our intuition tells us that one more shot could mess up that sweet bughole we have going – and our intuition is right! It’s exponentially easier to get a tiny group with just a couple of shots – but that largely has to do with luck. Fire enough 3-shot groups with virtually any rifle and you’ll eventually get a lucky one that measures under 0.25 MOA – and that looks cool on Instagram or a forum. But, the more shots we fire the more accurate read we get on a weapon system’s true dispersion.
“Some shooters may have two or three 3-shot groups to prove the load is really accurate. It really takes more shooting than that to make a judgment on a load’s accuracy potential. Three shots forming a tight cluster is nice to look at, but it is little more than an accident. Shooting three-shot groups to see how everything is working is essentially a waste of time and components. … Ten shots are a more reliable indicator when it comes to predicting what a load is likely to do in the future.” – Rick Jamison in an article published in Precision Reloading
When it comes to sampling size, most experts recommend 5, 7, or 10 shot groups for personal/hobby use. Those tasked with the most rigorous levels of research likely want much larger sample sizes (potentially 30, 50, or more). For example, the US Army Marksmanship Unit at Fort Benning, Georgia uses a minimum of 3 consecutive 10-shot groups fired with the rifle in a machine rest when testing service rifles.
At the 1992 Barcelona Olympics, USA Shooting Team members Launi Meili and Robert Foth won the gold and silver medals in the three-position rifle events. The Olympians used a type of Federal Gold Medal ammunition that was new at the time to aid them in their victories. That was the first time in over 30 years that an American won an Olympic medal in one of the small-bore shooting events while using American-made ammunition. When Federal’s Director of Product Engineering, Dave Longren, was asked about the accuracy/precision development and multifaceted testing of the Federal ammunition that helped the US Olympians win gold and silver medals, he had this to say: “The standard test string was three 10-shot groups, with the most attention paid to the 30-shot composite. When you’re working at this level, the traditional five 5-shot group test simply doesn’t give you statistically valid results.”
In general, statisticians consider any study with less than 30 data points to be a “small sample size.” So, it’s interesting to note the US Army Marksmanship Unit and Olympic-level competitors rely on three consecutive 10-shot groups – “with the most attention paid to the 30-shot composite.”
There was a good article posted by Molon on AR-15.com that captured an important point:
“Institutions and organizations that buy enormous amounts of ammunition and weapons are far more interested in facts than sales hype and propaganda and most of them demand that accuracy/precision testing be conducted using 10-shot groups. This includes the Federal Bureau of Investigation, Crane Naval Surface Warfare Center, the US Army Marksmanship Unit, and the US military’s acceptance testing of both 5.56mm ammunition and weapons. On the other hand, businesses that use 3-shot groups for making their accuracy claims are usually trying to sell something.”
When I first read that, I immediately recognized that statement had the ring of truth to it, but I also thought another scenario where it’s popular to cite 3-shot group sizes is when someone is trying to impress other people online. 😉 Molon went on to hit that head-on: “Ten shots are a more reliable indicator when it comes to predicting what a load is likely to do in the future. The problem with 10-shot groups is that when you report them, everyone thinks you aren’t shooting very well or that the ammunition is not good because the group sizes are so much larger than three or five-shot groups.” A large part of why so many continue with 3-shot groups likely gets back to Jayden’s comment about ego and the fact that “we don’t want to know how bad our system really is.”
The NRA’s 5×5 Method
The NRA’s standard for testing precision is to shoot five consecutive 5-shot groups and report the average extreme spread, which some serious statisticians say “is actually rather efficient.” As I mentioned in the first article in this series, when we’re firing groups to determine the capability of the system, often we’re doing that because we want to be able to predict how it will perform in the future. When it comes to predicting the future, we can’t do that with absolute certainty, so instead, it’s more appropriate to talk about ranges and probabilities with confidence intervals. So let’s do that in this case, and it’ll help us understand how reliable of a predictor the average extreme spread of 5 groups with 5 shots each really is. Let’s say that we measured the average ES of five consecutive 5-shot groups to be exactly 1.0 MOA. Ballistipedia.com tells us, the 90% confidence interval would be 0.81 MOA – 1.20 MOA. “This means, for example: if we measure an average 5×5 extreme spread of 1.0 MOA then, nine times out of ten, we would expect that same shooter, gun, and ammunition to produce five 5-shot groups measuring between 0.8 MOA and 1.2 MOA.”
However, while the NRA’s 25-shot, 5×5 method is good, “Following the best estimation methodology you can measure precision as effectively as the NRA protocol does but use only 3/4 as many bullets. Following the NRA protocol you’re spending 32% more ammo than necessary to get the same precision estimates.” (Ballistipedia). Granted, it might be easier to measure the average ES of 5 groups than to perform all the steps necessary for “the best estimation methodology,” but mean radius doesn’t involve a lot more complexity and can provide more reliable insight from less than 25 shots.
The Hard (But Important) Truth
Here are a few conclusions that I realize may challenge “conventional wisdom,” but they’re incredibly important:
“Targets are a bit more complex than muzzle velocities because a group is two-dimensional, not one-dimensional,” Denton Bramwell explains. “However, similar to the muzzle velocity case, group size is variation, and is a slippery devil to estimate using small samples. If you are using five-shot groups to evaluate group size, you should expect that groups will naturally vary plus or minus about 50% with no change whatever in the load, rifle, or shooter’s performance. … If the rifle is truly a 1” machine, then the shooter should expect that 95% of his five-shot groups will fall between ½” and 1 ½”, with absolutely no change in real performance. You cannot reliably estimate the long-term characteristic of the rifle with just one or two five-shot groups. Groups within plus or minus 50% of the true long term average do not indicate any real change.”
Remember what Jayden Quinlan, the Hornady ballistician, said: “We’ve done enough large sample size testing in 50 shot groups to understand that you’re chasing your tail sometimes when you’re trying to squeeze that last little bit out of it, and when you think you have it your sample size isn’t large enough for that to actually be valid.”
Engleman agrees, and takes the conclusion a step further: “I believe that you will find that by making statistically sound judgments, many loads produce statistically similar results and loads, in general, are not as finicky as current conventional wisdom would lead us to believe. … In reality, your shooting performance will probably improve since you can spend more time practicing with your set load and less time playing around with load combinations that have little or no statistical significance in terms of shooting performance.”
So What Now?
Obviously, we can’t be expected to shoot three consecutive 10-shot groups of each combination of powder load, bullet jump, primers, and other variables we’re trying to dial in for our rifle. I like the way Adam MacDonald said it in his article Thinking Statistically: “The rifle talks to us by generating samples at $1 a pop. If we want to know how it truly works, we need to play its game. With enough samples, we can try to measure the population, but it can be expensive.”
So what is a guy to do? What is the right balance between statistical accuracy and not burning out our barrel before we work up a good load? Here are a few tips and tricks that can help us find our way to a good load in fewer shots:
Tip #1: Start With What The Winners Are Using
Engleman suggests a good shortcut: “The more data you include in a calculation, the more statistically accurate it will be. So with respect to component choices, start with the same components the winners are shooting. There is far more data in all the rounds they fire collectively with good results than you can possibly collect on your chronograph. If it works for them, it will work for you.”
I realize that may sound like a lazy answer, but it is a more valid solution than the small sample size load development most shooters do. If nothing else, it is a logical and effective shortcut to help us narrow the search for a load that performs well in your rifle – with fewer shots.
Since 2012, I’ve been publishing extensive information on the components that the top-ranked long-range shooters in the country are using, and you can access those articles here: What The Pros Use, and look for articles about load data, bullets, powders, primers, brass, etc. for a variety of popular precision rifle calibers and cartridges.
Tip #2: Eliminate Bad Loads Early – Even Based On Small Sample Sizes
Here is another shortcut: Identify what doesn’t work quickly. “If you fire 5 shots and the group is big, you can pretty confidently rule out this load,” explains MacDonald. “The ES, SD, and your intuition would all agree – better steer clear of this one. Why? It’s only 5 shots! The reasoning is that the relation between a sample and its population is asymmetric. It’s more likely that a bad load will produce a small group than it is for a good load to produce a large group. We exploit this and it’s why iteration works.”
MacDonald goes on to explain one of the most important keys: “When you think you’ve found something good, fire a couple more of the same load and see if your fortune repeats itself. Then proceed with verification. Once you happen across a load that appears promising, don’t stop there.”
Tip #3: Include More Data For Free By Using Mean Radius
In statistics, the more data points you have, the more confident you can be in your results. So let’s think back on a quote from Bryan Litz mentioned at the top of this article: “When you look at the extreme spread of a 5-shot group, that measurement is determined by only 2 out of the 5 shots. In other words, only 40% of the shots are considered in the measurement. Even worse, for a 10-shot group, a center-to-center measurement is only using information from 20% of the total shots. Since the extreme spread, center-to-center measurement, is determined by only a small portion of the total shots available, it’s just sort of an indicator of precision.”
Extreme spread ignores a large portion of the shots we fired. We pay $1 or more for every round fired, so let’s make sure we get the full benefit from every one of them! Mean radius is the average distance to the center of the group for every bullet hole. That means if you fired 10 shots, the result is based on 10 measurements (i.e. the distance of each shot to the center of the group) – instead of just the measurement between the furthest two shots for ES. That is a big deal because it helps us more confidently characterize and quantify the precision of the weapon system.
Tip #4: Avoid Bugholes When Using Mean Radius
So we want groups with larger sample sizes, but that presents a problem when it comes to mean radius. Litz explains, “When groups of shots are so tight (bug-holes), you can’t always locate each shot, so a mean radius measurement wouldn’t be possible.” The 10-shot group below illustrates the problem. We can’t identify where the center of all 10 shots fell, so we actually can’t measure the distance from each shot to the center of the group. In the scenario below, we can only measure the ES of that paper target, unless we were using some type of electronic target system.
In this article, I referenced a lot of experts and serious researchers who recommended that 10-shot groups were more indicative of the rifle’s capability, but most of the time they were referring to ES group size measurements, not necessarily mean radius measurements. Because the mean radius is a more efficient and reliable statistic, it can more accurately quantify the rifle’s precision in fewer shots. However, that doesn’t mean we can fire a single 5-shot group, measure the mean radius, and arrive at the truth of our rifle’s performance.
To avoid the “bughole” problem Litz mentioned, I typically limit my groups to 5-shots per target so that I can more accurately identify each hole and calculate the mean radius, but I will fire multiple 5-shot groups. This approach is similar to the NRA 5×5 method, except I average the mean radius over all the groups, rather than ES. Sometimes I even overlay the targets on top of one another, so I can look at the composite picture as if I fired all of the groups at the same target. (Note: A killer feature of a target analysis app would be to allow a shooter to pick multiple target photos and have them automatically overlaid on top of each other. Not only could you do stats on the composite group, but you could also see any slight shifts in the center of the groups. Update 12/16: Some of my readers mentioned that OnTarget TDS has features similar to this.)
Tip #5: Resist Chasing That Last Little Bit Of Precision
I’ve been at the range doing load development before and thought I discovered something special. After trying countless permutations of components and specs, I’d finally found a load that was uber-optimized for my rifle. Thinking back on those moments when I was trying to fine-tune a load and squeeze out that last bit of precision from my rifle, I now realize I was likely just chasing my tail – just as Jayden said. I thought I found something, but I was making decisions on truly insufficient data. I didn’t have a large enough sample size to support my conclusions. It’s very likely that if I would have continued shooting more groups, my uber-optimized load would have ultimately performed very similar to other loads where the powder or components varied slightly. I was convinced that I saw a pattern in my group sizes, but in reality, it could have as easily been the natural variation that should be expected within small sample sizes – and it likely was.
As humans, we tend to see patterns everywhere. That’s important when making decisions and judgments and acquiring knowledge; we tend to be uneasy with chaos and chance (T. Gilovich, How We Know What Isn’t So, 1991). Unfortunately, that tendency can often lead to us finding meaningful patterns in meaningless noise. (Dr. Michael Shermer in Patternicity, Scientific American, Dec. 2008).
Statistics is a tool that can help us differentiate between true patterns and meaningless noise.
Our innate desire to see patterns in chaos can lead to some funny things when it comes to analyzing groups and load development. I’ve heard intelligent, accomplished shooters explain that if a 3-shot group forms an upside-down triangle then you know you found a very special load. Unfortunately, statistics seem to tell us that might be closer to palm reading than real science. Things like the ladder load development method, with only one shot at each powder charge, might be similar. I realize that may be tough for some veteran shooters to accept, but it’s the hard truth.
I’ll restate Engleman’s conclusion one more time because it’s such a key point:
“I believe that you will find that by making statistically sound judgments, many loads produce statistically similar results and loads, in general, are not as finicky as current conventional wisdom would lead us to believe. … In reality, your shooting performance will probably improve since you can spend more time practicing with your set load and less time playing around with load combinations that have little or no statistical significance in terms of shooting performance.”
Having said all that, it is likely wise to ignore any performance difference that is 10% or less – unless you are willing to fire 50-100+ shots to verify if there is a legitimate difference or that was simply the natural variation that we should expect to occur with so many random/independent factors at play.
One of the coolest benefits of PrecisionRifleBlog.com becoming so popular is that over the past several years I’ve had the opportunity to meet some of the most respected long-range shooters in the world and have some really interesting conversations with them. I’ve been struck by how little time many of them spend doing load development. That isn’t to say that none of them do extensive load development, but I bet most of them spend exponentially less time fine-tuning a load than you might think. Guys like Scott Satterlee and others have shared their load development methods (like on this podcast), and it doesn’t involve spending multiple days at the range tinkering.
That is why today, I no longer spend days at the range doing load development. I’ve followed the lead of several pros, and when I find a load that gives me good performance, I load up a bunch of it and go practice. The only time I might spend more than a day tinkering with a load is when I’m shooting ELR and my bullet’s time of flight will be extended to 3 seconds or more. In that niche scenario, super-consistent velocities can be the difference in a first-round hit or miss, so it may be worth the extra effort to fine-tune a load – but then again, if I choose to go down that path I know I’ll need to fire a lot of rounds to verify any slight improvements I find are real and not noise from natural variation.
Honestly, a lot of shooters enjoy tinkering with load development and even find it therapeutic. If that is fun for you – continue on. I often say, “Don’t fix happy!” However, it’s important to be aware that if you aren’t making decisions on significant sample sizes, then you are likely seeing patterns in noise. I hesitate to use this analogy because it’s crude – but I can’t think of a better one that captures the essence of this problem: “It’s like masturbation. It might make you feel good, but it sure isn’t getting you anywhere.” 😉
“It’s not just about firing more shots. It’s about understanding what to expect, and planning your tests so that you can walk away with confidence. Even 20 shot groups can be useless depending on the scenario, and the best of us make this mistake all the time.” – Adam McDonald
How Much Does Group Size Matter?
A couple of years ago, I took a very objective look at how much performance improvement shrinking group sizes would have in terms of hit probability at long range. I knew as a long-range shooter, I (like most of us) can obsess over every little detail. We think everything is important! After all, we’re trying to hit relatively small targets that are so far you may not even be able to see them with the naked eye. While you can get away with a lot of minor mistakes and still ring steel at short and medium ranges, as you extend the range small mistakes or tiny inconsistencies are magnified. So, most things are important … but to differing degrees. So I wanted to have some hard numbers to help me understand when I reached that point of diminishing returns in terms of group size and when squeezing out that last bit of precision might not have a measurable impact in terms of hits out in the field. I will say that I was more than a little surprised by the results, and I’d encourage you to go read that article: How Much Does Group Size Matter?
Summary & Key Points
Whew! We made it. We covered a lot of ground! Here’s a quick recap of the key takeaways from this article:
- Accuracy and precision are not the same things. Accuracy refers to how well a group of shots is centered on a target. Precision describes the spread of individual shots about the center point of a group of shots. Unlike accuracy, precision can’t be adjusted by dialing a knob on the weapon.
- Extreme Spread (ES) is not a very good measure of dispersion. Range statistics like ES are statistically far weaker because they virtually ignore inner data points. They are the least efficient statistics but are also the most commonly used because they are so easy to measure in the field and so familiar to shooters.
- Mean radius, also known as average to center, is the average distance from each shot to the center of the group.
- Mean radius uses information from every shot in a group, not just the two most extreme points. Because of this, mean radius can provide a higher confidence measure of precision than ES. However, mean radius is a little harder to measure than ES.
- Mean radius allows us to resolve smaller differences in precision with fewer shots. If you are comparing two loads or two rifles and trying to decide which is superior in terms of precision, comparing the mean radius of the groups fired will lead to more reliable conclusions than comparing ES.
- Don’t exclude outliers in your groups unless one was undeniably a result of human error.
- 10-shot groups are a more reliable indicator when it comes to predicting what a load is likely to do in the future.
- By making statistically sound judgments, you may find that many loads produce statistically similar results and loads, in general, are not as finicky as conventional wisdom would lead us to believe.
- As humans, we naturally tend to see patterns everywhere, but that can often lead to us finding meaningful patterns in meaningless noise. Statistics is a tool that can help us differentiate between true patterns and meaningless noise.
- It’s not just about firing more shots. Plan your tests and analyze your targets in a way that you’ll be able to walk away with confidence in your decisions.
I’ll leave you with one last quote from Statistics, Shooting & The Myth of the 3 Shot Group:
“Final conclusion: What counts is hitting the target. Realistic target engagement > punching paper at 100 yards 10 shots at a time > punching paper at 100 yards 3 shots at a time > sitting in my office writing about statistics.”
Isn’t that the truth?! Let’s get out and shoot! 😉
Other Articles In The “Statistics For Shooters” Series
Here is a recap of all the articles in this “Statistics for Shooters” Series:
- How To Predict The Future: Fundamentals of statistics for shooters
- Quantifying Muzzle Velocity Consistency: Gaining insight to minimize our shot-to-shot variation in velocity
- Quantifying Group Dispersion: Making better decisions when it comes to precision and how small our groups are (this article)
You can also view my full list of works cited if you’re interested in diving deeper into this topic.