What statistical analyses do MGS run on their data?

[Mitchlikesgolf]

Can someone point me to a discussion or article where MGS discusses what statistical analysis they run on their data? I haven't been able to find much. 

On their best golf ball or best driver series, they don't really mention how they analysed the data. Do they run regression models? Because the data is likely very skewed to the right, I'm assuming they do a log transformation or something similar? Do they calculate a standard error to see if their means are accurate? When they compare multiple balls or clubs, do the report pvalues to determine if the differences are actually significant? 

I'm guessing they hire a statistician to do all this, but they don't talk about in the articles. I'm definitely no stats expert, but it would be fun see how they analyze it all. 

Thanks for your help! 

Edited February 1, 2023 by Mitchlikesgolf

[Mitchlikesgolf]

Or maybe data like ball speed and distance follow a more normal distribution and only needs to be shifted or rescaled? However, I'm guessing the mean is typically much closer to the max than the min, which would probably require some other type of transformation for correcting right skew. 

[GolfSpy_APH]

https://mygolfspy.com/how-we-test-most-wanted/

Here is this right from the blog!

[Mitchlikesgolf]

2 minutes ago, GolfSpy_APH said:

https://mygolfspy.com/how-we-test-most-wanted/

Here is this right from the blog!

Thanks! But this just says how they collect the data. I'm looking for what stats they run on the data. I'm assumimg they aren't just comparing averages because those comparisons would be statistically meaningless. Basically, I'm looking for pvalues more than anything. If they say ball x is 1.2 mph faster than ball y, I want to know if that is a statistically significant difference between the means or if the difference can be chalked up to random chance. Thanks for the reply! 

[GolfSpy_APH]

5 minutes ago, Mitchlikesgolf said:

Thanks! But this just says how they collect the data. I'm looking for what stats they run on the data. I'm assumimg they aren't just comparing averages because those comparisons would be statistically meaningless. Basically, I'm looking for pvalues more than anything. If they say ball x is 1.2 mph faster than ball y, I want to know if that is a statistically significant difference between the means or if the difference can be chalked up to random chance. Thanks for the reply! 

To my knowledge a lot of that is proprietary data to them and not shared, or made public. 

That being said if it's ball data message Tony Convey (think his contact is on the MGS staff site.

If it's for Most Wanted then Phillip Bishop would be the best to contact.

[Mitchlikesgolf]

10 minutes ago, GolfSpy_APH said:

To my knowledge a lot of that is proprietary data to them and not shared, or made public. 

That being said if it's ball data message Tony Convey (think his contact is on the MGS staff site.

If it's for Most Wanted then Phillip Bishop would be the best to contact.

Thanks! I guess I understand that the data is proprietary, but I don't know why the stastical outputs would be? Wouldn't they want to publically explain their statistical methods in order to publish credible scientific research? I'll reach out to the staff for answers though, thanks!! 

[RichL85]

Statistical significance would likely negate a lot of the differences seen. Based on my experience with Minitab, it's surprising how large of an actual difference it takes to become statistically significant sometimes. I've seen general averages differ by 20-30% but not show any verifiable statistical difference.

[Mitchlikesgolf]

47 minutes ago, RichL85 said:

Statistical significance would likely negate a lot of the differences seen. Based on my experience with Minitab, it's surprising how large of an actual difference it takes to become statistically significant sometimes. I've seen general averages differ by 20-30% but not show any verifiable statistical difference.

That depends on the variance in the data and the sample size, right? I.e. If 1000 balls were hit with two brands and each brand had a very low variance or a tight standard deviation, then a difference in means of only a couple feet could prove to be significant. If it's statistically significant (for whatever pvalue, such as less than 0.05), that just means that there is real difference between the means that cannot be explained by random chance alone. If it's not significant, then there is a likelihood that the test results could be totally different if the test were run again, which would invalidate any claims. 

I don't know how MGS is run, but I'm guessing (or maybe hoping) that they design their experiments to have high power (probability to reject the null). Otherwise, all the claims don't really mean much, ya know? Again, the results might be significant, we just don't know unless they explain their statistical methods. 

[RichL85]

8 minutes ago, Mitchlikesgolf said:

That depends on the variance in the data and the sample size, right? I.e. If 1000 balls were hit with two brands and each brand had a very low variance or a tight standard deviation, then a difference in means of only a couple feet could prove to be significant. If it's statistically significant (for whatever pvalue, such as less than 0.05), that just means that there is real difference between the means that cannot be explained by random chance alone. If it's not significant, then there is a likelihood that the test results could be totally different if the test were run again, which would invalidate any claims. 

I don't know how MGS is run, but I'm guessing (or maybe hoping) that they design their experiments to have high power (probability to reject the null). Otherwise, all the claims don't really mean much, ya know? Again, the results might be significant, we just don't know unless they explain their statistical methods. 

Exactly. I deal with data measurements where I'm using Minitab to determine statistical significance for one variable in a measurement that can be affected by about 20 different other variables, so I can see some wide ranges of measurements. I would hope that a manufactured product like golf equipment doesn't have that same variability that I deal with, but statistical analysis would at least provide some additional credibility to the testing to eliminate as much outside, unmeasured variable contributions to the overall result there is. 

[G56788]

I’m also curious about who is doing the statistical analysis and what controls are in place to reduce or eliminate subjective bias. For example, is the data for a club labeled with the name of the club or assigned a random number to prevent the statistician from injecting bias into the interpretation, whether intentionally or not? As a recent thread pointed out, we all have our golf biases. 

[RichL85]

1 hour ago, G56744 said:

I’m also curious about who is doing the statistical analysis and what controls are in place to reduce or eliminate subjective bias. For example, is the data for a club labeled with the name of the club or assigned a random number to prevent the statistician from injecting bias into the interpretation, whether intentionally or not? As a recent thread pointed out, we all have our golf biases. 

It's pretty hard to have a subjective bias for pure data numbers. My initial thoughts are that it would be much tougher to eliminate the personal variables associated with the humans involved in the physical testing. If various individuals are involved in club testing for example, the visual bias, weighting differences, etc. all come into play and that makes it more difficult to isolate measurables for statistical comparison. However, for example, ball testing done with a robot would eliminate variables such as club differences, swing variability, etc. At that point, assuming the robot delivers whatever club is being used in a consistent manner, the data is the data. 

I guess short answer for me, is that I would be much more interested in how MGS goes about isolating/eliminating as many variables as possible more so than I would be critical of the statistical analysis methods used. The statistical methods would be interesting from a personal nerd standpoint, but MGS has been called out more than once for various tests that make a claim without properly isolating that as the one variable in a given test.

[cnosil]

7 minutes ago, RichL85 said:

IIf various individuals are involved in club testing for example, the visual bias, weighting differences, etc. all come into play and that makes it more difficult to isolate measurables for statistical comparison.
 

However, for example, ball testing done with a robot would eliminate variables such as club differences, swing variability, etc. At that point, assuming the robot delivers whatever club is being used in a consistent manner, the data is the data. 

 

The use of various individuals and those variable you mentioned is what the Most Wanted club testing is trying to gather.   The intent is to look at the variations across the pool of testers and identify which stock clubs might be the best for the player that walks into a store and buying off the rack.  I am pretty sure there is a weighting associated with the various metrics as part of the calculations; it could favor dispersion over distance, or it could favor distance over dispersion.  Also,  Just because  a club is identified as most wanted, doesn’t mean it is the best club for everyone.   For example, The “winning” club could have been the best club for 30% of the testing pool.
 

Their ball testing is done with a robot and shots are measured with both GC Quad and Trackman.  The testing in 2021 was done at PXG.  

[Mitchlikesgolf]

32 minutes ago, RichL85 said:

It's pretty hard to have a subjective bias for pure data numbers. My initial thoughts are that it would be much tougher to eliminate the personal variables associated with the humans involved in the physical testing. If various individuals are involved in club testing for example, the visual bias, weighting differences, etc. all come into play and that makes it more difficult to isolate measurables for statistical comparison. However, for example, ball testing done with a robot would eliminate variables such as club differences, swing variability, etc. At that point, assuming the robot delivers whatever club is being used in a consistent manner, the data is the data. 

I guess short answer for me, is that I would be much more interested in how MGS goes about isolating/eliminating as many variables as possible more so than I would be critical of the statistical analysis methods used. The statistical methods would be interesting from a personal nerd standpoint, but MGS has been called out more than once for various tests that make a claim without properly isolating that as the one variable in a given test.

An appropriate stastical analysis will actually isolate intended variables. For example, a single datum line should/could include tester number, tester handicap, swing speed, ball speed, ball distance, and what ever other variables you want to include. An ANOVA model or regression model or whatever statistical model is appropriate would be able to isolate each independent variables (tester number, swing speed, handicap) effect on the dependent vatiables (ball speed, distance, etc). You would be able to see if there are significant causal relationships between variables as well as if any interrelated variables have any significant effects. 

In short, even basic statistical models can be used to isolate variables. You wouldn't want to control for all of these variables in the experiment because "we" are curious to understand how these (independent) variables affect the results (dependent variables). 

[silver & black]

My head hurts. 

[G56788]

2 hours ago, silver & black said:

My head hurts. 

Hopefully not on my accord, I just wanted to know if the Excel spreadsheet is called “Titleist TSR2” or “Club #23-5374”

[cnosil]

1 hour ago, G56744 said:

Hopefully not on my accord, I just wanted to know if the Excel spreadsheet is called “Titleist TSR2” or “Club #23-5374”

Neither.  It is something like: tester initials and year tester started_what test is being performed_group number.   There are a couple of more things in the name but I don't remember since I don't save the data.   The file contains all the data for the clubs that were part of that sessions test group. 

[GolfSpy_APH]

11 hours ago, silver & black said:

My head hurts. 

This was kinda my thought.

Thanks to @cnosil for chiming in.

Love data and various studies, but for me at some point being an amateur my variables in my game overcome the statistical differences in various products.

 

[cnosil]

4 hours ago, GolfSpy_APH said:

Thanks to @cnosil for chiming in.

 

No problem; since I am one of the most wanted testers, I am fairly familiar with the data capture process.   
 

Anticipating the follow on questions, it is a standard gc quad export, the names of the clubs in the foresight software indicate the club model, and once all the data is captured they are sent to be imported into the software that does the data analysis. 
 

[Franc38]

The data in itself is important and interesting, for sure, and indeed, MGS seems to do a fairly decent job at gathering it relatively free of biases.
The data treatment, the statistical tests, and all the associated interpretations, that's what we would like to know.

And it's not proprietary, I can tell you... it's either published (and publicly known/available) stats or a bunch of nothing...

[cnosil]

3 hours ago, Franc38 said:

And it's not proprietary, I can tell you... it's either published (and publicly known/available) stats or a bunch of nothing...

What is known is that is takes the various results from the shots and does calculations.   What isn't know is the weighting of the various statistics used in the calculations.