Unofficial Review – How MGS chooses their reviewers, a non-scientific study of 18 months of member reviews.

[TBS]

Background 

After I took a few months off of MGS due to some work and life things that needed my attention, I finally got some time to log back on and post a long overdue review. After posting, I found myself curious as to how I was selected and the likelihood of being selected again, especially since my product was so niche and did not receive many interactions. So I took the challenge upon myself to do a not so scientific study of how reviewers are selected and compare it to the post (https://forum.mygolfspy.com/faq/testing/how-are-equipment-testers-chosen-r11/) by the Mods on how testers are selected. While there are no bombshells, I think we Spys understand the value of data in telling our story. So, I did my best at pulling all data from the past 18 months of reviews

 

Methods 

First off, I am no data scientist. From DJ to Bryson, I’m somewhere in the middle. I mainly do very basic data analysis in my role through excel. I know, I know, SQL is much more efficient and I’ll get around to learning it. For this study, I focused on 18 months of product reviews. I chose this timeframe to get some data points outside of the pandemic. I started going through the review pages and noticed it was not as organized as I thought, so I chose to do only 18 months of reviews that were already on the “Past Opportunities” page in the dropdown menu. Like I said, I believe that there may have been a few more reviews that have not been archived to this page, but I don’t think it was significant.

I then did the manual labor of data entry and collection. I clicked on each individual “TESTERS ANNOUNCED” within the time frame, recorded the following data points 

-       date of review published

-       product

-       rough product MSRP 

-       MGS username

-       If the User has a Donor badge on their profile

-       Content count

-       Total reputation

-       Month and year MGS account was created

-       City and State

Once all of this information was input into the spreadsheet, I then added a note if they had multiple reviews and I also created an additional metric to measure the quality of the posts. All analysis comes from these data points.

 

Limitations

Overall, the main limitation is that this dataset was manually curated, so there could be a mistake, but any mistake would be minor. I have checked that there is not an additional 0 added somewhere but there could be +/- 5 units off when it comes to reputation due to fat fingers on a keyboard. Additionally, the MSRP was calculated off a quick Google search. I get that there could be sales, or the price was actually $25 higher at the time of testing. Finally, the content created, and total reputation numbers are not accurate to the time the member was selected. There is no way for me to go back and see this information, but I proceeded with collecting this because for the most part, I believe that most people’s posts grow linearly. So, the quality of posts metric should not be too far off. I will address that in bigger reviews (Cobraconnect), this ratio can get skewed. Enough with the small talk, let’s take a look at the data.

 

Data:

Let’s take a look at the data!

By the numbers-

-       48 total products

-       202 unique testers

-       ~$119,625 total costs of products tested

 

As I said in the introduction, I will be using the MGS selection criteria as a framework of analysis.

Picture – I did not go through and collect this data. This is also the easiest of all of them. Put something up there! I’ve seen pictures of the people (I assume), cartoons, their children, memes, etc. Just put something up. (Inclusive, but probably true)

Prove review chops - 71 (35%) of people had more than 1 review. If you get chosen, even for the smallest thing, a pushcart distance tracker, a dozen balls, etc. take it seriously and write a good review. This means good pictures, concise statements that clearly articulate your opinion on a product, timely submissions, and good follow ups. A good first review will go a long way to getting your second review and the data clearly backs this up. Additionally, one anecdotal note, it was very common for there to be a progression of items to review, starting with a smaller item and then growing to more expensive items. (True)

Start posting – Please remember that this category is not perfect because I was not able to pull the post and reputation numbers at the time the testers were selected. I am assuming that people post at a linear rate. I looked at three areas for this section. Number of posts, Total reputation, and third metric that I created, Quality of post score. I wanted to look at how many interactions per post the reviewer gets. This metric speaks to higher quality of posts for the forum.

	Total number of posts	Total reputation	Quality of Post Score*
Range	5 - 14,307	3 - 26,497	0.09 - 5.8
Mean	1,412	2866	1.92
Median	513	824	1.85

Because of such a large range with bunching happening at the higher end of each category, I would suggest forum members look at the Median. The best chance for you to be chosen is to have over 500 posts with over 800 total reputation which comes out to 1.85 interactions per post. Like I said earlier, there were a number of members chosen with less than this number but most under 500 posts were in the 100-200 range. I think the biggest takeaway, is to post frequently and create posts that others want to engage in. I think an average user can get to 200+ posts in 2-3 months. 

There might be an issue that leads to inequity when it comes to being chosen. As I started reviewing some of the posts and interactions, if you are chosen for a more high-profile test, your posts and interactions will increase at a higher rate than other members. This easily leads to a boost in your profile making it easier to be selected. I don’t think there is a way around smoothing out this distinct increase, but it is something that I hope mods take notice of. (True)

 Become an MGS Donor

This category comes with major limitations. The only public piece of data that I was able to collect was if the member included the MGS Donor badge in their profile. From this information I found 59.9% of testers had a donor badge in their profile. I suspect that this number may be closer to 70-80% but these members have not reached out to get the donor badge added.

(Inconclusive but probably true)

Overall, MGS is clear about how testers are selected. For the most part their criteria is clear and the data shows that they follow the criteria for selection. 

Common misconceptions

I wanted to add a section that debunked some of the common misconceptions that I had when I first joined the forum. 

-       I have to be a member for a long time before I get anything (FALSE)

o   Over the past 18 months, members who were selected had profiles created from March 2010 – November 2020. 42 testers (21%) joined in 2020. If you’re new and are contributing to the forum, you have a decent chance of being selected.

-       Once I get one review, I won’t ever get another (FALSE)

o   71 (35%) members had more than 1 review in 18 months. I suspect if I go back even further, they had other reviews too. As I said in the above section, if you get chosen and do a good review, you actually put yourself at an advantage to be selected again.

o   No reviewer had more than 3 reviews in the past 18 months. It appears that this is the limit, it looks to be extremely rare for a member to have more than 1 review every 6 months. 

-       I have to be good at golf to be selected for a review (FALSE)

o   This idea is very much false, MGS wants members of all skill levels to be involved in testing. In order to prove this, I pulled the handicaps listed on members’ profiles. This is a limitation because it is member provided information that is most likely not updated regularly. Regardless, here is what I found.

• Range – (33 - +5)
• Mean – 10.09
• Median – 9.2

o   If anything, having an accurate handicap is beneficial to being selected because, it appears that MGS wants to have a broad range of testers for each product, specifically balls and clubs.

 

Conclusion

For the most part, MGS sticks to what they say when it comes to choosing testers but it appears that there is some non-measurables that they use when it comes to selecting first time testers. It does not appear that there is a formula or that it is completely random. There seems to be a human element which makes it hard to say exactly why specific members were chosen over others from the outside looking in. 

Shoutout to @Brian A, @jlukes, @B.Boston, @txgolfjunkie, @gavinski91, @Mtbryant01, @daviddvm for having been selected to review over $3,000 worth of equipment! These individuals were ones who had been selected for this year or last year's Cobraconnect challenge. So each of them had at least one product review in addition to cobra connect in the past 18 months. 

 

TL;DR - Overall, post quality content frequently, if you are given a chance to test an item, take it seriously, and keep posting. If you’re new and you stay involved, your time will come. 

 

Okay, I’m tired. Let me know you’re thoughts!

Edited May 13, 2021 by TBS
Added info on the big winners

[GolfSpy_APH]

Interesting breakdown. I know there is a human element behind it all and that we want to encourage new members join and be active as a chance to become a new tester along with MGS wanting to give new people a opportunity to test products. There are also others (like myself) who have been very fortunate to take part in multiple tests. They are a lot of work and a lot of fun.

Another part of it (at least I feel) after testing once we become more selective in what we apply for. Again speaking from personal experience I likely won't be applying for another putter review even if I love the putters that are available. Although I loved doing the review it was very hard and difficult to quantify results. Plus I like the putter so I think it will stay in the bag for a while longer. 

I would be curious to know what all the Golden Tickets were used on?

This is going back a little while, but there have also been testers where OEM's have got MGS to choose them privately (I remember there being a Bridgestone set and the Ping Blueprint Irons) these are likely outside of the 18 months you were looking at. Also there is the fact its important that the OEM get a quality review for the product they are giving away and for MGS it is important to select members who will provide that. 

In the end personally I feel a good introduction post, being active and taking interest in topics on the site (whatever they may be cooking, fantasy sports, CHA and more) will help get your name noticed. Doing a unofficial review on a product you have is another great way prove you have what it takes to do an Official Review. It certainly would have been interesting if there was a set formula, but much like golf there is a feel and rub of the green element and no way to predict what is going to happen or who will for sure be selected!

Great job putting all that data together and good recap as well, it certainly was interesting to go through and see some of those numbers which came up.  
 

[MattF]

5 hours ago, Apolloshowl said:

This is going back a little while, but there have also been testers where OEM's have got MGS to choose the privately (I remember there being a bridgestone set and the Ping Blueprint Irons)

Correct. Myself and @MaxEntropy were both chosen for the Bridgestone one...had to sign an NDA for that one too.

[TBS]

5 minutes ago, MattF said:

Correct. Myself and @MaxEntropy were both chosen for the Bridgestone one...had to sign an NDA for that one too.

Yes - That should be added somewhere. In all of those cases, these "special" selections were clearly stated in the Tester announcement if I remember correctly. 

[cnosil]

What your conclusions seem to indicate is exactly what the moderators state in their posts about how testers are chosen.  As you also indicated you had to make some assumptions on things that change over time like post counts and if/when a person became a donor, and their post frequency since becoming a member.  I have been a forum member since 2012, but didn’t get engaged until about 2017.   There are also quite a few frequent posters that never apply for tests as that isn’t the reason they are on the forum.  Even with the selection criteria there are still testers that take the product and don’t perform the reviews or have to be asked multiple times to post the review.  I would also guess that the mods keep track of who applies to what test opportunities to see who throws there name in the hat for everything versus those that apply for items they are really interested in and want to review....I know I probably would.    

as a tester you can’t control how people will respond to the review or even ask questions to help shape your review.  That said,  part of being a tester is trying to generate that interest by posting comments in the review thread to generate interest.  
 

The nice thing about your independent assessment is that it shows that anyone can be selected, being a part of the forum is important, and the criteria isn’t just be best friends with the people making the selections.  

[Headhammer]

Kudos for doing the work that proves MGS is true to their criteria in selecting testers. I've been lucky enough to be selected twice as a tester and while the reviews are a lot of work, I enjoyed the heck out of the process. One suggestion I have for members that want to be testers is to only apply for opportunities that you are passionate about. Because the reviews are a lot of work, if you are just entering to win something, you will find the process tedious and that will most likely be reflected in the quality of your review. 

[JonMUSC08]

@TBS Great job with this. Helps verify to everyone that certain criteria are looked at but also you dont have to have 1,000+ posts and reputation to win. I've seen that usually you're first opportunity will be a "smaller" one like a rangefinder/ balls/ shoes/ etc and not necessarily a set of irons or cobraconnect, etc. I have been fortunate enough to test a Precision Pro rangefinder and then was later selected for Ping irons BUT i've held out on applying for Cobra Connect bc i dont think i could keep up with the weekly contests. 

[MaxEntropy]

1 hour ago, MattF said:

Correct. Myself and @MaxEntropy were both chosen for the Bridgestone one...had to sign an NDA for that one too.

Yeah, that was kind of a strange experience - we were getting the clubs pre-release. I was paired with a bit of a club-ho early on who took way too much of an interest in them, but I could not discuss much about them. That led me to buying some iron covers to help hide them 

[Headhammer]

2 minutes ago, MaxEntropy said:

Yeah, that was kind of a strange experience - we were getting the clubs pre-release. I was paired with a bit of a club-ho early on who took way too much of an interest in them, but I could not discuss much about them. That led me to buying some iron covers to help hide them 

Yeah that was the reason you bought Iron covers!

 

[StrokerAce]

HAHAHAHA.... as someone who used to pick testers this is interesting....some of it is right, some of it is close and some of it is, well, I'll leave that up to the current moderators who select the testers now.

I will say though, you left one thing out:

If you are selected to test something - anything - and you don't post a word at all in the review thread or reach out to one of the mods and explain the extenuating circumstances (yes, they understand people have actual real lives and jobs too) you will never be selected to test again.

Not a warning, just some advice....

[TBS]

14 minutes ago, StrokerAce said:

HAHAHAHA.... as someone who used to pick testers this is interesting....some of it is right, some of it is close and some of it is, well, I'll leave that up to the current moderators who select the testers now.

Go ahead and share... Isn't MGS about providing "the truth and nothing but" and shouldn't their forum fall under the same mission?

[Lacassem]

Great information here and doing this as an outside eye is great. If you follow a majority of the tests just following the peak season (right now) reiterates the MGS selection criteria.

[GolfSpy_APH]

1 hour ago, TBS said:

Go ahead and share... Isn't MGS about providing "the truth and nothing but" and shouldn't their forum fall under the same mission?

Hopefully not to repeat what I said in my other post, but we know there is a human element to it and I think MGS, mods and the forum community is pretty open as to what in general it takes to be selected. You have mentioned several of those points in your post as well as there are several threads on what it takes to be selected. 

The forum's mission statement: “MyGolfSpy's Community Forum aspires to continued growth while adhering to its core goal of being a place for serious, honest and fun discussion on both golf and life, and a place to go for product reviews - a true online ‘19th Hole’ where all are welcome and all members are treated with respect and friendship.”

For some the opportunities come quicker and others it comes much further down the line. Pretty sure it took a couple years before I was selected for one and it was well worth the wait. There is so much that the forum has to offer and testing is only a small part of it. Yes, most come here to start with because of testing however as has been discussed in many threads it is the community and everything else that the site has to offer which keeps us here. 

Again a lot of the general guidelines are laid out as you have pointed out, from there provide value in the content you produce. It is one thing to have 1,000 comments in various threads it is another to have valuable content for the members here to read and react to. I still believe as a tester you have a big responsibility to the members, MGS and of course the OEM to provide a quality review. Whether you liked the product or not, a honest quality review is what you sign up for. Not just a piece of free gear (I want to be clear I don't believe you think that either, at all), with that there needs to be a level of trust and it is why I mentioned in my first post that a good intro, unofficial review can go a long way in helping members get noticed and gain that trust. 

I love seeing new people get selected as it gives me a chance to get to know them a little better. I have built some great friendships through this site and enjoy the community. I hope that all who are selected enjoy the process as well as can assist other new members in guiding them in the right direction to do the proper things to hopefully have them selected in the future as well. 

[Lacassem]

19 minutes ago, Apolloshowl said:

Hopefully not to repeat what I said in my other post, but we know there is a human element to it and I think MGS, mods and the forum community is pretty open as to what in general it takes to be selected. You have mentioned several of those points in your post as well as there are several threads on what it takes to be selected. 

The forum's mission statement: “MyGolfSpy's Community Forum aspires to continued growth while adhering to its core goal of being a place for serious, honest and fun discussion on both golf and life, and a place to go for product reviews - a true online ‘19th Hole’ where all are welcome and all members are treated with respect and friendship.”

For some the opportunities come quicker and others it comes much further down the line. Pretty sure it took a couple years before I was selected for one and it was well worth the wait. There is so much that the forum has to offer and testing is only a small part of it. Yes, most come here to start with because of testing however as has been discussed in many threads it is the community and everything else that the site has to offer which keeps us here. 

Again a lot of the general guidelines are laid out as you have pointed out, from there provide value in the content you produce. It is one thing to have 1,000 comments in various threads it is another to have valuable content for the members here to read and react to. I still believe as a tester you have a big responsibility to the members, MGS and of course the OEM to provide a quality review. Whether you liked the product or not, a honest quality review is what you sign up for. Not just a piece of free gear (I want to be clear I don't believe you think that either, at all), with that there needs to be a level of trust and it is why I mentioned in my first post that a good intro, unofficial review can go a long way in helping members get noticed and gain that trust. 

I love seeing new people get selected as it gives me a chance to get to know them a little better. I have built some great friendships through this site and enjoy the community. I hope that all who are selected enjoy the process as well as can assist other new members in guiding them in the right direction to do the proper things to hopefully have them selected in the future as well. 

Your original post hit on once selected you become a little more selective in your choices for applying to tests. I know this is true for me, once I got the first one done and had the lay of how reviews go (was not selected initially but someone had to back out due to the pandemic and  gave me the opportunity) I became comfortable with the process and have really tried to focus on things that truly interest me. The site has given me insight into things I never really looked at before. It has prompted a few of my purchases over the past year (push cart and planemate) and has given me some great information along the way. Also some great private chit chats about life in general of note @Shankster@bens197and getting out with some great members @StrokerAce and @Wedgieand of course my main group player @B.Boston. 

With the new process of getting folks involved to join from social media I hope a lot realize the benefits of the site and are able to enjoy it as much as I do!

[GolfSpy MPR]

Whoa! This is amazing work.

As the one who wrote most of the FAQ post and who has a hand in choosing testers, it's certainly reassuring to find data that says that we're doing what we say that we're doing. To be honest, I'd rather see the number of new testers be even higher than it is. We are always looking to get new members involved in testing. I know we have a ton of people who join the Forum just to apply for tests: if you're that guy and you're reading this, please start posting, even a little. It's not feasible for us to pick people who have no posting history; we just don't know if we're going to get a review, and ghosting reviews jeopardizes relationships with the OEMs that make the reviews possible. But if you post, even a little bit, we're going to notice, and we will try to get you involved.

On the Donor thing: here's a reminder that the Donor tags aren't automatically applied, and we Forum mods/admins don't have access to the donation history for the blog. If you want to be recognized as a donor to MGS on the Forum, shoot one of us a DM with your transaction ID. That stats bear out what we've said before: donations don't guarantee reviews (this isn't pay for play), but it's a component that we take into account, sometimes  to break ties.

As others have mentioned: being selective in applications will often help your cause. There are some who apply for everything, even when they've just been picked for something and haven't completed that review yet (hint: you'll never be picked for a review while you're still in process of another review). Selective application guy has a leg up on apply for everything guy.

As others have said, we aim to run a pretty open shop here. If you've got any other questions about the selection process, I'm happy to answer them.

[bens197]

55 minutes ago, Lacassem said:

Your original post hit on once selected you become a little more selective in your choices for applying to tests. I know this is true for me, once I got the first one done and had the lay of how reviews go (was not selected initially but someone had to back out due to the pandemic and  gave me the opportunity) I became comfortable with the process and have really tried to focus on things that truly interest me. The site has given me insight into things I never really looked at before. It has prompted a few of my purchases over the past year (push cart and planemate) and has given me some great information along the way. Also some great private chit chats about life in general of note @Shankster@bens197and getting out with some great members @StrokerAce and @Wedgieand of course my main group player @B.Boston. 

With the new process of getting folks involved to join from social media I hope a lot realize the benefits of the site and are able to enjoy it as much as I do!

Spot on ML. Aside from the equipment and content, there are some fantastic humans who occupy this site. The discourse is kind and even in disagreement, it’s always cordial. 

[kardboard.kid]

Thanks you, this is very well laid out.  As someone who has done reviews, I will also join in with the assertion, "That they are work".  The latest for me has been challenging, more so due to personal life issues, than actual product.  I also recognize the importance for timely reviews, and since I have only finished and posted half the review, I have decided, I would not put my name in the ring this year for any other review.  I enjoy being part of this group, both reading posts, and participating.  One other note to that I do not remember reading.  If you are selected, plan on spending some $$ on the review.  I was going to change out the shaft and grip on the 3I at some point, but the club pushed me along faster.  In the end, you want to make it yours, not some generic repost from the website.

[JohnSmalls]

This was a fun read. Good work on compiling all of this together.  Enjoy your time on the forum!

 

1 hour ago, Lacassem said:

Your original post hit on once selected you become a little more selective in your choices for applying to tests. I know this is true for me, once I got the first one done and had the lay of how reviews go (was not selected initially but someone had to back out due to the pandemic and  gave me the opportunity) I became comfortable with the process and have really tried to focus on things that truly interest me. The site has given me insight into things I never really looked at before. It has prompted a few of my purchases over the past year (push cart and planemate) and has given me some great information along the way. Also some great private chit chats about life in general of note @Shankster@bens197and getting out with some great members @StrokerAce and @Wedgieand of course my main group player @B.Boston. 

With the new process of getting folks involved to join from social media I hope a lot realize the benefits of the site and are able to enjoy it as much as I do!

Just wait until you venture to the south! 

[Lacassem]

2 minutes ago, JohnSmalls said:

This was a fun read. Good work on compiling all of this together.  Enjoy your time on the forum!

 

Just wait until you venture to the south! 

My wife is well aware when we are able to travel and eventually move I have a whole crew of folks I will be getting rounds in with lol! She always says shes a golf widow in the summer....Atleast its not 12 months right?!

[golfish!]

Definitely agree on the topic of getting selected for small things first then getting selected for bigger items. I started with a test for CBD oil, then got selected for a Ben Hogan 3W test, and now I am testing the Motocaddy electric push cart. Makes sense that the admins will bring you in on testing with a relatively lower value item to make sure you are reliable and can write a coherent review then put you in the hat for bigger ticket items. 

Also agree that as I have been selected for testing, I become more selective on my applications. Before my first test, I applied for everything! After actually completing a test, I realized how much work goes into an actual review and now I only apply for items I am actually interested in and willing to put in the work. Plus, I am a perfectionist so I end up writing this huge review and then have to pare it down to keep it within the guidelines of the review so extra work for me xp.

The way that I approach testing opportunities is also from the perspective of sharing is caring. There are only so many testing opportunities and many times more eligible forum members. As a courtesy, if you have already been chosen ( as @GolfSpy MPR mentioned, if you are currently testing you won't be selected but that should be a given!) for some testing opportunities you should give others an opportunity (and give the admins less applications they have to go through lol) 

Overall great post @TBS!