Rating the Rater
This past week I wrote my 500th Yelp review. With little fanfare, in my own bizarre celebration, I decided to model a predictive algorithm for my rating of any given restaurant based on existing online reviews. For example, I would want to be able to say that my Yelp rating will be the general Yelp rating*.9+.1*the general Google rating. I had been planning this for a while because I choose a lot of restaurants for myself and others. Even a slight improvement in my picking would mean a couple more enjoyable restaurant experiences per month. The time seemed right with my personal rating dataset at 500. This algorithm I describe is, albeit simple, tailored to my tastes and ratings history. It will not necessarily match your tastes. My goal was to see how Yelp, Google, and Facebook did at recommending me restaurants. I learned a few things about online ratings. I will change how I approach the way I choose restaurants.
- Yelp’s predictive power could be improved by consulting Google and Facebook.
- Very popular places would be worse because they were “touristy” or too crowded.
Gathering the Data
I sent my Yelp profile through Upwork.com to a virtual assistant for gathering the necessary data. He gathered Yelp, Google, and Facebook ratings for each of the 500 restaurants I visited over the past 3 years. It took him two weeks’ by calendar (not nearly 80 hours) to get this data into a spreadsheet. With the spreadsheet, I was able to begin analyzing my ratings and their comparison to the general web consensi on Yelp, Google and Facebook.
Analyzing the Data
My median rating on Yelp is a 4. The median general rating for those restaurants is a 4.25 on Yelp, 4.5 on Google, and 4.7 on Facebook. These site’s rate restaurants progressively higher. Practically this means, on average, a 4.7 on Facebook is a 4.25 on Yelp and a 4.5 on Google. I rate restaurants more harshly than any of the sites analyzed.
Takeaway: Be less impressed with high scores on Facebook than on Google. Be even more with high scores on Yelp.
Developing a Prediction Model
I first analyzed this dataset myself with a free Mac data analysis program called Gertl as well as the Solver add-in on Excel. This allowed me to see how strongly correlated my scores were to the general Yelp, Google, and Facebook scores. The strongest correlators with my restaurant ratings were Yelp rating (r=.9) and Facebook rating (r=.8). This means that 89% of changes in my rating can be explained by the general Yelp rating and 64% can be explained by the Facebook rating.
Preliminary Takeaways: Yelp is the best predictor of my restaurant ratings. It alone is better than any combined Yelp/Facebook/Google aggregate rating. Facebook is the second best predictor. Google was not statistically reliable.
Improving the Model
I looked for a more technically skilled assistant to help me build a predictive model beyond these insights. I found a Ph.d in Financial Modeling from France to help me out. Because he asked too much money for the assignment, I told him I didn’t want the contract. In response, he solved the problem for me FOR FREE. He ran several regression models and found a simple solution. His model was slightly better than mine [mine was to use a simple linearly adjusted Yelp rating(yelpscore*8.899=myyelpprediction)]. He found the best predictor of my restaurant rating was to use Excel’s “trunc” function on the general Yelp rating. “Truncate” removes the decimal from a number (4.5 becomes 4, 3.5 becomes 3). With his model, 86% of the predictions fall within 1 of my actual rating (and it gets it right 36%). That is 9% closer than using the actual Yelp rating (which only gets it EXACTLY right 15% of the time) and 6% closer than just guessing a 4 (which gets it EXACTLY right 34% of the time).
Takeaway: The Truncated Yelp score is the best predictor of my actual Yelp restaurant rating.
How to Pick Restaurants?
I haven’t learning a ton from this exercise. The models I found only marginally improve the predictive power of Yelp’s provided ratings. I suspect all the variations in my analysis are significantly better than randomly eating at a restaurant, except maybe Google.
- Yelp is much better than any other site at predicting how much I’ll enjoy a restaurant. The truncated Yelp score is particularly powerful. (Hypothesis 1 above disproven)
- The number of reviews a place has on any platform (except when less than the Yelp reviews are <25, when Facebook is a better predictor, probably because Facebook has a larger sample size) is irrelevant to my rating. I had thought that super popular or unknown places were best avoided, but this turned out statistically false. (Hypothesis 2 above disproven)
- Facebook and Google both give restaurants a higher average score than Yelp (this is partially due to their extra degree of freedom.) This should be taken into account when viewing those sites.
- Google ratings were not statistically significant enough to have any correlation to my ratings. I will not use Google for restaurants in the future.