## Wisdom of the Crowds at the Graduate School Christmas Party

In 1906 Francis Galton was at a country fair and there was a guess the ox competition. He took all 787 guesses and took the average. This was 1,197 pounds. The actual weight? 1.198 pounds! In effect the wisdom of the crowds gave a perfect answer. This was the start of The Wisdom of the Crowds.

Our Graduate School held its Christmas Party yesterday (18th Dec 2015) and they were kind enough to invite me. When I got there, I noticed that there was a jar of sweets, inviting people to guess how many sweets were in the jar. This reminded me of the story above.

When the competition ended, the person who had the closest guess would win the jar of sweets.

The jar held 149 sweets. The closest guess (by Oppong Kyekyeku) was 130; nineteen away – but good enough to win the prize.

When we looked at all the entries (see below), we found we had 22 entries, with an average of 152 (actually 152.409). That is just three away.

To be honest, with such a small sample size, I was surprised that the wisdom of the crowds (well small gathering) had beaten every other guess and had got within four of the right answer.

Francis Bacon would have been proud!

(This post also appeard on the University of Nottingham Research and Knowledge Exchange blog)

# Introduction

The 2015 UK General Election looks like being one of the closest, and hardest to predict, for many years. With 650 seats being contested, one party needs to win more than half the seats (326) to be able to form a government. Most, if not all, polls are predicting a hung parliament, with the likelihood that the UK will have another coalition government, though what form that will take is open to much debate.

It is not difficult to find predictions for the election result. They tend to fall into two categories; the percentage share of the vote or the number of seats that will be won by each party. Of most interest is the number of seats that will be won by each party, as this is what determines the formation of the next government.

# Wisdom of the Crowds

In 1907, Francis Galton reported in Nature an event that had taken place at a country fair, where around 800 people were asked to guess the weight of an ox. The average guess was 1,197 pounds. The actual weight was 1,198 pounds, which is close to the average guess to be considered just about spot on. Importantly, many of the people who participated could be considered experts, such as farmers and butchers, but many people were far from experts – just being people attending the fair. Also, importantly, not a single person guessed the correct weight and only one person guessed 1,197 and two people guessed 1,199.

This concept of the Wisdom of the Crowds was popularised in a 2004 book by James Surowiecki, arguing that the opinion of a large number of people will do better than the judgement of a few experts.

# 2010 General Election

Wisdom of the Crowds was used to predict the 2010 general election. Martin Boon, of ICM Research, showed that “that the Wisdom of Crowds approach at the 2010 general election would have produced the most accurate final pre-election prediction.

# Henretty and Jennings

Chris Henretty and Will Jennings have used the Wisdom of Crowds to predict the number of seats for each major party in the 2015 General Election. They surveyed 2,338 people, with 537 responding. They asked two questions, one about the percentage share of the votes and one about the number of seats for the major parties. Their report (published on 03 Mar 2015) gives the following predicted seats.

 Party Seats Con 278.4 Lab 282.3 Lib Dem 24.8 SNP 28.7 Greens 1.9 Plaid Cymru 3.3 UKIP 6.6 Others 13.4

# Our Data

Drawing inspiration from this study, we utilise other predictions, to see how it compares with the study of Henretty and Jennings. Our study looks at 24 different predictions, aggregating them to produce our predictions.

Our data is drawn from a variety of sources.

• The Henretty and Jennings study is used, recognising that it incoporates over 500 individual predictions.
• A recent BBC Panorama program asked Nate Silver for his predictions. Silver is an American statistician who has successfully predicted the outcome of the last two US presidential elections.
• Data was taken from several spread betting firms, taking the middle of the spread as their prediction.
• The London School of Economics asked a number of election forecasters at a conference they held on the 27 Mar 2015 for their predictions. These have been incorporated into our predictions below.
• Some newspapers publish predictions, and these are used in our model.
• Some on-line prediction web sites were used.

We decided against using the 2010 results, or the current parliamentary standings, although we show the predictions using these two additional pieces of data, just for completeness.

One issue that has to be considered is missing data. Predictors do not always provide predictions for all the parties, but provide an aggregated figure in Others for some of the parties. Some predictors also exclude Northern Ireland so they only supply 632 predictions, rather than the full 650. We work around this as best we can.

In order to calculate our predictions, we averaged all the polls under consideration. We normalise the figures for each party so that the total number of seats adds up to 650.

# Predictions

Our predictions are shown in the table below. The Excl. 2010 column shows the predictions when the 2010 results or the current parliamentary standings are not taken into account. The Incl. 2010 results are shown just for comparison.

 Excl. 2010 Incl. 2010 Con 279 281 Lab 274 272 Lib Dem 24 27 SNP 46 42 Greens 1 1 Plaid Cymru 3 3 UKIP 3 3 Others 20 21

The two sets of figures are reasonably close with the obvious differences being the higher prediction of the SNP and the lower prediction of the Lib Dems, which reflects the (potential) changing fortunes of the two parties since the last general election.

# Concluding Remarks

I guess, not surprisingly, we are also predicting a hund parliament, with the Conservatives having a slight lead over Labour. If our predictions are accurate, a coalition with the SNP would give a combined total of 325 seats – not quite the 326 needed to give an overall majority. Now that would be interesting!

## Time to switch to Java for a football prediction project

I have decided that it was time to switch to Java for a football prediction project that I have been planning for some time.

I want to do the project justice so I thought I would start from the most basic decision. What programming language should I use?

# The Problem with C++

For the past 20 years, I have been using C++ (before that I was using all sorts of mainframe languages). I can do what I need to do with C++, but I have never been entirely happy with it.

As a language, I quite like it, but it is all the stuff that goes with it that has always been frustrating. And most of that stuff is around Visual Studio (VS), MFC (Microsoft Foundation Classes) and Templates. Don’t get me wrong, this is not a Microsoft bashing exercise. I know of many people who use VS and are very happy with it, develop great applications and know how to use the tools that are available.

But I have never really got my head around it. I run into all sorts of problems with namespaces, linking errors, deciding whether to use MFC (Micosoft Foundation Classes), or not – and then regretting it, whether to use the classes available in VS so that I could port my code to another compiler should I wish to do so; and the list goes on.

The end result is that I have never really developed as a C++ programmer. I can do all the usual stuff (classes, inheritance, operator overloading, polymorphism) but I have never been able to get to grips with Windows (Graphical User Interface – GUI) programming and so have always stuck with a Command Line Interface (CLI).

To be honest, using a CLI has served me well and I have managed to churn out some nice programs. But for my football prediction project, I really want to develop some sort of GUI. The question I asked is, should I have another go at getting to grips with MFC, Windows programming and all that is need to get an application developed in VS to display a Window on the screen, or do I change to something else, in the hope that I will find it a little easier to understand?

# So why Java?

After a lot of soul searching, and Googling, I decided that it was time to switch to Java. I have written a couple of programs in Java before, but nothing too much beyond “Hello World”.

But is like C (syntax wise), and like C++ (class wise), it is well supported, many of my academic colleagues use it, it is platform independent and, I hope, that GUI programming is a little easier than VS.

Of course, making the decision to use Java just raises a whole load more questions. What IDE should I use (I have chosen Eclipse), how big is the learning curve, can I easily access Excel files, can I (should I) use MySQL, what Java tools are available to support developing a a GUI etc.

All these questions will have to wait. For now, I have to learn the basics of Java and get my head around the Eclipse IDE (Intergrated Development Environment).

Wish me luck.

## Football fixture forecasting. Are you any good?

Football fixture forecasting is something I have expressed an interest in recently. Actually, this blog contains a forecasting category, that you might be interested in.

In my last post, I mentioned a crowdfunding project that I am trying to get off the ground. This prroject aims to investigate football fixture forecasting, utilising methodologies such as Artificial Neural Networks and algorithms based on Darwin’s prnciples of natural evolution (survival of the fittest).

Doing research for this project, I looked around a few football forums and found that there is a brisk business in football forecasting competitions (usually just for fun).

For example:

• FootballForums.net have a Football Forum thread, which can be accessed here.
• Total Football Forum also has a prediction thread.

I mention these now, as they might be of interest as the new season approaches.

Of course, one of football’s most well known pundits (Mark Lawrenson) makes predictions every week (on Radio Five Live I believe) and these are tracked at various web sites including MyFootballFacts.com. This site does a great job of showing the actual league tables and what would have happended if all Lawro’s predictions had come true. It’s interesting to see the fairly signisficant differences league table based on actual resuts. I guess it is no that surprising that there are large differences as, if Lawro, was an excellent preditor all the time, the bookies would be bankrupt.

I think I am also right in saying that Lawro tries to predict the actual score. This has got to be difficult. If I ever get my project off the ground, I think I’ll just try to predict whether the home team will Win, Lose or Draw.

## Using ELO ratings for match result prediction in association football

I occasionally comment on a scientific paper that is of interest to me. This time, it was:

Lars Magnus Hvattuma and Halvard Arntzenb (20010) Using ELO ratings for match result prediction in association football, International Journal of Forecasting, 26(3), 460-470 (doi).

This paper falls into the broader categories of Football, Forecasting and Sport, which I have blogged about in various ways.

ELO ratings were originally designed for Chess. In brief, if you play an opponent who is lot weaker than you and you lose, your rating would reduce a lot more than if you were beaten by a player who is only slightly weaker than you. Similarly, if you beat a player who is a lot stronger than you, your rating will increase a lot more than if you beat a player who is only slightly stronger than you. It is usually (if not always) a zero sum game so that what ever points you gain (or lose) the same number of points is lost (or won) by your opponent. There are a number of variations as to how ratings are updated but once you have a rating you can compare yourself to another player (or team) to decide who is the better player.

The idea behind ELO ratings have now been used for a number of sports. Relevant to this blog post, it has been applied to football. In fact, there is a web site that maintains a list of ELO ratings for world football. At the time of writing, the latest match played (on the 15th Aug 2012) was a friendly between Albania and Moldova. The score was 0-0 and the teams gained 5 points (Moldova) and lost 5 points (Albania) respectively. Their points are now 1461 (Albania) and 1376 (Moldova). The fact that Albania were higher ranked meant they lost points against a lower ranked team when they could only manage a draw. They also had home advantage, though I am not sure that this is a factor in this model. Looking at the web page, I don’t think so.

In the paper, which is the subject of this post, the authors define two ELO models to try and rate teams. The first uses a basic ELO to work out ratings and then uses the difference between the two ratings as a covariate (a variable that could be a predictor of the study under investigation) in a regression model. This seems reasonable, as the the ELO rating is a measure of the current strength of the team. A second ELO model is also used which also takes into account the margin of victory. That is, a 3-0 win is better than a 2-1 win.

To compare the two ELO models the authors define another six prediction methods. The first (which they call UNI) simply chooses an outcome at random. The second (FRQ) uses the frequency of home wins, draws and away wins to make its prediction. For example, over a range of matches if two thirds result in a home win, then FRQ would predict that two thirds of the matches it predicts are also predicted as home wins. It is noted that UNI and FRQ are quite naive prediction methods. Two other benchmarks are derived from a paper by John Goddard (Goddard J. (2005) Regression models for forecasting goals and match results in association football, International Journal of Forecasting, 21(2), 331-340 (doi)). The fifth comparative model is based on bookmakers odds, by taking the average of a set of odds from various bookmakers. The final model is almost the same as the fifth, but instead of using the average odds, the maximum odds are used.

The paper also uses various staking strategies. A fixed bet of one unit, a stake in proportion to the odds, to return one unit and stakes based on the Kelly system which is an optimal staking system if you know the exact probabilities.

The various prediction models and staking plans were tested on eight seasons of data (2000-2001 to 2007-2008). Previous seasons data was used, to train the various models.

So, what were the results? The two ELO models were found to be significantly worse than two of the other six models (those based on bookmaker odds), but the ELO models were better than the other four models. The conclusion is that this is a promising result but perhaps more covariates are required to predict even more accurately.

## Odds-setters as forecasters: The case of English Football

I sometimes comment on a scientific paper that has caught my eye. This time, it was (comments on other papers are also available):

David Forrest, John Goddard and Robert Simmons (2005) Odds-setters as forecasters: The case of English football, International Journal of Forecasting, 21(3), 551-564 (doi).

This paper falls into the broader categories of Football, Forecasting and Sport, which I have blogged about in various ways.

The premise of this paper is that the odds set by bookmakers are in fact predictions by experts. Not a bad assumption to make given that the odds that are set equates to real money coming in, or going out. It was also motivated by the fact that a previous study had concluded that a statistical model performed better than newspaper forecasters.

But is it true that bookmakers odds are good predictors?

To answer this question, the authors took almost 10,000 games, between 1998 and 2003, and compared the bookmaker’s predictions with forecasts made by a statistical model. The model they use is fairly complex (in my opinion) although understandable (and, importantly, reproducible) to the interested, and mathematically competent,  reader The model also looks back over fifteen seasons so if you wanted to implement it, there is quite a lot of data that needs to be collected.

The comparisons are quite interesting between the different forecasting methods. Over the five seasons considered, 1998-1999 (1944 matches), 1999-2000 (1946 matches), 2000-2001 (1946 matches), 2001-2002 (1946 matches) and 2002-2003 (1945 matches) it showed that the statistical model performed better in the earlier seasons, but by the end of the period under investigation the odds-setters were performing better. Looking at just the odds-setters they did worse in the two seasons at the ends of the period under investigation (that is, 1998-1999 and 2002-2003). This is presumed to be there being more noise in the match results.

If you look at the data another way, there is little to choose from the home, draw and away predictions between the two prediction methods.

A follow up comparison attempts to estimate if the odds-setters are capturing all the information contained in the model (if you look at the paper, the model contains many elements such as whether a team is still in the FA Cup, if the match is important to a particular team, attendance information etc.) All the odds-setters have is one figure (the odds) and that has to incorporate everything in order to provide a prediction. The figures presented suggest that, over time, the odds-setters get better at incorporating information into their odds. It is shown that incorporating the bookmakers odds into the statistical model gives superior performance, suggesting that the bookmakers are privy to information that is not included in the benchmark model. This was not the case in the earlier paper, where this test failed when considering tipsters.

Next the paper investigates if indiscriminate betting (placing bets,  using identical stakes, on any outcome) on matches produces better or worse returns than if you follow the benchmark model. Indiscriminate betting, maybe not surprisingly, produced a loss at a rate that a bookmaker might expect in the long term (termed the over-round), around 12%. By comparison if you place a bet on the outcome with the highest expected return, as indicated by the benchmark model, then you would do much better. Over the five seasons you would have only lost -0.2%.

A possible downside is next considered. The benchmark model is the same for every season, whereas the odds-setters are (probably) learning each season or, perhaps, using different people/methods to set the odds. Actually, the model did not give superior predictions by refining it as the simulation progressed from season to season, or even as the season progressed.

One of the conclusions of the paper is that experts (in this case odds-setters) are actually quite good at predicting the outcome of matches. This is attributed to a variety of factors including the fact that their livelihood depends on it and also the sector is much more competitive (due to technology and tax reforms) that have required them to become better to remain competitive.

My conclusion? Use the model, and incorporate bookmakers odds – that gives you the best predictions. But it is quite a lot of data to collect. Alternatively, develop your own predictions methodology and see how it performs.

## A compound framework for sports results prediction: A football case study

The latest paper that caught my attention, that I thought I would comment on is (other publications I have commented are can be seen here).

Byungho Min, Jinhyuck Kim, Chongyoun Choe, Hyeonsang Eom and R.I. (Bob) McKay (2008) A compound framework for sports results prediction: A football case study, Knowledge Based Systems, 21(7), 551-562 (doi).

You might be able to download a copy of the paper from here. Note that this link may not be permanent and it may not be an exact copy of the paper that was published (although it does look like it).

The paper presents a framework which is designed to predict the outcome of football matches. They call their system FRES (Football Result Expert System).

The authors note that most previous research focuses on a single factor when predicting the outcome of a football match, and the main factor that is used usually the score data. Even when other factors are taken into account, the score tends to still dominate the prediction process.

Within FRES, two machine learning methodologies are utilised, a rule-based system and Bayesian networks. The paper describes how they are used within FRES in enough detail to allow readers to produce (as all good scientific papers should do) the system.

FRES is tested on the 2002 World Cup tournament. Most football prediction systems are tested on league competitions, where teams (typically) play a double round robin tournament. Testing their approach on a the 2002 World Cup means that the system cannot easily be compared to other systems. Where previous approaches have been tested on other tournaments (for example, previous World Cups) not all the data was available to enable FRES to make those predictions. In the words of the authors, “In the case of the few works which predict a tournament such as the World Cup, the available evaluation was conducted with old data, such as the World Cup 1994, 1998, which would unfairly hobble FRES, since some of the data it relies on are not available for these earlier tournaments.

Although not a scientific term (at least not one I am familiar with!), I do like the term unfairly hobble.

In order to provide some sort of comparison with FRES, the paper implements two other systems, a historic predictor and a discounted historic predictor.

FRES was able to predict six countries out of the top eight in the tournament, The other predictors were able to predict five. Moreover, various statistical tests are conducted which confirms that FRES is statistically better than the other two methods.

One thing I like about the FRES system is that is has a lookahead mechanism. Based on this, England does not rate very highly as, due to the draw, there is a high probability that they will meet Brazil in the quarter finals. Turkey, on the other hand are rated more highly due to the perceived easier draw.

It would be useful to have FRES tested on league competitions, so that better comparisons could be made with more prediction systems that have been reported in the scientific literature. Perhaps the authors are working on that now? It would, for example, be interesting to see if it beats a random method, or a method which always predicts a home win (as the authors did in the paper I discussed a few days ago).

## Sports Forecasting: A Comparison of the Forecast Accuracy of Prediction Markets, Betting Odds and Tipsters

In some of my posts I comment on a scientific paper that has caught my eye. There is no particular reason for the papers that I choose, they are just of interest to me. In this post, the paper that caught my eye was (comments on other papers can be seen here).

Martin Spann and Bernd Skiera (2009) Sports forecasting: a comparison of the forecast accuracy of prediction markets, betting odds and tipsters, Journal of Forecasting, 28(1), 55-72 (doi).

This paper looks at three different prediction methods, and assesses their effectiveness in predicting the outcome of premier league matches from the German football league. The three methods that are investigated are prediction markets, tipsters and betting odds.

Prediction Markets are based on various people taking a stance on the same event and willing to back their hunch by paying (or collecting) money should their hunch be wrong (or right). Given that a number of people are taking a stance on the same event, it can be seen as a predictive model of the event.

Tipsters are (or should be) the views of experts who publish their predictions in newspapers, on web sites etc. The advice from tipsters is often based on their expertise, rather than applying some system or formal model. The paper (citing Forrest and Simmons, 2000 as its source) says that tipsters can often beat a random selection method, but does worse than simply choosing a home win every time. It also cites Andersson et al., saying that soccer experts often do worse that people who are less well informed about the game.

Betting Odds, in previous work, have found to be a good forecasting method (not surprising I suppose seeing as the bookmakers rely on setting the correct prices to make their living). Of course, the bookmakers can change their odds but when they publish fixed odds (on say a special betting coupon), this can be seen a prediction of the match outcome.

The games that are forecast in this paper are those from the German premier league from three seasons (1999-2000, 2000-2001 and 2001-2002).  The number of games predicted by each method varied (Prediction Markets and Betting Odds = 837, Tipsters = 721 and Prediction Markets, Betting Odds and Tipsters = 678). The number of predictions for each method varied simply due to the data that was available and where the number of games is between two, or three, methods, this is the intersection of the games that that method was able to predict.

To evaluate each method, the authors calculate the percentage of correct predictions. They also calculate the root mean squared error, as well as the amount of money that each method would have won (three figures are given, a 25% fee, a 12% fee and no fee). Comparisons are also made with a random selection policy as well as a naive selection policy, which simply assume a home win.

So, what did the authors find? Over the 837 games, the prediction market and betting odds were able to predict 52.69% and 52.93% of games respectively. If there was no fee this would have returned a profit of 12.30% and 11.92% respectively. The naive model (pick home wins) predicted 50.42% correctly and returned a profit of 11.79% The random method only managed to predict 37.98% of games correctly.

If we look at the 678 games that all three methods could predict, then the percentages of correct predictions were 54.28% (prediction market), 53.69% (betting odds), 42.63% (tipsters), 50.88% (naive model) and 37.98% (random). The returns (assuming no fee) were 16.20% (prediction market), 13.49% (betting odds), -0.19% (tipsters) and 12.44% (naive model).

I’m not sure why, but profit information is not given for the random model but it would almost certainly result in a loss.

A further test is also carried out. Only games where methods agree on the selection are bet upon. For example, of the 678 games, there are 380 games where the three methods agree on the result. If we only bet on those games, we get a correct prediction percentage of 57.11%, higher than any of the methods used in isolation, and betting on every game. The profit return would be 13.86% (no fee), 1.66% (12% fee) and -8.72% (25% fee).

The authors conclude that the prediction market and the betting odds provide the best indication of the outcome. They agree with previous work that tipsters are generally quite poor at prediction.

## Can Forecasters Forecast Successfully?: Evidence from UK Betting Markets

I am occasionally blog on a paper that is of interest. Well, of interest to me. The latest paper to catch my eye is (other papers I have commented on can be seen here).

Leighton Vaughan Williams (2000) Can Forecasters Forecast Successfully?: Evidence from UK Betting Markets, Journal of Forecasting, 19(6), 505-513 (doi).

The reason that this paper was of interest is because I was reading Leighton’s book (Betting to Win : A Professional Guide to Profitable Betting, High Stakes Publishing, 2002, ISBN: 1-84344-015-6) and this paper was mentioned. Many (many, many) years ago, long before I was an academic, I went through a phase where I collected all sort of horse racing systems. My idea was to test them all out to see if any of them worked. I never actually placed a bet and I never really tested any systems as they generally involved too much time to collect all the data, process the data etc. etc.

Since then I have still thought that it would be interesting to look at various horse racing systems to see if they worked.

This is what this paper does. Unlike my idea though, it takes tips from services that you subscribe to, either by paying money or by contacting them using a premium rate phone service. This seems a lot more sensible, rather than having to enter all the data yourself.

This paper looks at the performance of tipping services, with the analysis being carried out in 1995. Five services were compared. Four of these were subscription based. That is a fee is paid, and you gets tips at various times. In 1995, these services cost at least 99 GBP per month, which seems a lot to me now, let alone in 1995. The other service was a premium rate phone number, where you phone up to receive the tip and the costs of the phone call effectively covers the cost of the service. These five services were chosen as they were amongst the top tipping services as assessed by the Racing Information Database (I have tried to google this and am not sure that it is in existence anymore, but would be willing to be corrected, and update this post to provide a link).

The paper goes through each of the tipping services and evaluates how many tips were provided (and over what period – some, for example, were analysed over three months, others over six months – I think the period was probably chosen to ensure that a sufficient number of tips were analysed as not all services provide tips at the same interval), any conditions associated with the service (for example, only bet if a certain price is available), the profit (or loss) from investing in that service etc.

The good news is that all the tipping services produced a pre-tax profit when used with the relevant staking/price plans. Leighton also makes the point that none of these profits could be said to be significant. It was also interesting to note that increased profits could have been achieved if some of the lesser supported tips were ignored. Of course, this would be a hindsight examination and the obvious question would be, when in play, what tips do you ignore, and what ones do you actively bet upon? There is also evidence that you should use a variable staking plan, rather than a flat stakes method.

If you use a tipping service, there are also other factors to take into account. There is an upfront investment (which you may never recover). Unlike an academic study, you will probably only choose one and which one do you choose? There is also (as pointed out by Leighton, in chapter 20 of his book) the fact that you have to take what the tipping services advertise with a pinch of salt. As an example, a service might only say bet if you can get a price of 4-1 or better. What happens if that price is almost impossible (or even actually impossible) to get, will the service still include that in their results if the horse should win? And what if there was a price available (even for a few seconds) at 6-1, would the service return that as the price you could have got even though, unless you were very quick, or very lucky, you would have struggled to get on at 6-1.

It is twelve years since the paper was published, and seventeen years since the analysis was carried out and things have moved on. Tipping services (I suspect) come and go, technology has moved on, the tax regime has changed and there are now many other ways to bet which were not so predominant at the time. I am thinking specifically of spread betting and betting exchanges. These have, undoubtedly, made a big difference to the industry.

I am not up to date with the scientific literature in the area of sports forecasting and I suspect that there are many papers out there that provide various comparisons and analysis. If you know of any, or know of a good review paper in this area, I would be very interested if you could post a comment giving the reference.

I would also be interested in hearing from any professional tipping services (no matter what sport, but UK based as I don’t claim to understand American sports or markets) who wish to subject their service to scientific analysis. Note, this is not an open invite to advertise your service on this blog. I get enough spam as it is (and moderate it out) and I don’t want the comments box filled up with lightly disguised adverts to various web sites that claim to make millionaires from people who subscribe. But serious enquiries are welcomed.