Tuesday, August 15, 2006

Claim Definition in Prediction Markets

Claim definition in Prediction Markets is hard to get right, but good claim descriptions make judging easier. An intrinsic part of the challenge is that the best claims are close calls until near the ending date. That means you want to choose a deadline and terms on which people disagree about the likely outcomes at the outset. You also want claims that will resolve themselves, and be easy to judge at the end; it's a benefit to everyone if the answer is clear when the claim closes. This is much easier to achieve with practice in any particular area--routine claims formats can be polished over time. So the easiest claims to write are for highly repetitive events like sporting events and elections. I expect the same will become true of predicting project completions and corporate quarterly performance targets, but currently those issues are being described, argued about and judged in private, so there's no shared repository of experience to draw on.

The fact that pay-outs are limited to the amount spent to purchase claims is integral to the institution of prediction markets. If market operators ever pay off both sides of a claim, that is likely to encourage investors to protest many more close calls. Refunding all investments would have a much less deleterious effect than paying off all parties. In disputed calls in which evidence may not be available immediately after an event occurs, it's an advantage to the market to be able to roll back betting that occurred after the event was retroactively determined to have occurred. (TradeSports has a clause in their rules (Contract Rules 1.8) that allows them to unwind trades occurring after one hour before the first press report of outcome. The rule applies to claims when this is explicitly stated beforehand. In the NK Missile case, not having received confirmation, there's no time to roll back to. If they had decided that the terms had been satisfied, they would have been able to roll back the post-launch betting.)

As long as claims are designed so the market operator doesn't take a position on the claim, the operator won't have any conflict of interest in deciding the claim. There are other good reasons for choosing an outside judge. The market operator may be pressured to change a judgment or to pay both sides in a disputed claim; if they are also the judge, this pressure is stronger. If the judgment is from an external party, it will at least be seen as independent. If all judgments are issued with the authority of the exchange, losing bettors will blame the exchange itself. I think identifying the judges and showing their track records is a service to the investors. There may not be many businesses willing to trust judging to outside parties, however. I fear they will feel the heat on close calls, and there will necessarily be close calls.

It's important to distinguish sources from judges. The judge makes the decision for the exchange about the outcome. In my view, claims should include a reference to preferred sources. For some subjects it's reasonable to rely on newspapers, for others peer-reviewed journals are more appropriate. Often a government agency has the responsibility to and a history of reporting on a particular class of outcome. If no source is specified, there will be disputes (at least when money is at stake) based on obscure media reports. It's better to make it clear what level of assurance is required. It's unfortunate, but sources sometimes go against their historical practices, so it's usually valuable to say how the outcome will be judged in the absence of an official statement. When sources are named, it's crucial to the exchange's reliability that they not be changed without prior announcement. (I don't fault TEN (TradeSports) for continuing to rely on the DoD in the recent contretemps over the North Korean missiles. They said the DoD would be their source, and some traders were relying on that statement. It would have been improper for TEN to act without a public statement from the DoD or a direct statement to TEN; a statement to a subscriber isn't something TEN should rely on for a judgment.)

Chris Masse has suggested that the intent and the wording of a claim should always coincide. But this isn't possible in practice, as is well known to anyone who has managed or participated in the process from start to finish. Claims that seem clear when written (remember that it's a goal to make the outcome hard to determine at the outset) often turn out to be unclear later. Over time, the meanings of terms mutate, and their application to specific questions is modified as events transpire.

Since we strive for close calls, the timing is also often sensitive. Some events aren't publicized immediately, so claims should specify whether the date of the event or the date of publication controls the outcome. (Often in FX, both are specified for science claims.)

We often look for externally judged claims; if there's a prize being awarded, that often makes a good claim. But the claims need to be careful to specify whether the underlying achievement controls or the awarding of the prize determines the outcome. What happens if the judging doesn't occur (the judging organization goes out of business or stops overseeing that type of event) or if the judge awards a consolation prize? (See the recent SENS challenge at Technology Review.) Sometimes the disappearance of the judge or the dropping of a contest is material; often the participants won't agree on how "the spirit of the claim" applies to these edge cases.

In general, the judge should pay attention first to the text of the claim, and second (in cases where this is accessible) to the discussion of the claim before the wording is finalized. On FX there is often plenty of context on what a claim means that doesn't make it into the claim. The conversation between sponsor, judge, and community can be very informative including what wording was intentionally not included in the claim. It's unfortunate that most of the high-profile markets have a closed conversation (if any) on claim wording, and then produce short claims with no discussion of edge cases.

If there's no visible discussion of the intent of the claim, the only thing investors have to go on is the plain text. Asking them to guess how the judge or exchange will interpret the intent is much less certain than assuring them that the wording will control. Everyone has a good argument for why their interpretation of the intent is the one that makes sense. There's no reason to let this be the basis for complaints and arguments.

TEN hasn't addressed this issue, but it seems clear to me that they consider their positive claims on current events to be the controlling language. The negative position is a shorthand description that the event didn't take place. When they cite a source and require confirmation, then an absence of confirmation clearly requires a "no" outcome. NewsFutures ensures that the two sides of a claim are duals; neither has precedence over the other in the way the interface presents them, but this gives them a reason to make the wording consistent. The arguments that TEN needed to confirm that the missiles weren't fired or didn't land in international waters don't seem reasonable to me. I'm reasonably sure some of the proponents haven't found this distinction between TEN's positive and negative claims. Maybe they'll add "or no confirmation is obtained" to their standard list of negative cases.


Anonymous said...

Hi Chris Hibbert,

Thanks for this great post.

You write:

"The arguments that [TradeSports / InTrade] needed to confirm that the missiles weren't fired or didn't land in international waters don't seem reasonable to me."

Would you mind elaborating? Does this mean that you'd favor a expiry to 100? (I asked somebody, and he said "no", so there's something I don't get right now.)

Thanks in advance for your clarification.

Chris Hibbert said...

According to the rules that TradeSports seems to be following, they needed confirmation of the positive outcome in order to issue a "yes" ruling. If they can't get confirmation of the positive result, I expect them to reward the negative position at the expiration.

I don't think I'd say that I "favor" this outcome, but that's what I'd "expect" given the way they seem to interpret their claims. I'm not happy that they were trapped by their own wording into waiting for confirmation from the DOD which they couldn't get to their satisfaction. But I do think they were right to stick to the description they gave.