Saturday, December 19, 2009

Malclm Gladwell: Outliers

Malcolm Gladwell's Outliers was an engaging read, but a disappointing argument. Gladwell presents a series of separate incidents, each painted in a fair amount of detail. He doesn't focus on the common threads, and sometimes it's hard to see how the stories are pulling in the same direction, since he fills in the theme with a really light hand.

Gladwell presents a number of success stories and a few stories of failure in order to show that luck and circumstances contribute substantially to both. The preface shows us a small Pennsylvania town whose residents mostly came from a single small Italian town. Both towns show unusual longevity, and Gladwell points to a low-pressure lifestyle with lots of community contact as the reason. The first chapter shows that the top Canadian hockey players (and around the world) all have birthdays that are clustered just after January 1(toward the beginning of the year (January 1 is the cutoff date for entry into the youngest organized leagues). This means that each year, this cohort includes the biggest kids, who get the most attention and training, and the investment compounds over the years.

Lewis Terman's gifted kids are presented to show that intelligence isn't a sure precursor for success. There were very few successes out of the class of extremely bright kids that Terman followed for decades.

Gladwell then presents details about various successes who happened to already have the right preparation when their skills came to be valued by the marketplace (Bill Gates and Bill Joy, as well as Jewish lawyers who'd been handling corporate proxy fights when they'd been out of favor by the white shoe firms in New York). Gladwell argues that 10,000 hours of focused practice is necessary to turn someone into the kind of expert how can take advantage of situations like this. He seems to want us to believe that luck determines which of the available experts will ultimately succeed, but he fails to establish that 10,000 hours is an important benchmark. It's certainly plausible that in order to be considered a pioneer, you have to have picked up the expertise before it was obvious that there was a field available to excel in.

Gladwell tries to put the Beatles in the category of people who picked up the 10,000 hours of experience, and then won because the time was right. The evidence he shows makes it look like they worked very hard in Germany for a few summers before they achieved their success, but it doesn't look like 10,000 hours. And it's not obvious what wave of change they rode to gain their success, comparable to the opportunities available to Gates, Joy, and the New York lawyers who were ready for the wave of corporate lawsuits in the 1970's.

Along the way, we also get a few stories of surprising failures, uniformly due to socialization. Some cultures are poorly suited to producing successes in particular fields. He focuses on a particular bad period for Korean Air Lines, which the long term investigations eventually laid at the feet of the extreme deference due to pilots (and high status individuals in general) in Korean society. Gladwell dissects several crashes to show that even when the co-pilot could tell the plane was in trouble, Korean social mores prevented him from saying anything directly to the pilot. Eventually, the international community convinced Korean aviation to change their training to ensure that cockpits were much more egalitarian, so communication didn't have this fatal flaw. There are a couple of other stories of individuals from dysfunctional societies, or of dysfunctional societies themselves.

The book closes with the story of KIPP, a free open-enrollment school that has shown that pretty much all kids can be successful and prepared for college if the social environment is appropriate. It takes a culture that embraces hard work (which Gladwell also emphasized in the previous chapter on Asian farmers), but doesn't require selective admissions.

Gladwell tells a good story, but he didn't spend much time stitching it together. I had to review the whole array of pieces in order to see how they fit together. Before going to that extra effort, I had a different impression of the intended moral. The high-profile success stories (the Beatles, Bill Joy, Bill Gates, and Joe Flom the Jewish lawyer) were the most vivid, so I remembered it as a story of how hard work makes you eligible for success, but requires the addition of fortuitous timing in order to win the brass ring. Once I brought in the pro athletes, Terman's gifted kids, and KIPP, I could see that the point must have had something to do with those who just missed greatness, too. The commonality that Gladwell wants us to find is that success is mostly a matter of luck and circumstance. And he we wants us to know that luck and circumstance can also work against us. (That's how the Korean pilots fit in and southern culture's deleterious effects on socialization.)

So he wants us to believe that community and context matter more than ability. Hard work apparently gives you a chance at success, but the chance is out of your control. All you can do is pick something you care about and work hard at it. If you're lucky things might turn out well for you, but they probably won't.

This is a pretty discouraging story if you stop there. While it may be a reasonable story about which people get to be the big winners, it's misleading as a guide to living a successful life. Success at the level of the characters he describes probably is mostly a matter of luck, but ordinary success and modest achievement is much more attainable, as the KIPP example shows. There aren't many fields like pro hockey or pro baseball in which the winners are picked early, and there's no reasonable chance to catch up if you miss the initial cut-off. But Gladwell doesn't provide any hints about that and he makes this story harder to follow than it needs to be.

Friday, December 18, 2009

Health Care is not like Agriculture

Atul Gawande is an excellent writer and has a lot to say about the practice of medicine. His columns in the New Yorker have been very educational, and his observations about what some doctors do to make medicine safer and more effective are really incisive. His June article on the difference in costs of providing approximately equivalent services across the country caught a lot of people's attention. He has continued to explore issues related to health care reform, but I'm afraid his most recent article misses the mark.

In December's New Yorker, Dr. Gawande describes how the US Department of Agriculture spread information about the practice of farming by hiring extension agents who convinced local farmers to try out pilot projects to demonstrate the benefits of scientific farming, and where they had successes, recruiting additional farmers by the power of example. Gawande seems to think that most of the improvement in American farm productivity is due to the work of the USDA. He then proposes that this approach is hidden in the draft health care bills, and that it is likely to work as well for health care as it did for agriculture.

I'm very doubtful, and the reason is that everything the USDA did was voluntary. That's why they got volunteers to run pilot projects--they needed to show that the ideas worked, and they couldn't force anyone to go along with it. And voluntarism is one thing you can be sure will be missing from the pilot projects (disguised as targetted programs for particular districts) included in the health care bill. The ideas may well be tried out in one local area at a time, but I'd bet each pilot will be mandatory for some set of doctors, clinics, or insurers. The kinds of approaches that could be made to work if practitioners were allowed to try them are very different from the kinds that might work when practitioners are forced to follow them. The big problem in health care is that there's already way too much regulation. Adding more layers of regulation, and more requirements, won't give any of the participants in this industry any incentive to improve results or lower prices.

A solution would have to include finding ways to make more of medicine look like lasek surgery, or any other competitive industry, but nothing like that is on the table. The Cato institute has been pushing for opening up insterstate competition in various ways, but I don't get the impression that anyone with the ability to influence outcomes is listening to them.

Friday, December 11, 2009

C. J. Cherryh: Forty Thousand in Gehenna

C. J. Cherryh's Forty Thousand in Gehenna (1983) is a classic, and shows Cherryh's mastery of the presentation of alien minds. In this case, she makes the non-humans even more alien by having humans become incomprehensible to their own kind in the process of living and working with them.

The story takes place on the border between Alliance and Union space. In a political move, the Union sets up a colony on a remote world, and then fails to resupply it, leaving the colonists on their own. The colonists discover that the planet is inhabited, something the surveys failed to notice. The Caliban (various species of lizard, from smaller than a dog to as large as a brontosaurus) don't seem to be intelligent, though it's hard for the under-supplied colonists to control them and keep them out of the settlement.

The settlers start out as a mix of a few thousand natural-born humans and the titular 40,000 Azi (programmed humans). The plan of the settlement is to remove the programming once the settlement is set up, and allow the Azi to marry, raise families, and farm the land. The first generation of Azi have a hard time adapting to their freedom, and don't have any experience of family, so they don't do a good job with the next generation. At first arrival, the Azi vastly outnumber the free humans, so the resulting culture is a result of the natural forces that arise from the mixing of untutored second generation Azi and caliban than anything the Union planners might have intended.

Two generations later, a resupply expedition arrives, and tries to figure out how to deal with the cultural mix of Humans and Caliban. The expedition's leaders are slow to realize that the caliban represent a separate intelligence from the feral colonists. Eventually, anthropologists learn enough from the divergent societies (the colonists have split into warring factions) to understand that there are two alien groups (diverged humans and the Caliban who have adapted to a human presence) to be integrated into the galactic civilization.

I've read many stories set in Cherryh's Alliance-Union universe, and this is probably the only one in which Azis play a prominent role as individual characters. Normally they're present to show how a high-tech society would hold slaves. They're treated relatively well physically, but have their mental lives completely controlled. In this story, they're prepped for the mission with hints and build up that they will be learning to live on their own and that this is a great honor and an important mission. The latter is part of their standard indoctrination, so while they believe it, they don't assign it any particular significance. There's a little bit of hinting that they are excited about the opportunity to be free individuals, but as it turns out, their foreboding about having to manage their own affairs is more to the point. Shortly after the colony is founded, the resupply mission fails to arrive, and the technology that was used to give them reassuranc and training starts to break down. Since additional tools and technical assistance would also have arrived at the same time, the Azi and the other colonists experience their new freedom as part of a package deal with the gradual decay of their technology base. It's too small a colony to be self sufficient in maintaining the technology, though they are capable of feeding themselves as long as everyone works the land. In the end, what could have been an interesting story about discovering how to live a self-directed life is side-steped because the manumission happens in conjunction with a general breakdown in the social order.

It's not surprising that the children of the colonists, growing up in an impoverished settlement, surrounded by nearly incomprehensible but strangely communicative alien beasts grow up estranged from the previous generation. Few of the elders know enough about survival skills in farming or exploration to be of much help, their myths are suited to a much more technological society, and the vast majority of the older generation learned skills as they needed them from the tapes they were fed along with the programming that kept them docile. Given this, it's not surprising that their children charted a new course, but it is interesting to see how Cherryh presents their divergence, and the fumbling steps the envoys from civilization take in their attempts to control and understand them. Cherryh also does a good job of showing how bureaucracy and politics interfere in the task.

I apparently started reading this once before, since I see that Google's cached summary of a 2005 review I wrote has it in my "Currently Reading" list. I'm pretty sure I didn't get very far, because the story was all new to me this time. It's also been on my list for a while, but it took me a while to find a copy. (And then to find another.)

Sunday, November 01, 2009

Neal Gershenfeld: Programming Bits and Atoms

I followed a pointer from the Long Now Blog to The Perimeter Institute's Quantum to Cosmos Festival, and found some interesting talks. I enjoyed listening to Lee Smolin talking to Neal Stephenson and Jaron Lanier about how science informs and learns from fiction. But I had a much stronger reaction to Neil Gershenfeld's talk on what he's been doing at the Center for Bits and Atoms. It involves multi-scale parallel computronium, and fabricators in the hands of kids all over the world who are vaulting past us in understanding what it means to build stuff with embedded intelligence. A fascinating talk.

Thursday, October 22, 2009

Alex Tabarrok, Provocative Lecture @SJSU

A few years ago, I found a pointer to the Provocative Lecture series at San Jose State University, which had had several very interesting speakers, whom I was sorry I had missed. I couldn't find a stable place at SJSU where announcements were posted, so I ended up creating a google alert that looks for SJSU and "Provocative Lecture". It has fired periodically over the years, but it has always been a false alarm to date. More often than not it was for a review of a lecture that I had never seen announced.

Well I finally saw an announcement for a talk that hasn't happened yet. Next week, Alex Tabarrok, co-blogger on Marginal Revolution, will talk about wether the FDA is helping or hurting. October 27, 5:15, Morris Daily Auditorium.

And I still haven't seen an announcement from SJSU. The economics department's web page for the series gives details about talks from 2007 and earlier. This announcement came from a posting for the Santa Clara County Libertarian Party.

Saturday, October 17, 2009

Brian Francis Slattery: Liberation

Brian Francis Slattery's Liberation (with a 16-word subtitle) seemed to me to be closer to mainstream fiction than to the fantasy it's packaged as or to science fiction. It was still a fun read, but quite bizarre. The prose was quite lyrical and the timing mesmerizing. The story itself was weird, disjointed, and acausal, but it drew me in none-the-less.

The story replays the adventures and reunion of a dispersed band of mythic super-powered robbers and avengers after the apocalyptic collapse of the USA. The country has dissolved into fragments which are each ruled by whatever warlord can hold power (if anyone can). Some sections of the country don't hold together, and there are only local communities. Communication among the parts is slapdash, and it takes luck to get from place to place. One kind of luck is to happen on Dr. San Diego's apparently magic bus, which is drug infused and ferries people to and from unreachable places.

Anyway, the slick six have parted ways, but they'll need to regroup to defeat the forces of evil, slavery, and tyranny. They mostly do, though it requires a bunch of peripatetic adventuring to accomplish, and a few superhuman feats. I'm not sure there's much more that's coherent to say about this, but the language definitely kept my interest, even if the plot summary is mostly superfluous.

Friday, October 16, 2009

Gregory Cochran and Henry Harpending: The 10,000 Year Explosion

Gregory Cochran and Henry Harpending's The 10,000 Year Explosion talks about how humans have evolved over the period since we've been relatively civilized. They explicitly want to challenge the (sometimes vociferously propounded, but seldom cogently defended) notions that humanity hasn't evolved significantly since homo sapiens emerged, and that there's no significant genetic difference among populations in different places. They open Chapter 1 with quotes from Stephen Jay Gould and Ernst Mayr to demonstrate that they aren't fighting a straw man.

Mostly, this notion seems to be held in order to defend a liberal notion of equality, as if we could only defend equal treatment if we all have equal abilities, endowments, and if none of our observable differences are innate. If this notion isn't supportable, we'll have to be clearer that equal treatment is right for other reasons, and perhaps we'll have to be articulate about what those other reasons are. But that's an argument that Cochran and Harpending leave for someone else.

Cochran and Harpending's argument here is that there are quite a few differences between different modern populations, and that many are clearly genetic in origin. They first go to some pains to show that genetic changes can easily arise in this kind of time period. Particular examples are dogs, which have evolved all their modern variety since separating from wolves only about 15000 years ago, and domesticated plants which have changed enormously since the end of the last ice age 11,500 years ago. There are particular changes in humans that are also clearly of recent origin including skin color, eye color, lactose tolerance, and resistance to various diseases, all of which can be shown to be related to geography and to react to evolutionary pressures on much shorter time scales.

With that as background they make a couple of (they expect) radical arguments, and explain a few things that seemed puzzling before. Their first radical argument is that modern humans probably interbred with Neanderthals in Europe, and that therefore, the populations that left Africa probably got a significant contribution from them. I thought they did a reasonable job of demonstrating opportunity, plausibility, and some indications from recent genetic studies that some variations were introduced in the right time frame to have come from Neanderthals. This argument was presented as if the authors expected people to be "outraged at the charge", but it seemed sensible and plausible to me.

The radical argument that I expected to have trouble with is the claim that there's something genetically different about the Ashkenazi Jews. One of the members of the reading group I attend has been making this point for years, and I've been passively resisting it for just as long. Well, Cochran and Harpending put an end to that, easily and without much fight. They showed that the time-frame isn't extreme, that there were sufficient environmental pressures to push for particular changes, and that there's a reasonable case that the Ashkenazi were genetically isolated for long enough for the hypothesized changes in intelligence and susceptibility to diseases to have arisen. They add in some evidence that the diseases specific to the Ashkenazi are tied to genetic changes in neuron development to hammer home the point that the changes in brain function and disease susceptibility are probably tied together. The fact that we already knew (even if we didn't admit it in discussions of evolution) that Tay-Sachs is specific to Ashkenazi and is of genetic origin helps cement the case that evolution has continued into the modern era.

If there had been any remaining doubt that there are genetic predispositions to varying intelligence by race, this pretty clearly puts them to bed. I still agree with the sentiments Les Earnest expressed in his 1989 article Can Computers Cope with Human Races? It's not clear that there's anything useful to be done with this fact, and it's pretty clear that many people mis-apply the fact, but it's a fact none-the-less.

Explosion makes a fairly strong case that evolutionary stasis doesn't happen without a static environment. Humanity hasn't been static over the last 100,000 years--there have been many changes in our way of life over that period, and the changes keep accruing faster than evolution has been able to catch up. Our bodies are still adapting to changes in diet since the agricultural revolution and continuing changes in the sources of our food. Our susceptibility to disease has varied dramatically across populations, and the differences haven't settled down yet. There are still vast differences in hygiene between first world and developing nations and many places in the world where populations are sparser and provide fertile ground for new diseases to arise or transfer to human hosts. There is less of a case for sufficient continuing genetic isolation to drive differing evolutionary pressures for intelligence, but there is certainly pressure for differing intellectual capabilities than were selected for 200 or 500 years ago, much less 1000 years ago.

We haven't finished adapting to the civilization that surrounds us, and the form of our civilization continues to change. We shouldn't expect continuing evolution to be visible on a human time scale, but we shouldn't be surprised that many of the differences among people can be explained as the effect of different evolutionary pressures on our ancestors. In some cases, like disease susceptibility, we can take advantage of it if we stop treating it as tainted information. In others, as race, we should in most cases pay more attention to the abilities of individuals, rather than to the predispositions predicted by apparent racial categories.

Thursday, October 15, 2009

IntelliJ IDEA open sourced

JetBrains has just announced that they've released the IntelliJ platform as open source! This is great news, as I've always maintained that IntelliJ IDEA is significantly better than all the other Java IDEs, and their main disadvantage was price, and the second biggest weakness was Eclipse's ability to let users add support for new languages. At one stroke, JetBrains has undercut both of these problems and enabled many more people to make use of the platform.

I only know what I read in the announcement and the FAQ, but it appears that JetBrains has done this for the right reasons, and probably in the right way. Their goal is to increase adoption of the platform, and to encourage third-party developers to work with IntelliJ rather than competitive products. They have kept some of their technology proprietary in order to be able to continue to run a business, and they're focusing on the larger enterprises that they make most of their money from. This is reasonable and proper.

The license they're using is Apache, which is very open and provides no significant restrictions on re-use. I'm very happy about this.

Sunday, September 27, 2009

C. J. Cherryh: Cloud's Rider

C. J. Cherryh's Cloud's Rider is the successor to her Rider at the Gate, which I reviewed last year. This book was far more psychological than the previous, delving deeply into the reactions of all the characters to the dark telepathic sendings of the rogue nighthorse stalking the village.

In Rider, Danny Fisher teams up with Cloud, one of the telepathic Nighthorses on Finisterre, where a small human outpost seems to have been abandoned on a remote hostile planet. The humans who remain cloistered in the villages try to hide from the power of the ambient telepathy of the native fauna, and as a result are unable to exploit the resources of the world. The riders have allied with the nighthorses, which enables them to travel more freely, which makes them a crucial lifeline connecting the remote villages. But there's a lot the humans don't know about the planet.

In Rider, we see that nighthorses can sometimes go rogue (in that case because its rider died in an accident) with disastrous consequences for an entire village. In the end, Danny Fisher agrees to escort the survivors, the three Goss children, Carlos, Randy, and Brionne. Brionne was at the center of the tragedy, having been the focus of the rogue's attention, but at the divide between the two books, she is in a coma.

Cloud's Rider starts out with the quartet struggling up a mountain road in a blinding snowstorm. They end up walking for a couple of days, dragging Brionne behind them on an improvised travois. They had intended to stop halfway up the mountain at a permanent rider's shelter with provisions that ought to be enough to sustain them for quite a while, but the snowstorm causes them to miss the shelter and end up in Evergreen village--not large enough to stand up to the sendings of the rogue that seems to have followed them up the mountain.

The bulk of the story takes place in Evergreen, where the politics is intense among the villagers, the miners, the preacher, and the doctor who adopts Brionne. Some of the villagers quickly recognize that the now abandoned village at the foot of the hill represents a major source of wealth for whoever can claim it when spring brings an end to the unrelenting winter blizzards.

But the focus is on how Danny, Carlos, Randy, and Brionne react to their circumstances and the malevolent presence in the telepathic ambient. As villagers, Carlos and Randy would normally be expected to be oblivious to the rogue's sendings, but they were in Tarmin when everything came crashing down and took a long trek in the presence of Cloud, so they know it's all real. Randy is drawn to the nighthorses and envies the romance of the riders' way of life. Carlos just wants to get back to Tarmin and make use of his blacksmithing skills. Brionne's outlook has been warped by the sendings of the rogue, and her delusions and paranoia are affecting the villagers around her.

Once again, Cherryh has done a masterful job of developing characters whose differences and similarities are highlighted by the juxtaposition with alien thoughts and alien approaches that are perfectly consistent and highly intriguing. I very much enjoyed this story sequence.

Thursday, September 17, 2009

John C. Wright: The Golden Age

John C. Wright's The Golden Age didn't work for me, though it would seem to have a lot of things in its favor. It's a futuristic high-tech story with a focus on personal enhancement and social interaction technology and little in the way of weapons or overt conflict. There is little in the way of visible government and secures cooperation with the society's mores via social pressure. Since everyone has access to largely unlimited resources, anything short of the ultimate form of ostracism leaves a lot of room for many variations in behavior and personal expression. I think the beginning of the story failed to grab me because of the reluctant hero angle. The protagonist, Phaeton, takes a while to admit that there's a problem and he has to take the responsibility to figure out what happened and why he's not getting what he wants even if he doesn't understand how he could have caused the problem.

Eventually Phaeton realizes that his memory may have been altered, and begins to suspect a vast conspiracy to silence him for something he doesn't remember doing or wanting. He bucks a lot of social pressure to investigate his past and figure out what he must have been trying to accomplish and who would have been trying to stop him. I thought the evidence he found seemed flimsy in the context of the story-with even his ostensible friends urging him to accept the world as it seemed to be, his constant bucking of the system seemed more like a habit of being contrarian than a dogged determination to find the truth. Of course, according to Wright's story, Phaeton was correct to trust his instincts, and there really was a vast conspiracy to suppress the evidence he'd discovered, and the entire society was at risk. But like Cassandra, Phaeton is doomed to be doubted, shunned, and ignored. In a society with serious life extension this is a long-lasting problem. The novel ends inconclusively, to be continued in a later volume.

I liked the technology and the way the society was organized. But from Phaeton's viewpoint, it's presented as an instrument of the society's downfall. No one is in charge, so no one is responsible for battling the external existential threat. The lack of centralized monitoring or control enables Phaeton to continue his investigation, and to amass enormous tools and weapons, but his refusal to cooperate with others leaves him ultimately very weak, and unable to recruit allies to his project.

Tuesday, September 08, 2009

Alistair Reynolds: Pushing Ice

Alastair Reynolds's Pushing Ice is a great story, weakened slightly by following the first contact story that justified the initial adventure to a lot of inter-species intrigue. The adventure is a good psychological thriller with lots of interactions among the crew of an asteroid miner that gets unexpectedly diverted to an interstellar chase in the wake of Janus, one of Jupiter's moons that suddenly leaves orbit and heads for the stars. This is a great premise that justifies a good amount of conflict and chaos, as they move from studying Janus, to realizing the implications of effectively being towed at high speed toward distant stars, to finding a way to survive on the surface of a not-quite dormant alien ship.

The crew goes through internal struggles over whether to abandon the chase while there's still some hope of returning to earth, which involves both politics and some violent attempts to overthrow the captain. The hard feelings left behind color all the crew's later attempts to survive long term starting with supplies meant for a much shorter trip. They spend their time studying the star they're heading for and the moribund uncrewed craft they're tethered to.

Eventually they arrive at an immense structure that provides room for ships from several different species. The Fountainheads are highly advanced and provides rejuvenation facilities that restore several of the human crew (including one who had been in cryonic suspension for most of the trip). The humans get conflicting advice from different alien groups about who they can trust and who is dangerous, and not surprisingly, different factions decide to trust different groups. Eventually, Janus itself is the target of several alien species who expect to be able to reap large energy resources from the craft. I thought the interactions with the aliens and the human political machinations occasioned by the aliens to be much less interesting and more poorly motivated than the first half of the story, concerning how the human crew got along and what they had to do to survive the long trip. The first half definitely made the story worth reading, though.

Sunday, August 30, 2009

Peter Leeson: The Invisible Hook

Peter Leeson's The Invisible Hook is a fun read with some serious points to make about economics. Leeson is a George Mason economist who has been a fan of pirate literature and lore most of his life. At some point he realized that he knew enough about the real history of pirates, particularly the ones that have been romanticized in our literature, to say something interesting about how economics drove and shaped them. Of course this also gave him an excuse to spend more time exploring pirate mythology and a way to get academic credit for it.

Leeson's most interesting finding is that the pirates often had written constitutions and elected their captains and other officers, and could remove them when the crew pleased. He also investigates the feedback mechanisms and social forces that reinforced this system and kept it stable. Some of it has to do with the fact that Piracy was a cooperative enterprise, and that pirates had no recourse to the law to enforce their rules. This leads to a need for agreement among the crew on how they are to operate, and consensus that breaches can be punished. The modern pirates off Somalia or South East Asia have a different set of contingencies, since it's easier for individual pirates to quit without endangering their own lives or those of their mates.

A lot of what Leeson talks about falls in the area that economists call signalling, by which they mean actions taken by an agent to convince someone of their likely behavior. The fundamental question about signals is what keeps others from copying the signal in order to fool the signal's target. Leeson puts the skull and crossbones and stories about pirate viciousness in this category.

Pirates cultivated an image of being ruthless killers who would never-the-less not physically harm victims who gave up without a fight. Their goal, of course, was to reduce the amount of fighting they had to do. So they did their best to convince honest seafarers that if you fought with pirates they would treat you harshly. The question this raises is why didn't privateers (who had government charters to act like pirates) also fly the jolly roger, and Leeson's answer was that it was too costly for them, since flying the pirate flag led to easy convictions for being pirates, even if you have the government charter. So Pirates competed for a rough reputation, and their competition had to convince their prey to surrender in more costly ways, since their charters proscribed torture and wanton violence.

There's a lot more than this in the book, and just enough asides of "Avast" and "Aargh" to raise an occasional chuckle. If you still retain your youthful enthusiasm for pirates, and are interested in the economic way of analyzing behavior, this is an entertaining approach to the subject that is also educational.

Tuesday, August 04, 2009

Outrageously Inhumane

Ken MacLeod has a short post on his recent attendance at a con marking the recent Apollo anniversary. Ken's highlight was a talk by a self-professed techno-utopian, who said, among other things "The idea that we should use less energy is outrageously inhumane and regressive" and "Most reycling schemes are feel-good rather than do-good, condemning us to pre-industrial, manual rooting about in rubbish." Ken has good sense and good tase. That may be why I like his science fiction.

Thursday, July 30, 2009

On "Organizational scar tissue"

John Cook blogged about stupid policies within organizations, in response to a quote he saw:

Policies are organizational scar tissue. They are codiļ¬ed overreactions to unlikely-to-happen-again situations.

John mostly agreed and pointed out that policies sometimes arise for good reasons, though the reasons may be hard to reconstruct later. I want to take the conversation in a different direction: How do you design an organization so that the policies can change if the circumstances have changed?

There are policies designed to handle unlikely events and policies designed to handle everyday events. Both kinds can atrophy over time and no longer be relevant. The ones for unlikely situations are harder to set up time-outs for, but the ones for everyday use are the more costly when they outlive their usefulness.

The hard part of the situation John described is figuring out a different approach that leaves you more flexibility. (If you've built a Maginot line, as in his metaphor, you're pretty much stuck.) So the challenge is designing an organizational culture that can identify outmoded policies and change or discard them.

In a small, agile organization I once worked in, we developed a checklist for use by the project manager (me!) during the release process. The reason was that there were things I occasionally forgot to do that came back to bite me or my boss, the CEO. The rule about the checklist was that it was for me to fill out on my own responsibility. There were items that asked for initials from other people, but I was allowed to ignore them on my own initiative. We updated the checklist often enough that I could remove useless items whenever I noticed that their relevance had disappeared.

For a larger organization, you don't want an individual to make those decisions unless there's a way for them to know the original reason for the policy. So writing up the rationale along with the policy makes it possible for the person on the spot to know whether they should ignore or attempt to change the policy. In a small agile organization you don't want the overhead of having to write up the rationale every time you add a process step, but if the organization grows, you'll need the rationale. It's a dynamic tension that organizations have to learn to adapt to as they grow.

ADDED During the ensuing discussion on The Endeavor, someone pointed to NetFlix' internal presentation on their corporate culture. Very impressive. If I wanted a job, that's the kind of company I want to work in.

Tuesday, June 23, 2009

Charles Stross: Saturn's Children

Charles Stross's Saturn's Children depicts a post-humanity future in the solar system; all the characters are various forms of robots, and many have recognizable and sympathetic feelings, goals and aspirations. The humans died out a few hundred years ago but neglected to think about how their posterity would get along in their absence.

Stross has created a very interesting society here. Many of the robots (the focal characters, particularly) were designed to be personal servants to the humans, and their drives are very much shaped around the desires of their absent masters. Others, less constrained, use these drives to control and shape them. The basic conflict is over whether it's possible or desirable to bring back humans. If this could be done and they could be managed, then whoever controlled the humans would have immense power because of the drive to serve the humans that is built in to so many of the robots.

The background is fascinating in that any robot that was expected to work in the presence of humans was designed to have strong strictures to honor and obey them, which leaves them in a strangely constrained state now that the humans aren't around. All the apparatus of government is still in place, and no decisions can be made or changed without humans to vote or make an administrative decision. So the government runs on auto-pilot, and many projects that made sense--terraforming to create new habitable territories--are senseless when the robots can adapt to harsh environments much more easily. And robots that are more able to bend their internal definitions of what the humans wanted are less constrained.

The basic conflict sheds light on thoughts about freedom and free will. Freya, the main character, was designed as a sex-bot, but she was first activated long after the humans had died out, so her drive to serve them is unrequited. Her design gives her some special abilities (cosmetic enhancement gives her an innate talent for disguise and her adjustable high-heeled feet (as Laurie Anderson sang) come in handy in combat and escape. But most of the time, she's a free agent, other than having to serve employers and mentors. Self-ownership is a concrete idea in this society. All the robots originally had owners, but the ultimate owners are now gone from the scene, so each robot is owned by some corporation. Being self-owned means controlling the corporation that owns your body. Unfortunately, some of the robots are rich enough that they can sue others into bankruptcy, and buy their corporations, so there's no safety in owning your corporation. In addition to the slave chips that get used occasionally, ownership conveys control by allowing one being to assert "control level nine" and require absolute obedience of another. Of course, if the subservient one can convince herself of doubts about the authenticity of the ownership, she can escape the fetters.

Overall, this is a fun romp. Stross displays a strong sense of humor, and Freya is a joy to watch. Many of the service machines and tools are aware and interactive, and Stross is quite inventive in giving them interesting drives and quirks. The political implications are interesting, but mostly not the focus of the story. It calls itself "A Space Opera" and it is that, with the addition of interesting personalities and new viewpoints on how our robot progeny will live and think. Saturn's Children is a finalist for the Prometheus Award. The freedom orientation is too subtle to win in what has turned out to be a year with some very strong candidates. It's a lot of fun to read none-the-less.

Saturday, June 13, 2009

Dean Keith Simonton: Greatness

Dean Keith Simonton's Greatness is an attempt to catalog all the influences that allow some people to have a larger effect on the world than their fellows. It takes a more wide-ranging and less data-driven approach to the question than Charles Murray's Human Accomplishment. I thought it was less successful, mostly because there were too many sections that were speculative, not well grounded, or inconclusive.

There were, of course, sections that contained important insights, but if one part in three, scattered evenly throughout a 500 page book is unfocused material, the useful portions are harder to identify and lose much of their impact. The most important conclusion I found in the book is that great results are the consequence not just of intelligence, insight, or drive, but that perseverance matters, even for people with huge natural talents. Simonton shows that even for the giants who are widely acknowledged to have remade their fields and outshone everyone else who has worked in the same fields, their masterworks appear in about the same proportion as for others who ended up contributing less overall. Their secret was no secret: they merely worked harder and longer and produced more. Their natural talents sometimes give them a slightly higher batting average, but overall productivity of great works and long-lived impact is a result of starting early and working more hours over more years than their rivals. Great achievers all have mediocre and uninteresting works mixed into their corpus, it's just not so noticeable given their best output. You'll find similar statistics for how long after someone started working that they produced their best work for the great and the near-great. The great simply find more time to be productive, and continue longer. It's possible that early productivity and continuing results enable the greatest producers in any field to continue contributing longer, which adds to their records, but single-mindedness and continuing focus is also crucial.

Simonton covers topics including intelligence, personality, birth order effects, genetics (mostly focusing on the extent to which greatness runs in families), pathology, and what makes some periods more fertile than others. The lack of mathematical models makes most of his speculation hard to trust as he seldom compares the statistics of the successful with those of their surroundings. If you don't know how often familial connections should arise by chance, it's hard to conclude that any particular number of examples demonstrates the existence or lack of any effect. Overall, there were lots of interesting facts and factoids, and the writing is engaging, but it's hard to take many of the conclusions very seriously as anything more than anecdote.

Monday, June 08, 2009

Neal Stephenson: Anathem

Neal Stephensonhas written another fascinating, grand, opus, Anathem. He's back to writing actual science fiction, and it's pretty impressive. It's somewhat of a surprise that the book wasn't broken up into a few smaller books, since it consists of a sequence of mostly separate adventures by a single hero, each explored in depth. (Actually, now that I look, it breaks easily into three segments of about 300 pages each.) The story starts with Erasmus as a young avout at a sanctuary for intellectuals. The first third of the story takes place as Erasmus is learning the avout way of thinking, and figuring out his place in his world. Stephenson does a good job of introducing in a concrete format several deep philosophical concepts that the reader will want to understand in the final section of the story. The second part of the story takes Erasmus and some of his fellows on a journey around Arbre, their world, after they are expelled from the sanctuary as a result of an emergency in the outside world. I'm not going to talk much about the final section, since it would spoil the story to know what the conflict is about, but lets just say it's a continuation of the journey to further and stranger places.

Erasmus' community is part of a system of monastaries spread around the world that serves to bleed off some of the presure for relentless progress from their society. The outside world still has an irregular cycle of boom and bust, but the mathic communities are isolated and serve to preserve the learning through the tough times. They have a very long view of the changing times--in fact Stephenson developed the ideas while working on some projects for the Long Now Foundation. The communities only have contact with the outside world once a year, and internally, the concents are further divided into subcommunities that only have external contact every decade or century. The more isolated each group, the more deeply they delve into various abstract and theoretical ideas. The less isolated groups treat learning almost as a competitive sport, and their different colleges emphasize different approaches to learning.

Erasmus and his cohort explore the nature of our universe, and some interesting philosophical issues that turn out to be relevant for the finale of the story. The characters explore higher math, philosophical issues such as personal identity, and quantum physics. I found the story very engaging, but I'd expect the depth and detail to turn off many readers. Stephenson is well in his element, exploring many issues in the back story while keeping an interesting story going in the foreground. The foreground story has love and loss, battles and chase scenes, extraterrestrials and high tech, and plenty of fun.

Anathem was nominated for this year's Prometheus award, but isn't a finalist. Libertarians will find it interesting for its exploration of some of the issues of governance of separated societies, and self determination. But it doesn't have the immediate political connection of some of this year's other nominees. I found it to be a fun read, and enjoyed the characters and situations as well as the exporations of philosophy and math.

Sunday, June 07, 2009

2009 Hellyer Park Bike Racing

I know I've mentioned my interest in the bike racing down at Hellyer park before, but it's time (apparently past time) for the annual update. They'll be running the Friday night race series in south San Jose on June 19, July 17, August 14, and September 4 and 18. Racing starts at 7pm and includes a variety of events: sprints, points races, miss & out, and scratch races.

There's also a special event (American Velodrome Challenge) June 26 and 27. The Friday night session (also starting at 7pm) includes Miss & Out, Kierin, Madison, points and scratch races. On Saturday, they'll have two sessions. The morning session lasts from 9:30am to 1pm, and the evening session starts at 6pm. The morning seems to be all sprints, while the evening has sprint finals, keirin, miss & out, scratch and points races.

Monday, June 01, 2009

Michael Flynn: January Dancer

Michael Flynn's The January Dancer is a finalist for this year's Prometheus Award because of Flynn's focus on the corrupting influence of the prospect of absolute power, and some of the characters' attempts to avoid it. Along the way we get a fascinating interstellar romp through a new way of navigating wormholes and get to visit a couple of worlds with some interesting variations on ways of organizing anarchic societies.

The story advances in two parallel streams, which eventually converge. The framework comes from an itinerant storyteller (a seanachy) gathering information about a story she has heard from an old worn-out man who is familiar with a lot of the details. These intercalary chapters alternate with the real action. It starts out with Captain Amos January and his small, but varied crew getting stranded on a small, unpopulated planet off the main trade routes. They know how to forage for the fuel they need without help, but find a treasure room with an inscrutably flowing artifact before they escape the planet.

Their dangerous flight from the planet leaves the ship damaged enough that Captain January must pledge the artifact as collateral to pay for repairs when they reach a civilized port. While they make a cargo run so they can afford to redeem the artifact, rumors start to circulate that the artifact has mysterious powers corresponding to an old legend about an alien mechanism that gives the holder powers of persuasion.

The January Dancer takes us on a tour that includes rebellion, piracy, imperial ambition, quiet guard duty, superhuman secret agents feared and obeyed by everyone, a quiet ungoverned unconquerable planet, and a hero willing to fore-go absolute power because of his concern for what would happen after his custodianship eventually ends: He can see that the artifact will spend more time in the hands of ambitious immoral men than in the care of those with a conscious.

The pacing is good, the characters are strong, and the scenery is varied without being too rushed. We get to spend the most time on New Eireann, a hard-scrabble planet that had hired a commercial firm to

manage their government contract. [The firm] sent in an honest administrator. By all accounts he ran a clean and honest administration though at first the Eireannaughta didn't realize that because they didn't know what one looked like. When they did, they revolted, because an administration that won't take bribes generally won't hand out favors, either.

That firm is replaced with the Interstellar Cargo Company (ICC), which is more amenable to skirting the edges, but seems to do a good job of managing commerce, keeping goods flowing efficiently among all the reachable stars.

The groups Flynn keeps an eye on include January's crew, the government and rebels on New Eireann, and the Hounds' Watch (the previously mentioned small corps of near-supermen responsible for ferreting out momentous plots and keeping the peace). Each has its internal conflicts and clues to add to the developing plot.

<SPOILER>I found it interesting that the ICC is depicted as being accepted by everyone for having cleaned up and regularized commerce while making a reasonable profit. Everyone seems content with this, until someone learns that they've discovered a way of circumventing the wormholes everyone uses, giving them a near instantaneous communications path. For some reason people consider this an unfair advantage, rather than a normal commercial development that benefits all of their customers. In most fiction, the opprobrium would have started with the fact that they were making money without doing anything more than ensuring that goods moved smoothly between planets. The fact that the disapproval didn't start until people found out that they had a secret technology was a little surprising.</SPOILER>

Most of the book is taken up with the adventuring, exploring, and intrigue. We don't find out until fairly late in the story what the artifact is good for, and only one of the characters thinks about any consequences beyond either ensuring the bads guys don't have it or trying to obtain it for themselves. The fact that his concern is expressed after the fact and not telegraphed to the readers or any other characters reduces its impact as a choice. It's a fait accompli by the time we hear about the issue, so there's no time to consider the issue. I thougth this reduced the dramatic impact considerably of what was apparently the core conflict in the story.

Sunday, March 15, 2009

Terry Pratchett: Going Postal

Terry Pratchett's's Going Postal is another in Pratchett's Discworld series. I've only read a handful of these stories, but they've all been fun, lightweight fantasies that explore greater and lesser societal issues in a skew universe.

In this story, Pratchett looks at competition in business and the effect of the competent entrepreneur. Okay, you'd be unlikely to see it that way if you read the book, but those are the themes that I extracted from the story. Lord Vetinari wants to rejuvenate the Ankh-Morpork postal service, and he needs a motivated, creative manager to manage it. He plucks Moist von Lipwig from the scaffold and makes him an offer he can't refuse. Moist turns out to be very creative; his history as a schemer and con artist having prepared him to read people on the spot and make up convincing stories which he can fill in the details of later when he figures out which direction he's going to take the scam.

In this case, Moist is going up against the corrupt monopoly that runs a private semaphore-based messaging system that is closer to a telegraph than a message carrier. The Grand Trunk also has labor relations problems, since their lack of attention to maintenance issues has led to a surfeit of injuries and deaths among the employees. Moist is able to out-compete the Grand Trunk in order to regain the customer base the official post office lost long ago and use an occasional bit of sabotage to reduce the Grand Trunk's ability to stay in the game.

In the end, Moist resurrects the post office by acting as the nimble entrepreneur (with a side of underhandedness) in competition with an entrenched bureaucracy. He has enough obstacles (decades of undelivered mail piled up in the post office, attacks by the competition) and humanity (pursuing a surprising love interest) to be a sympathetic character. Pratchett turns the competitive environment on its head, but I think most readers will see that it's normally government monopolies that are resistant to change, and the story shows that it's the stodginess and resistance to change that lead to poor service. If the competitors have the ability to try new things (normally, not including violence and sabotage), the customers come out ahead, and competition improves all of the enterprises touched by it.

Thursday, March 12, 2009

Julia Robinson Mathematics Festival: March 29 and May 2

If you're interested in math, puzzles, and encouraging middle and high school students to learn more about math, I recommend taking some time to help out at the Julia Robinson Mathematics Festival on March 29th in Emeryville. I heard about the festival at a talk last year. I actually helped out as a puzzle mentor for the geeky (adult) audience of this talk, and had a great time talking people through one of the puzzle sets.

The JRMF presents great puzzles--graduated challenges, interesting applications--in a context that encourages kids to work together on solving them. This both encourages the ones who don't get the solution right away and cements the mastery of those who caught on more quickly. Everyone has fun, and many youngsters are exposed to the notion that math is relevant and applies in many everyday situations.

They currently have an event scheduled at Pixar in Emeryville on Sunday March 29, and are planning another for May 2, but don't have a firm location for the latter yet. They're looking for adult volunteers to staff math tables. Here's the announcement I got via email.

General information:
The Julia Robinson Math Festival
The festival, for students in grades 6-12, will consist of morning mathematical activities lunch, an accessible (to middle school and high school students) math talk, and, of course, prizes. Julia Robinson is the Berkeley mathematician who, among other important discoveries, made significant contributions to the solution of Hilbert's Tenth Problem. Here's a bit of biography, and a bit about Hilbert Tenth.

The morning activities will cover a wide variety of mathematical topics, including abstract problem-solving techniques like symmetry and parity as well as content ranging from arithmetic and algebra through combinatorics and topology. Students will have time to visit at least five or six activities during the morning session. For a sample activity, see the Candy Conundrum (and Teacher's Guide). (There are downloadable links at Each activity table will be staffed by an expert, and upon reaching certain milestones in their understanding of the mathematics behind the puzzle or game or activity, students will be rewarded with raffle tickets for the afternoon's prize drawings. Thanks to our sponsors, the desJardins/Blachman fund and Pixar, we have an impressive array of prizes!

For those who want to staff a math table:We need people who are comfortable with hard math problems, and who can patiently work with kids to guide them toward progress. We are looking for guides who will listen to what kids have worked on, ask some (perhaps leading) questions, and then let the kids continue to progress. If you're interested, contact Joshua Zucker (email: first.last at stanfordalumni dot org), and if you're not sure you're interested, contact him anyway and he can give more detailed information about the expectations of the table leaders.

Monday, March 09, 2009

Harry Turtledove, Opening Atlantis

Harry Turtledove's Opening Atlantis is a candidate for the Prometheus Award. I suspect it'll be chosen as a finalist since it's well written and interesting and has strong pro-freedom themes. Unfortunately for Turtledove (who was a co-winner last year for The Gladiator), some of my other favorites have more topical themes and are just as fun to read.

Opening Atlantis posits that an extra contintent has been inserted in the middle of the Atlantic, but otherwise history and geography are pretty much unchanged compared to our timeline. Not surprisingly, the new continent is noticed during the Age of Discovery, and since it's located in the Atlantic, it is named for the (realized to be mythological) Atlantis. The geography, flora, and fauna are surprising to the discoverers and eventual settlers. One recurring side note is the characters who recognize that the indigenous species are different in the old world, on Atlantis, and in Terranova (North America). They don't quite point to the possibility of evolution explicitly, but it's the kind of idle wonder that one imagines occurs many times before an explanation is found.

Turtledove's story is presented in three parts from successive eras, which are roughly Settlement and Independence, Battling the Pirates, and Dragged into the European War. Each of the segments has Edward Radcliffe or his descendants as central figures, and the first and last each features a different battle over freedom-related issues. In the first part, Atlantis is discovered and settled by English, French, and Spanish fishermen and their families. Lured by familiar climate, each group settles on a different section of the coast, and builds multiple towns that trade, but maintain separate cultures. The conflict in Part one comes when a nobleman sent away by the British Crown decides that British Atlantis is his new fiefdom. Since the settlers intentionally migrated in order to get away from kings and lords and their incessant war and taxation, it isn't long before they rise up and defeat the invaders. They are helped a lot by their familiarity with local conditions, and their willingness to fight as insurgents.

In the second part, two of Edward's descendants, Red Rodney Radcliffe (a pirate captain) and William Radcliff (a shipping magnate) battle over the fate of a pirate stronghold on the western edge of Atlantis. William allies with other major shipping owners to defeat the pirates and ensure that they'll be able to trade freely. I could say that the freedom-related themes have to do with free trade and cooperation among private enterprises to address a common problem, but truthfully this is just a setup for a series of naval battles. The good guys win.

The third part (my nickname was "Dragged into the European War") covers a period of strife between Britain and France. Early on, there was some discussion of which side Spain would take, and there are hints that other powers are engaged in other theaters, but we only see the battles that take place in Atlantis. Victor Radcliff leads the British-allied settlers, and his French-settler counterpart Roland Kersauzon is descended from the frenchman who first identified the new continent. Early on, the French settlers are reinforced by French troops, but for most of this war, the British navy controls the oceans but isn't concerned about the fate of Atlantis, so the armies on the ground are on their own. Tactics rule the day, and the settlers teach the regular armies several lessons about dealing with insurgents and familiarity with local terrain.

Victor Radcliff is joined by Blaise, an escaped slave who becomes a leader among the British irregulars. A lot of the conflict in this section is about how competent Blaise is and how he is able to win over British settlers who haven't had much previous contact with Blacks. The French and Spanish they run into (and the British regulars) are much less tolerant. Blaise finds several opportunities to point out the parallels between his situation and the freedom that the other settlers are fighting for.

Overall, it's quite a fun read and well up to Turtledove's normal high standards. The characters and their battles and strategic surprises are well-drawn and plausible.

Monday, March 02, 2009

Stephen Ziliak and Deirdre McCloskey: The Cult of Statistical Significance

Stephen Ziliak and Deirdre McCloskey's The Cult of Statistical Significance is a poorly argued rant about what appears to be an important topic on the pursuit of scientific knowledge. Ziliak and McCloskey argue that many of the statistical sciences have been using the wrong metric to determine whether the results of experiments are interesting and relevant. They report on a few detailed reviews of articles in top journals in economics, psychology, and other fields to show that the problem they describe is real and pervasive. Unfortunately, they are much more interested in casting aspersions on the work and influence of Ronald Fisher and building up his colleague William Gosset, and so they don't actually explain how to apply their preferred approach. In amongst the rant, they do manage to make the defects of Fisher's approach clear, though it's tedious reading.

The basic story is that Fisher argued that the main point of science is establishing what we know, and to that end, the important result of any scientific experiment is a clear statement of whether the results are statistically significant. According to Fisher, that tells you what confidence you should have that the results would be repeated if you ran the experiment again. Ziliak and McCloskey want you to understand that a result can be statistically significant but practically useless. And there are worse cases, where statistical significance and Fisher's approach leads scientists to hide more relevant results, or worse to conclude that a proposal was ineffective when the data show that a large effect might be present, but the experiment failed to show that it was certain. Ziliak and McCloskey want scientists to primarily report the size of the effects they find, and their confidence in the result. To Ziliak and McCloskey, a large effect discovered in noisy data is far more important than a small effect in very clear data. They point out that with a large enough sample, every effect will be statistically significant. (Though they don't explain this point in any detail, nor give any numbers on what "large enough" means. I have an intuitive feeling for why this might be true, but this was just one of many points that wasn't presented clearly.)

They describe a few stories in detail to show the consequences for public policy. Vioxx was approved, they claim, because the tests of statistical significance allowed the scientists to fudge their results sufficiently to hide the deleterious effects. (It's not clear why this should be blamed on statistical significance rather than corruption.) They also present a case that a study of unemployment insurance in Illinois found a large effect ($4.29 in benefit for every dollar spent), but gave the Fisherian conclusion, not just that the result wasn't statistically significant, but that there was no effect. It turned out that a careful review of the data showed that the program had a statistically significant benefit-cost ratio of $7.07 for white women, but the overall benefit-cost ratio was not statistically significant because the $4.29 was only statistically significant at the .12 level, while under .05 or less is required by Fisher's followers.

Ziliak and McCloskey demonstrate that they're on the right side of the epistemological debate by supporting the use of Bayes' Law in describing scientific results, but beyond one example, they don't explain how a scientific paper should use it in presenting results. The use of Fisher's approach gives a clear guide: describe some hypotheses, perform some tests, finally analyze the results to show which relationships are significant. With Bayes, the reasoning, approach and explanation are more complicated; but Ziliak and McCloskey don't tell how to do it. Of the 29 references to Bayesian Theory in the index, 24 of them have descriptions like "Feynman advocates ...", or "Orthodox Fisherians oppose ...". There aren't any examples of how one might write a conclusion to a paper and show Bayesian reasoning, even though they pervasively give examples of analogous Fisherian reasoning that they find unacceptable.

Another significance question that Ziliak and McCloskey argue is important (but that they don't explain adequately) and that statistical significance hides is how much various treatments or alternate policy approaches might cost. Fisher's approach allows authors to publish that some proposal would have a statistically significant effect on a societal problem or the course of a disease and not mention that the cost is exorbitant and the effect small (though likely). Ziliak and McCloskey argue that journal editors should require authors to publish the magnitude of any effects and a comparison of costs and benefits. According to the reviews they've done and others they cite, it's common in top journals to omit this level of detail and to focus on whether experimental results are significantly different from zero.

Another of the authors' pet peeves is "testing for difference from zero". They claim that it's common for papers to report results as "statistically different from zero", when they're barely so. They use the epithet "sign testing" for this case. The lack of attention to the size of an effect that significance testing allows means that papers get published showing that some effects have a positive effect on a problem, even when the effect is barely different from a placebo. And there are enough scientists performing enough experiments today that many treatments with no real effect will reach this level of significance purely by chance.

Overall, the book spends far too much time on personalities and politics. Even when the discussion is substantive, too much effort goes into why the standard approach is mistaken and far too little on how to do science right, or why their preferred approaches would actually lead to better science.

For the layperson trying to follow the progress of science, and occasionally to dip into the literature to make a decision about what treatment to recommend to a family member or what supplements would best enhance longevity or health, the point is that scientific papers have to be read more carefully. Ziliak and McCloskey argue that editors, even of prestigious journals, are using the wrong metrics in choosing what papers to accept, and often pressure authors to present their results in formats that aren't useful for this purpose.

When reading papers, concentrate on the size and the costs of the effects being described. Significance can be relevant, but the fact that a paper appeared in a major publication doesn't mean that the effects being described are important or useful. Don't be surprised if the most-cited papers in some area don't actually present the circumstances in which an intervention would be useful. Don't assume that all "significant" effects are relevant or strong.

Tuesday, February 24, 2009

Financial Crisis: Regulatory Arbitrage

I've been following the financial debacle pretty closely. I can explain a fair part of it, describing all the moving parts and how they interacted and why regulators and participants had a hard time seeing that their individual parts in it would lead to a calamity given what everyone else was doing. The main message is that it didn't start and end with the housing bubble and crash. The bubble had causes, (and the legislators behind it aren't backing off; they continue to think that more people should own houses whether they can afford them or not) and so did the crash.

But I've had a hard time summarizing the whole thing and naming a single root cause (other than vague "government interference in the markets") or a suggestion for what should be done differently that would lead to different outcomes next time. The whole thing seems so complex, and everyone in it merely responds to local incentives so it's hard to see what different incentives would lead to a a more stable outcome.

But I recently followed a pointer to an article by a group at NYU who gave a good summary and pointed to one of the parts of my story as the root cause, and I now think they're right. The root cause was regulatory arbitrage at the banks. Regulatory arbitrage describes actions someone takes in order to avoid the affects of some set of regulations that might apply to them if they ran their business differently. Rather than buying some asset in a jurisdiction where it is taxed, you have a subsidiary buy it, or you buy an option, or you buy a company that already owns it. If one kind of institution can't engage in a certain practice that looks like it'll make money, then people will invent a new kind of institution that's subject to different regulators which isn't so prohibited.

Part of my long riff on the crash has been that practically every financial institution that exists now is the result of regulatory arbitrage of some kind. Consumer banks accept deposits, but their activities are tightly regulated in order to qualify for deposit insurance, so commercial banks don't take deposits from consumers. Credit Unions have a different set of restrictions on their activities. Savings and Loans were restricted in the interest they could charge on loans, so when they had to compete on the interest they paid on deposits, they took excessive risks leading to the S&L crisis. Commercial bank investments are regulated and limited, so there are investment banks. Those have their own regulations, so we saw the rise of hedge funds which didn't have to report to anyone except their investors.

The regulatory arbitrage at the root of this crisis was that the consumer banks were restricted in what assets they can hold and what assets they can sell. So the mortgage-backed securities (MBS) they were selling stripped out the lucrative part of the loan repayment income stream and sold that for cash they could use to make more loans, while they ended up keeping the riskiest part on their books. Most of them found tools that appeared to insure against the remaining risks, but those were systemically flawed--all the banks relied on the same few institutions, and their back-up plans would only have worked if problems were isolated. When the crunch came it was general, and so all the back-up plans failed together.

As the NYU group said, "They were in effect equivalent to writing out-of-the-money put options on aggregate crises." In plain English, that means the insurers were selling assets that allowed their counter-parties to demand money from them if certain events took place. Initially, those seemed unlikely. Unfortunately, the events that would trigger the payout were defined so that when enough policies were written, only a general failure would suffice to pull the trigger. What wasn't obvious until later was that the combination of bank loan practices (required by regulators) and hedging strategies made that general failure more likely. An important thing to realize is that neither the regulators nor the banks realized when they introduced the instruments that this would be the outcome. They weren't knowingly making a heads-I-win,-tails-you-lose bet in the beginning. When you've only written the first few policies, the systemic risk is small. The risk grew as an ever-increasing number of institutions bought insurance against the same contingencies from the same counterparty. Normally, insurance companies reinsure to spread the risk to other companies, but AIG held all the credit default swaps itself.

Most of the recommendations from the NYU group make sense. More regulation, better regulation, or smarter regulators wouldn't have made anything better, and won't solve the problem next time either. Part of the dynamic with regulatory arbitrage includes regulatory capture: we have to assume that the regulations and the regulators will eventually be in the pockets of the regulated industry. The experienced people in any industry have the most information and interest in the details, so they're the natural experts to address any problems and to oversee any restrictions. The approach that can work is a set of regulations that requires visibility so that customers, clients, competitors, and counter-parties can see what their exposure is. These are the parties with a stake in the outcome who were stymied in the run-up to the debacle. Another principal is to change the nature of the compensation traders and bankers receive so that their incentives favor the long term stability of their institutions rather than short-term results. This is hard to set up, but possible: one idea is to pay bonuses in a long-term asset that vests slowly.

It's still a complex vexing subject, but understanding how regulatory arbitrage was an underlying cause at least provides a guideline to evaluating proposals to address the problem as we go forward. Bad solutions attempt to forbid certain kinds of actions or investments, since they provide an incentive to find a new kind of institution that can exploit the abandoned opportunity. It's better, when we detect a kind of transaction that is destabilizing in one way or another to find a way to allow it that makes its impact and extent visible and provides incentives to moderate the impact. That's not easy, and it's probably not the direction that regulators and legislators will want to go, but forbidding lucrative practices doesn't prevent them, it drives them underground and out of sight.

Monday, February 09, 2009

Iain Banks: Matter

Iain Banks' Matter is set in his Culture universe, and it's nearly as interesting as the best of the Culture novels. The high point of Matter is seeing people (and entities) at different technological levels interacting and moving back and forth between societies. The main focus is the Sarl, an industrializing society living on level 8 of Sursamen, an immense shellworld, and their war of conquest with one of their neighbors. The Sarl are getting help from a more advanced species, and being led to believe their own abilities are behind their successes. The royal family has a daughter Djan who was taken away and adopted by Special Circumstances, which gives a reason for them to have someone on the scene, even though Sursamen is outside their sphere of influence.

I think this is the first time we have directly seen that Banks' Culture universe includes competing civilizations at the level of the Culture. Previously we'd been led to believe that the Culture was an over-arching universal civilization that enforced a live-and-let-live stable peace among all species. Now we see that there are others at the level of the Culture, and the Morthenveld are powerful enough that the Culture mostly leaves their sectors and galaxies alone. When a Special Circumstances agent wants to visit a region under Morthenveld control she has to give up most of her powerful built-in weapons, skills, and communications devices.

Amid the local politics, and while following Djan's perigrinations as she attempts to return to Sursamen to find out what happened in the wake of her father's death, we discover that a deeper plot is afoot. Vast forces gather in darkness and secrecy, and Djan has to figure out what their goal is decide whether their plans are aligned with or opposed to those of the Culture before deciding what action to take.

The story has great descriptions of vast stellar constructions, many interacting civilizations, species and people. One of the things I like about the Culture novels is that even though we get to see grand conflicts in the foreground of the stories, it's clear that they are just the front page news of far bigger civilizations with vast abilities and gazillions of citizens, who mostly interact through peaceful commerce. We see a bit of that in a visit to a Morthenveld nestworld, which is huge beyond the scale of the entire Culture civilization.

While the book was nominated for this year's Prometheus award, I don't think it'll be a strong contender. It's perfectly sympathetic to libertarian goals and tastes, but the big conflicts don't directly apply to any hot-button issues. The book is very well written and a lot of fun to read, but this year there are books with a much tighter focus on out-of-control totalitarian governments to contend with, and Banks' Matter doesn't play in that arena.

Monday, February 02, 2009

Scott McCloud at TED

Scott McCloud, who understands comics better than anyone, gave a recent TED talk in which he explained comics while demonstrating how they work. It's a lot of fun.

Monday, January 26, 2009

Cory Doctorow: Little Brother

Cory Doctorow's Little Brother is a fun, modern, flashy, but stark warning in the vein of Lewis' classic It Can't Happen Here. Like MacLeod's The Execution Channel from last year, it takes place in the current anti-terror climate, but Doctorow focuses more on the consequences and costs of the resulting repression and shows how we (or at least those with a connection to technology and time on their hands, meaning smart high school kids) might fight back.

Doctorow even has an afterword from Bruce Schneier and a bibliography to underline the fact that technologies for privacy are available and urging people to do more to prepare for the resistance now. Since I can attest that all the technology in the book is either available or easily developed, the book has to be placed in the future history category to qualify as science fiction. It has been nominated for this year's Prometheus and I'd have to say it's the best candidate I've read so far.

The story is told from the point of view of a small group of teenagers in San Francisco who are imprisoned by DHS in a general sweep after a terrorist attack on the Bay Bridge and BART tunnel. Once they are released, they (principally Marcus Yallow, a hacker and LARP player) work to undermine the terrorist state and build tools that their friends can use to communicate privately and organize out of the government's sight. There are enough details about what tools they build and what systems they compromise to serve as an outline for budding hackers who aren't sure how to fight back. These details occasionally intrude into the story in the form of Marcus explaining things to his audience, but I think readers for whom they aren't obvious will take it as necessary background.

The story also shows clearly how torture can come about, and without taking their viewpoint makes the actions and motivations of the guards and torturers believable. Marcus and his friends react to their harsh imprisonment in a variety of ways, just as the people on the outside react to the increasing repression differently. It's not surprising that kids in school (who are used to hiding some of their activities from their parents and teachers) adapt readily to using surreptitious means to hide from DHS as well.

The story doesn't have a young-adult feel; even though the protagonists are mainly teenagers, the story is told with an adult sensibility. Being youngsters, Doctorow has plenty of opportunity to show them growing intellectually and emotionally. The characters are well filled out, and have appropriate conflicts that drive the story. The story is exciting and well motivated. Marcus starts working with a journalist early enough in the story that her role in the denouement isn't a surprise. And compared to the Queen's similar role in Walton's Half a Crown, it seems completely plausible.

I really enjoyed this book. It had some of the flavor of Vinge's Fast Times at Fairmont High, without being as far ahead of the curve, and gave an exciting depiction of the fight against an enveloping tyranny (harking back to The Moon is a Harsh Mistress) along with a concrete vision of how and why we'd fight back set in an all-too-plausible near future.

Tuesday, January 13, 2009

Jo Walton: Half a Crown

Jo Walton's Half a Crown completes her "Small Change" series with a distinctly different ending. The second novel in the trilogy, Ha'Penny, was a co-winner of last year's Prometheus Award.

Small Change is an alternate history in which Britain made peace with Hitler and itself moved toward fascism. The story follows Peter Carmichael, a police inspector in the first novel, now head of Britain's Secret Police, The Watch. Carmichael is gay, and this fact is being used by his superiors to keep him in line. But they don't seem to know that Carmichael has been secretly operating an underground railroad from inside The Watch. In Half a Crown, Carmichael additionally struggles to deal with the troubles of his protege Elvira Royston, a debutante planning for her presentation to the Queen. Elvira accompanies a friend to a political rally to watch the parades, and is rounded up along with the provocateurs after a riot erupts. The government decides to make an example of the provocateurs and Carmichael has to scramble to extract Elvira from the mess at great cost to himself and his friends.

The books provide a clear depiction of innocent well-meaning people getting caught up in a totalitarian struggle, and having to choose which of the things they value they will work to preserve and at what cost to their other values and to the rest of society. The first two books had downbeat endings, as Carmichael and others gave in on major issues that allowed the totalitarian government to take power in order to preserve a small amount of personal autonomy. This third book has the same feeling most of the way, but in the end Elvira finds a way to turn the tables and expose the machinations that led to the government takeover. I don't know if Britain's government really would work the way Walton portrays it, but as an American, it felt like deus ex machina.

The characterization is interesting. We've come to know Carmichael from the previous books, and his motivations (protecting both his lover Jack and Elvira, furthering the secret projects that allow some people to escape) are clear and well established. Elvira is a newcomer to the story, and Walton demonstrates her thinking and motivations clearly in alternating chapters that Elvira tells in the first person. The others, which mostly follow Carmichael, are given in third person, which allows Walton to follow other characters when necessary.

Half a Crown has been nominated for this year's Prometheus, and it's a strong candidate. The application to libertarianism is clear, but I think there are other books which will do better. Cory Doctorow's Little Brother seems a more powerful cautionary tale along the lines of It Can't Happen Here, though I'm not quite finished with it yet. The book is well written, and my only complaint is that the happy ending seemed forced. I don't think dystopias have to end in a downbeat to be effective, but the total collapse of Britain's fascist government seemed to run against a lot of previous description showing how that the government had co-opted most of the country's leaders and that the institutionalized prejudices were in harmony with those of the populace who were learning to get along with the other consequences of institutionalized repression. The quick turnaround in response to a single speech was a surprise. This is, after all, still an alternate history in which the Germans and the Japanese have taken over two thirds of the globe. Britain will have to figure out how to co-exist with an external world dominated by fascism.

Thursday, January 08, 2009

Diane Coyle, The Soulful Science

Diane Coyle's goal in writing The Soulful Science was to explain why economics is relevant and to convince readers that the bad rap that economics seems to carry is undeserved. She argues that modern economics studies important topics like the causes of growth and of happiness, and thereby provides important clues as to how we can relieve poverty and increase personal well-being. I think the book is a good introduction to the economic way of thinking that might draw a more humanistic audience into the field.

Coyle's approach is non-technical and engaging. The book has only a few graphs and charts, and I don't remember seeing any equations. Coyle tells the story of the people who made the advances that interest her and what their results imply about how we should organize society, what society can and cannot accomplish, and how these advances in knowledge affect our lives.

The first two-thirds of the book contains most of the meat. Part one talks about what economists have learned over the last several decades about what makes economies grow, and by implication, what we can do to reduce poverty where growth is still sparse. She starts out this section with a chapter extolling the virtues of the economic historians who have dug into obscure archives and surprising sources to collect detailed data about how people lived before record keeping was as persistent and consistent as it has become. This work required a fair bit of inference and a lot of perseverance, but in the end, we have a good view of how well people got on in different times and places. This data can tell us a lot about how quickly growth occurred in different circumstances. From this we can deduce a fair bit about when and where progress flowered, which shows some facts that seem to nearly always be true beforehand. (Protection of property rights, respect for education) But the inconsistencies; other societies that seem to have all the prerequisites, but don't enjoy the growth that occurred elsewhere show that it's an incomplete model.

In her chapter on alleviating poverty, Coyle admits that we don't have a formula that works reliably, but argues that we have identified some policies that are worth encouraging and we have learned about some approaches that don't work. If developmental economists and leaders of underdeveloped countries pay attention to these lessons they'll be able to lay some groundwork that will put them in a better position to take action as we continue to learn more. It'll also stop people from hoping for quick results from ideas that we now know don't work (natural resource discovery and exploitation seldom leads to improved living conditions for the populace.)

The second part of the book focuses on people, usually the province of microeconomics. But Coyle focuses instead on recent advance in personal happiness and work on rationality and biases. She does a thorough job of presenting an overview of the implications of recent research for how to live your life, and on what interventions actually make people happier. She also talks about how people behave in markets, and what the economists have figured out about our behavior based on theory and experiment. Theory has filled in the consequences of asymmetrical information: how it affects negotiations and outcomes, and how people gain an advantage in business or politics by controlling access to information. Experimental economics has taught us a lot both about how people actually act in market environments as well as what kinds of institutions and frameworks make markets more effective at allocating goods and how to reduce undesired outcomes like pollution.

The final section addresses larger topics that have been attracting economists' attention recently: how evolutionary and chaos theory can be used to enhance classical economics' approach to the emergence of behavior from collections of interacting agents, and public choice theory's observations about how incentives on bureaucrats as individuals reduce the effectiveness of government as a tool for effecting change. This section is interesting, but doesn't get into a lot of detail.

Overall, The Soulful Science is a well-written non-technical introduction to the most interesting fields in modern economics. Coyle is an insider and quite familiar with the personalities in the field and she presents recent findings well while giving a sense of how economists work together to develop and explore these ideas.