Stackoverflow.com

There is a good article on the principles driving the development of stackoverflow.com, a site where programmers get help with their coding problems on ReadWriteWeb.

I was particularly struck by the design points where Spolsky highlights the frustration created wrong answers and obsolete results.

I can remember when I was able to circumnavigate the web through a search engine for the topic of history of photography. It was that small. I could see everything there was to see about history of photography online in a week, a week of drudgery wading through duplicate results page after duplicate results page, until I had made sure I had seen everything about my topic. Although filled with a fair amount of junk and duplicates, I was still able to find a single web page if it contained sufficiently unique keywords, until about a year before Google emerged, I had relied on AltaVista to take me back to a web page in one go, when I could not remember where I had found a code solution on some obscure personal page, for example. Then the search engines began to fail me, and single pages I had found before became nearly impossible to find, but eventually, search engine technology improved and with Google, you could find that one blog page with the coding. That was one the solution to the problem of finding things.

Spolsky is right to observe the problem now is that search is failing to distinguish between correct and incorrect answers; between current and obsolete answers to technical questions.

When I first started programming using Microsoft Visual C++ (I was just a dabbler), I had a question about how to render bitmap graphics. I turned to the library of articles and code intended to help developers. I was happy when search quickly turned up an article on how to introduce bitmaps into your application. After an hour or two of reading, it slowly dawned on me the author was not talking about what I was familiar with, Microsoft Foundation Class applications. I was seeing unfamiliar code and unfamiliar techniques. I glanced up at the date. The article was from the mid 1990s. It was about coding C under Windows before MFC was introduced. The first, supposedly most relevant, documents search had brought up from MSDN was completely obsolete and about coding without an application framework. I had wasted hours reading the wrong articles.

Stackoverflow.com is an example of a great site. It is well designed, the developers learned the lessons of the last fifteen years of web technology and applied them. It is clean, beautifully presented and well organized site. I have to admit they did right what I failed to do with phphelp.com, which started by envisioning many of the same goals. They had to courage to go ahead with "soft security," collaborative editing, and content surfacing and valuing through a user voting system. Of course, with the volume of content and edits, such tools are necessary. What two humans could watch and police such a flow of content while doing their day job? User contributed and curated content is the only rational answer.

(By the way, it would probably be better to describe their principles as being informed by behavioral economics or an evolutionary branch of the field, than anthropology or social psychology, I feel the way people use voting systems to surface content, how "soft" social engineering strategies are employed on wikis, etc. to be close to the phenomena studied by behavioral economics, not just financial choices.)

A Twitter Wiki

As the popularity of short, fragmentary messages grows, I have become concerned the public conversation may lose the capacity for thoughtfulness and reflection. At the same time, I would like to caution those who condemn Twitter or other systems based on micro content to not throw the baby out with the bath water. The long form newspaper article found in the New York Times or Washington Post contains a lot of material used to provide background for the reader, often at the end of the article. Not only is this text boring and redundant to the knowledgeable reader, it takes up previous space. The one thing the web is good at is connecting one piece of knowledge to a broader context of other pieces of knowledge. There is no sane reason to continue repeating background and further reading material in a long form newspaper article when on the web, a writer can simply link to the information.

The brief, concise texts of micro content can be connected to many other sources of information, some just as concise (a kind of "blizzard" of small pieces connected loosely) as well as to other longer, deeper and reflective sources. This loose, disjoint and connected type of writing is simply the network native way of writing and connecting information. It is beneficial, as long as both kinds of writing and forms of content are available and can be connected.

My concern is really with lowering the barrier of entry, enabling and encouraging those longer, deeper and reflective forms of writing. I recognize that there are benefits from shorter, more concise writing, which leaves redundant, expansive or source material hidden (properly) under a link or conntected through a network of tags or a network of people. Perhaps will will see fewer long texts divided up by headings and sections and more smaller texts connected together through search, tags and linkages into a variety of wholes, determined by the user's interests and needs.

About ten years ago, I was fascinated by the idea of a long text (article, book, etc.) entirely constructed of fragments, similar to the kind of texts you see posted on Twitter today, which could be freely rearranged similar to those magnets used to write poetry on refrigerator doors. I imagined that instead of writing a large text with a single coherent whole, they way books have always been written, the pieces of information on a topic could be combined to create a "book" in innumerable ways by rearranging those pieces.

It would be like taking all the paragraphs in a book, shaking them out on to the floor, and then allowing or enabling those pieces to be rearranged for each reader or interest. The pieces would be tied together by keyword or by search result and only lastly by links. I coded a small prototype application called Strands to test the idea, but work and life caught up with me and I shelved it. I was and am still surprised by the ease and rapidity with which people have adopted Twitter.

Not only are people using Twitter, despite the fragmentary nature of its texts, they are participating creatively in shaping the technology and usage of this kind of system based on fragmentary texts.

The use of tagging emerged spontaneously from the user base. Using "hashtags" brief texts can be connected to media, such as images and video, with the tag at the center of a network of content.

Also, I've noticed users are starting to fit the tag word into their text. Some examples are:

"Young Nebraska farmer explains how limiting direct payments would affect his #farm at www.nefb.org"
(Tweet from http://twitter.com/farmradio)

and

"farmanddairyGet four issues of #Farm and Dairy FREE! Click on the big promo on our home page: http://www.farmanddairy.com/"
(Tweet from http://twitter.com/farmanddairy)

At the heart of my Strands prototype were small texts connected by keywords. I wanted to create the lowest possible barrier of entry, so a user could create a keyword (essentially a tag, I called them "strand words") just by writing it into the text. In this system, what was essentially a tag was created by writing it (texts were scanned on post or edit for the presences of tags and any new ones added to an index), which is hauntingly similar to how people have started using tags on Twitter. They started out adding the tags to the end of a message, but have now begun incorporating them directly into the flow of text. I hesitated to continue working in this direction on Strands, partly because I expected people would find the tags sprinkled through the text troublesome.

My current interest is in providing tools or ideas that will encourage and enable a society addicted to short messages, however beneficial they may be, however native to the networked way of writing and reading in a connected fashion, to engage in greater contextualization and thoughtful reflection, to enable collecting some of the knowledge quickly flying by in the "Twitterverse" into slower, more reflective pools of knowledge, like eddies on the edges of a fast flowing stream.

The first tool I want to build is a "Twitter Wiki" enabling anyone to associate a text of any length with a Tweet and anyone to edit it. If I have the energy, I will post any experiments on my site or at least attempt to describe it.

The Folksnet: Folk Culture and Web 2.0

Although I am not a folklorist, through my work on the Follkstreams website, I have come to appreciate the study of folk culture and understand how expansive a field it represents. Folk culture is the culture people make for themselves and share with others. Web 2.0 is a folk culture, it even coined a phrase "folksonomy" to denote a system of categorization that replaces a vocabulary controlled by an authority or group of authorities with a vocabulary created ad hoc by the people involved in using the system of categorization. The media sharing sites like YouTube and MySpace and "mashup" systems like Yahoo Pipes and Microsoft Popfly enable a rich shared folk media culture to arise. Increasingly, as we head into the 21st century, it is a supreme irony that folklore is taking on greater importance as nearly everything is democratized and individualized, and is poised to become perhaps the most significant field of research in this century, after having for centuries remained an overlooked backwater in academia.

Folklore already possess the vocabulary and methods to comprehend and measure such a brave new world. It has the attitudes and assumptions, the knowledge frameworks ready at hand. As we understand how the mind works, as we democratize, we will realize the central role played by narrative in the workings of the mind and society, that society is a kind of virtual reality similar to the internet in which abstract things affect the real world, that narrative and oral tradition exist everywhere and explain how people know what they know and apply what they know to the world around them. This is a profound shift from the rationalist assumptions that have driven Western civilization since the beginning of the Enlightenment.

Out of many, one: The acceptance of many views.

I've talked before about the need to accept the inconvenient existence of multiple of truths that exists in genealogy. Incomplete knowledge about the past is unavoidable. The past is gone and we are not getting back to put under a microscope. Even the present is difficult to pin down. We only know what we experience or someone tells us, which is pretty much what we know about the past, only through source material and what someone tells us. We are left frequently with only sketchy knowledge about family history. This leads to different families claiming the same individual, each with their own basket of evidence and story. I've learned to accept this as a reality and moreover, I've learned to accept this as being a Good Thing (or at least the best thing we can expect given the nature of reality).

The net it turns out is very good at handling incomplete information as it rapidly emerges and changes from multiple authorities. The applications emerging ont he web are gradually all taking on a similar shape. They all in one way or another incorporate the acceptance of many views. The wiki synthesizes a single view out of the many views of its authors. Social bookmarking (and other social networking) sites allow multiple "truths" to exist within the same space. The social network creates an ecology where authority can develop implicitly, without saying. Most of the social networks incorporate the many views or truths into some kind of aggregate view that is useful, a kind of single view out of many. This represents a democratizing of knowledge, but I hesitate to call it democracy since that is just one particular method for synthesizing a single truth out of many views. Democracy works in a very crude way by voting and we know that voting systems are subject to gaming by malicious people and other flaws. The kinds of systems, wikis, social networks, voting systems used by various collaborative news sites all represent vastly more sophisticated methods of synthesizing a single view out of many than democracy, which is relatively weak and produces a "tyranny of the majority" when not mediated by some system of individual rights.

I was explaining how social networking works to Tom Davenport today in regard to a farm website we are developing. I explained to him how if he had an account on a social bookmarking site, he would for his own benefit maintain and organize his bookmarks online. He would bookmark sites on pork and beef as he does now in Firefox. To do this he would create tags for Pork and Beef, organizing sites about pork and beef under those headings. Because the bookmarks are shared publicly and the tags exposed to to browse and search, a person can click on the Beef tag and discover his bookmarks (among others sharing their bookmarks). That person might click on his user profile to look to see what links he has on Beef. They might find his bookmarks are highly reliable and useful. Therefore, the user would be likely to turn to Tom's bookmarks when looking for accurate information on beef and cattle raising. They would not necessarily even know that Tom is a farmer, but they would discover him as an authority simply by observing the quality of his bookmarks on the topic. Tom Davenport implicitly becomes an authority. He implicitly shares his expertise with others. All without declaring himself a farmer or an expert on anything. Of course, he might mention in his profile he is a farmer; he might link to his farm site and you might have more reason to trust his bookmarks.

I tell this story because it illustrates the acceptance of many truths that lies behind the way the web works today. There may be ten thousand people on a social bookmarking site who think they know something about beef. Each may have a different idea of how to raise beef. Their bookmarks will implicitly reflect their knowledge, experiences and differences of opinion with others. The gestalt of the social network will reflect this diversity. The more accurate providers of bookmarks will become popular, the ones with less accurate bookmarks, reflecting radical, not very useful or very different views will remain less popular. One might object that this creates a kind of stagnation on popularity, but in reality it relates directly to the idea of the "Long Tail" where more people may be accessing the less popular bookmarks more than the popular. So the social network embodies two kinds of authority simultaneously. The authority of popularity and the authority derived from the long tail...the authority of individualism, of the disruptive idea, gives freedom to both kinds of authority and the freedom to move back and forth between the two kinds of authority...for the disruptive idea to start as a seed and grow to an oak, to move from being "indie" knowledge to "popular" knowledge all within the same framework.

It is fascinating that the web reflects this reality by its nature. That a concept coming from an obscure activity like genealogy is moving to the center of intellectual pursuits. That it can create a framework where out of many views a single truth can emerge without denying all the other views. It reminds me of the vast jumble of "junk genes" that we carry along in our DNA from our distant past, which are there because they might just come in handy some day. It reminds me of how organic the web is and utterly incomprehensible within the old framework of bell, book and schoolhouse knowledge it is becoming.

Only something organic can be becoming. And the web is always becoming. Always becoming something. A book is never becoming, it only was or is. Scholarship is locked into this model since the Enlightenment (oops, the E. slipped in there...was hoping not to mention it), what it means to posses knowledge, to share knowledge, to build knowledge and discover the truth is all changing now that we are connected to knowledge on the network. So strangely different than books. I've rambled enough for now and must retreat to the high tower of Brandymore again for the night.

The Acceptance of Oracles

In a previous post I said no one truly knows anything. Of course, we must use reasonable knowledge. The knowledge we act on is always imperfect, it's important to recognize that, since many people go about their lives believing they are acting on evidence. I doubt anyone really makes their daily decisions on evidence.

There is always a smarty pants who says that saying "nothing is truly knowable,"is just a high toned stance of philosophers and not really anything people in the real world should go by. The problem I have with such statements is that there exist people in our world, called rationalists, who pretend to act only on evidence in their decisions and lives. They act as if everything is knowable and that everything they know comes from evidence. They are frequently vocal advocates of science and reason as the only legitimate truth. That may be, but since we know reason is subject to fallacy, and that reason is a product of the human mind, it lacks the power to discern truth. Reason is a tool to be used to extend scientific knowledge, but it requires experiment, observation and prediction to verify that flights of reason are representative of reality. Even then, this mechanism may fail since we may lack complete observations, miss the one exception that proves the rule or disproves it. We may look at the data we have and say, continents never drift, then look again at later data as see that nevertheless the move. Science isn't truth. It's a successive approximation to the truth. It may be reasonable to trust science more than what someone says to you on the street, or what the good book tells you about the creation, but that does not mean either of those sources are wrong. In fact, if the lesson the Getty museum learned over the purchase of the Greek statute (Gladwell, Blink!), which science "verified" and an expert determined was a fake from a single glance lasting a few milliseconds, has anything to say about truth, is that sometimes individuals without any evidence are more accurate than scientists with rooms full of instruments and the wrong assumptions. It may just be, we have to recognize that there are times we must depend on oracles.

Some day, we may be required to accept the pronouncements of software intelligence, without any evidence at all, maybe not even evidence we could comprehend, with our limited mental capacity.

Citizendium: Multiple Truths Welcome?

I've been reading the Citizendium's approach to governing what goes into their content. This new attempt at a wiki encyclopedia favors an approach with less rules, greater oversight and tries to accommodate a multiplicity of views on truth.

http://en.citizendium.org/wiki/CZ:Neutrality_Policy

I wish they would not call it a neutrality policy. I strongly dislike the oxymoronic "neutral point of view."

I don't know how far the Citizendium will go, but I do believe multiple truths should be represented transparently. There is no reason not to accommodate multiple truths and no reason not to build information systems capable of accommodating multiple truths.

Contrary to popular belief, there does exist more than more than one truth. In genealogy, for example, the idea of multiple truths is necessary, since the same individual is frequently claimed by more than one family. Information about the past is sketchy and subject to interpretation. When the first online genealogies were being discussed on the GenWeb mailing list, it was ultimately concluded that there should not be a single unified global genealogy, since the "facts" could never be reconciled perfectly. There would by necessity be a need for representing multiple truths, based upon facts weighted by how much confidence we have in the sources (familiar to anyone who sources their genealogy).

Moreover, the truths we hold in our minds are imperfect, and emerge from our folk knowledge, through narrative and are based upon assumptions, which generally are made not on evidence (and probably can never be made upon anything else), but on the folk knowledge we absorb from our surroundings.

An absolute truth may exist and be determined by the physical universe, but there are many questions that arise about the human mind, society and the constructions of the human mind, which society is an example, which have no physical existence at all and it may never be able to determine what is true. There is also the nature of our knowledge existing only as sense perceptions, which makes science a kind of honorable delusion, as accurate as we can determine to agree on shared descriptions of phenomena. We have a reasonable idea of what we know is true or not through careful scientific inquiry, assisted by not guaranteed by reason, but in the end the only thing we know is: No one truly knows anything.