Information Evolves and Other Stuff

I've learned to avoid precategorizing anything in my bookmarks.

I don't make a category unless it is necessary, unless I am using it. For example, I need to bookmark Amazon web services, so I create an Amazon Web Services folder, but I don't create a Web folder, with a Web Services folder inside, which I then put the AWS folder in. I don't have any other web services bookmarks yet in Google Chrome so I leave this for later. It just creates more folder depth to dig through before it's needed.

I also try to avoid adding a bookmark just for reference. That just leads to clutter, where I can't find the bookmarks I use on a daily basis, because when you categorize information according to its classification or how it relates to other information, you lose how it relates to you, to useage. For example, if in browsing the web I find a half dozen interesting resources on manual focus lenses, but for cameras I don't use, the bookmarks will obscure the resources I use for manual focus lenses for cameras I do use. What I do now is add bookmarks only when I use the content or need the content now, not for reference or anticipating future use, placing them in the categorized heirarchy. The others I place in Uncategorized (what a wonderful idea, that Uncategorized anti-category!) awaiting the day they become useful and can be categorized, or I place them in a special heirachy called Reference. I don't know if a parallel heirarchy will work but it does keep them out of my way.

I hate digging through deep categories. Yet, for proper categorization to find things later, when you've forgotten where they are, they need deep categories. If I have have five different web services providers, each one needs its own folder and there will be clutter if I just create them all at the same level. So I need to create a Web Services folder, which then adds another annoying, slowing, confusing layer to finding what I want and to my thinking. I want Amazon Web Services when I want it, not digging through Web, Web Services to get to it. What if I use it every day? I have to dig each time.

This brings up another issue: information structures evolve.

One of the problems library scientists create is through this need for pre-creation of categories. They must predict every category that will be needed ahead of time. I was once told I needed to create "name authority records" for every photographer in a database I envisioned of 19th century photographers, before a database collecting names from old card photographs could be built. At that rate, the database would never be built and besides the whole purpose of the project was to collect the names so we could see who was doing what and look for patterns. If we had an authoritative name for each one, we wouldn't be doing the research.

Don't engineer. Evolve. Evolve. Evolve.

We don't need architects and engineers, we need some new job description with a new name, evolvineer or something, for the person who creates a framework for information evolution (maybe like the game Spore?). Perhaps databases like multivariate or Lotus Notes will help get us there.

Social File Sharing

I recently received an email from a well known supplier of enterprise level file sharing systems. In the enterprise, one solution is called "Wide Area File Services." There are other ad hoc solutions. I am not very familiar with the details of these systems, but understand some of the problems they are trying to solve. A corporation wants fast, simple access to files from any location (anywhere their employees are) while ensuring users access a single version of the file. The are also concerned about the cost of bandwidth (which ensuring a single file helps, since users normally waste resources copying and forwarding a file or video by email, since they are really not aware of the consequences and generally do not understand they can just forward a link).

Although these issues are important, I think this perspective misunderstands the most important need today. Corporations are always concerned about meeting requirements, being defensive, controlling their population of employees more than they are about doing something new or finding new and better ways to do something. They are blinded to solving the problems of how to do more things better by the need to clean up the messes their productivity and growth creates. This is why they are so often blindsided by innovation.

What we really need is social file sharing. What good is sharing a file, a digital photo, video or spreadsheet without knowing who it came from and what group it belongs to? It starts with a simple idea:

Every piece of information should be accompanied by the identity of persons or group to which it belongs wherever it goes.

I've given this issue some thought before, but the email reminded me of it. By "belongs" I mean to include both the individual or the group to which the file is associated with in a given social network. For example, we already see an example of social content sharing through sites like Facebook. Of course, YouTube is also a kind of platform for social sharing of content, but there the concept of "file" or a package of information anyone can take with them and carry it onto their PC or laptop or cell phone or save on a CD is missing. And no, YouTube is not good enough. What we need is a way to retain a media object's social connections as it is transferred from system to system. To do otherwise would imprison files on their respective platforms.

When I upload an image to flickr, any social connections formed around the image is contained within the flickr ecosystem. If I download the image and then share it with someone, the social connections are lost. If I share the original image with someone by email, it lacks the social connections the version on flickr acquired. Why can't all these versions of the image somehow carry social connections the same way EXIF data carries meta data about the production and authorship of the image?

Maybe someone is working on this right now, perhaps a modification of existing RSS standards to allow social network information along with an attachment, creating a kind of "podcast" that could bring social data along with the file. Maybe Google's open social network framework is looking at this. But whoever does it, it is important that it gets done.

Out of many, one: The acceptance of many views.

I've talked before about the need to accept the inconvenient existence of multiple of truths that exists in genealogy. Incomplete knowledge about the past is unavoidable. The past is gone and we are not getting back to put under a microscope. Even the present is difficult to pin down. We only know what we experience or someone tells us, which is pretty much what we know about the past, only through source material and what someone tells us. We are left frequently with only sketchy knowledge about family history. This leads to different families claiming the same individual, each with their own basket of evidence and story. I've learned to accept this as a reality and moreover, I've learned to accept this as being a Good Thing (or at least the best thing we can expect given the nature of reality).

The net it turns out is very good at handling incomplete information as it rapidly emerges and changes from multiple authorities. The applications emerging ont he web are gradually all taking on a similar shape. They all in one way or another incorporate the acceptance of many views. The wiki synthesizes a single view out of the many views of its authors. Social bookmarking (and other social networking) sites allow multiple "truths" to exist within the same space. The social network creates an ecology where authority can develop implicitly, without saying. Most of the social networks incorporate the many views or truths into some kind of aggregate view that is useful, a kind of single view out of many. This represents a democratizing of knowledge, but I hesitate to call it democracy since that is just one particular method for synthesizing a single truth out of many views. Democracy works in a very crude way by voting and we know that voting systems are subject to gaming by malicious people and other flaws. The kinds of systems, wikis, social networks, voting systems used by various collaborative news sites all represent vastly more sophisticated methods of synthesizing a single view out of many than democracy, which is relatively weak and produces a "tyranny of the majority" when not mediated by some system of individual rights.

I was explaining how social networking works to Tom Davenport today in regard to a farm website we are developing. I explained to him how if he had an account on a social bookmarking site, he would for his own benefit maintain and organize his bookmarks online. He would bookmark sites on pork and beef as he does now in Firefox. To do this he would create tags for Pork and Beef, organizing sites about pork and beef under those headings. Because the bookmarks are shared publicly and the tags exposed to to browse and search, a person can click on the Beef tag and discover his bookmarks (among others sharing their bookmarks). That person might click on his user profile to look to see what links he has on Beef. They might find his bookmarks are highly reliable and useful. Therefore, the user would be likely to turn to Tom's bookmarks when looking for accurate information on beef and cattle raising. They would not necessarily even know that Tom is a farmer, but they would discover him as an authority simply by observing the quality of his bookmarks on the topic. Tom Davenport implicitly becomes an authority. He implicitly shares his expertise with others. All without declaring himself a farmer or an expert on anything. Of course, he might mention in his profile he is a farmer; he might link to his farm site and you might have more reason to trust his bookmarks.

I tell this story because it illustrates the acceptance of many truths that lies behind the way the web works today. There may be ten thousand people on a social bookmarking site who think they know something about beef. Each may have a different idea of how to raise beef. Their bookmarks will implicitly reflect their knowledge, experiences and differences of opinion with others. The gestalt of the social network will reflect this diversity. The more accurate providers of bookmarks will become popular, the ones with less accurate bookmarks, reflecting radical, not very useful or very different views will remain less popular. One might object that this creates a kind of stagnation on popularity, but in reality it relates directly to the idea of the "Long Tail" where more people may be accessing the less popular bookmarks more than the popular. So the social network embodies two kinds of authority simultaneously. The authority of popularity and the authority derived from the long tail...the authority of individualism, of the disruptive idea, gives freedom to both kinds of authority and the freedom to move back and forth between the two kinds of authority...for the disruptive idea to start as a seed and grow to an oak, to move from being "indie" knowledge to "popular" knowledge all within the same framework.

It is fascinating that the web reflects this reality by its nature. That a concept coming from an obscure activity like genealogy is moving to the center of intellectual pursuits. That it can create a framework where out of many views a single truth can emerge without denying all the other views. It reminds me of the vast jumble of "junk genes" that we carry along in our DNA from our distant past, which are there because they might just come in handy some day. It reminds me of how organic the web is and utterly incomprehensible within the old framework of bell, book and schoolhouse knowledge it is becoming.

Only something organic can be becoming. And the web is always becoming. Always becoming something. A book is never becoming, it only was or is. Scholarship is locked into this model since the Enlightenment (oops, the E. slipped in there...was hoping not to mention it), what it means to posses knowledge, to share knowledge, to build knowledge and discover the truth is all changing now that we are connected to knowledge on the network. So strangely different than books. I've rambled enough for now and must retreat to the high tower of Brandymore again for the night.

The Swicki: Collaborative Search

I've long thought that the fatal flaw of what library science calls "finding aids" is that they only organize information according to how it relates to other information. What I've always wanted is a search that relates information to what I care about, to my interests, to me. I've thought about "personalized searches" but the trouble with this approach is that it is time consuming to express to a computer exactly what it is you want. You must set up some kind of criteria and the the search returns results for you based upon it, such as the simple eBay search notification. If lots of people are going to use this with the efficiency they now get from Google, something else is needed. We don't have expert systems and artificial intelligence yet, so what is a possible solution?

Some are experimenting with attention. By tracking what you look at online, a profile of your interests can be built, which can then drive a personal search engine. People are really bad at expressing what they really want (as product developers and marketers have discovered) so the non-intrusive method of observing behavior may just work.

Another experiment takes a different approach. Why not let other people help refine your search? That is what the people behind Swicki seem to have done. If you could gather like minded individuals into one location where they could influence the accuracy of the search, the "search in a can" model could be improved. It goes farther than that. By becoming an aggregator of search results the system can ride on top of the web and use it as a database in a way similar to Yahoo Pipes. The most revolutionary aspect of Swicki is user created search engines. Instead of needing millions of dollars and massive servers, Swicki piggy backs on existing search results to enable anyone to create a web search engine. This kind of democratizing is a defining quality of web two point oh applications.

I see how the canned search model can be turned inside out, by allowing a group of users to collaboratively refine the canned search to improve it. Instead of empowering the computer to be smarter, it empowers people to create a smarter resource. It definitely becomes a kind of search-wiki. It competes in some ways with the idea of folksonomy. We have now user created taxonomies and now user created searches. What I like about both developments is how it democratizes the organization and finding of information. The folksonomy enables people to create their own vocabularies, perhaps multiple vocabularies for the same subject area. The wiki search enables people to create alternative search results for the same subject. My background is in a subject where unresolvable disagreement is commonplace. It's called genealogy, where there are no facts, only interpretations and sometimes two families claim the same individual. This is not something for concern in genealogy and I like the way more than one truth can exist within the same framework, it's much better than declaring one view right and all others wrong and working hard to keep your opponent's views out of view. Despite what some may claim, there can be more than one version of the truth. Let an idea gain mindshare on its own merits.

I've thought before about a search engine where you could search the web by creating predefined searches, but never thought of letting everyone edit your predefined searches, that is novel, just as with social bookmarking

You can see the Swicki I created in a few minutes in the sidebar of this site (as long as it's there).

A Good Introduction to Yahoo Pipes

A good introduction to what Yahoo Pipes are and the possibilities they offer posted to Read/WriteWeb. Also, a simple, clear, concise explanation of emergence:

It is this automagical process through which elements of a system give rise to a higher order system. Emergence is how physics becomes chemistry and chemistry becomes biology.