Shortly after my article was published, I got a tweet, from Eugene Mandel, who invited me to discuss the whole concept of data mining on Twitter.  That tweet led to a very collegial telephone conversation between us, in which we exchanged ideas which will provide the basis for this article.

Eugene is a very engaging young entrepreneur in Silicon Valley, who has been involved with a number of startups.  His current plans integrate very nicely with my concept of Twitter data mining.  My prediction is that his newest venture, mustexist.com, is destined to become an important tool for Twitter users.  He recently started the company with his pal and co-founder, Alex Sherstinsky, a Ph.D. from MIT.  These guys are very serious players.

Mustexist’s current product is called list tags. It allows one to take a twitter name, and creates a ‘cloud’ of lists in which that Twitter name appears, roughly signifying the importance of keywords in that person’s list audience by the size of the type in the cloud.

Here’s a copy of Glenn Beck’s tag cloud.  I shrunk it down to accommodate the page size, and its contextual importance in this article.  Beck is on some 3800 lists.  Politics, News, and political head the parade, but further down the list are lists named things like ‘nutcase’, and other less desirable things to be associated with.  Mandel goes through the Twitter API, and finds all the lists on which Glenn Beck is mentioned.  Then, he queries those lists and finds the most popular keywords in their title.  Then he ranks those keywords by their frequency of appearance in Beck’s universe of lists, and performs a simple statistical conversion to interpolate the list keyword frequency to the type size in the cloud.

Pretty neat trick, eh?

But there’s more: list_tags enumerates some of the lists on which the Twitter member appears.  Here’s a picture of the TopLists for “Politics” in which Glenn Beck appears.  Now think, for a moment how valuable this information can be.  Many of these “top” lists contain 400–500 members.  These are lists for the top topic which have the largest number of members – regardless of whether or not the subject appears on those lists.  That’s valuable information for marketers, or for researchers and even job seekers.  But more than that– everyone who is maintaining such a list is someone who has enough interest in your subject to go to the trouble of maintaining a list of people who he/she considers are the leaders in the field.  And if that’s not enough, the lists editors is another valuable source of fertile relationships to cultivate.

The second. longer list is a list of which Glenn is a member.  So, in the case of a celebrity, it will not be unlikely that the same names will be on both lists.  I’ll not reproduce the Lists for ‘politics’ because it is too long – and because in appearance it strongly resembles in format the ‘Top List’.

So the stage is set now with a resource now to empower Twitter users with access to many lists.  A little resourcefulness will enable you to harvest those lists, and, by extension, to knowledge of all of their membership.  This will give you first hand access to communicating with the most well-regarded people in any given field!  And with the LIstimonkey resource, you can monitor the conversations of those experts for keywords.  With a little software jiggering and less than an hour’s effort, using our Glenn Beck example, we could identify, harvest and communicate with thousands of political junkies.

Mandel, however, is talking about taking Mustexist to a whole other level.  Using the database he has assembled for the list-tag project, he is planning to offer in the near future an interactive, iterative way of ‘surfing’ the tweetstream of these lists, in much the same was that Google lets you surf for articles.  He made the analogy of a newspaper in our conversation.  Each newspaper has several sections: sport, business, local, etc.  If you accept the proposition that a list’s tweetstream is like a newspaper, then it would be very productive to skim through the newspaper, looking only for articles that interested you.  You could skip the car ads, and dating club ads, and focus on baseball, or international news.

Similarly, over a period of time, a list’s tweetstream will have a lot of chaff: someone’s opinion of last night’s American Idol, inclement weather, mother-in-law coming for a visit…  But what if you could adopt a list, or amalgam of similar interest lists?  What if you could in real time and on an ad hoc basis query the stream for stuff like: climategate?  cap and trade?  world health organization?  BTU content of coal?  And how powerful would it be if you could pursue your line of inquiry immediately, based on feedback you got from the previous question?

I think that MustExist is on the verge of somthing quite important that will add a new dimension of empowerment to the serious Twitter user.  There are currently some 23 million Twitter users, 75% of which are either classified as ‘addicts’ or ‘regulars’.  That’s a reliable 18 million pairs of eyeballs scanning and interacting with Twitter several times a week – sometimes several times a day.  Any tool that can reliably and easily segment relevant portions of that population and then address them is bound to be very successful.

Please add your comments.  It is helpful to hear the opinions of our readers!

I have developed a system for fitting our blog writing into our daily routine, and it involves the use of technology by gaining efficiency from productive software and streamlining the thought processes.  This helps get around the press of servicing clients in our software development and web business at Paladin Consultants, LLC .

Inertia and the lack of a clear idea of what to write about has been the number one blogging stumbling block for me, along with a cumbersome production process of the blog article itself.  I have solved this in several ways.

1. I found I had a lot of ideas, but didn’t have a way of recalling them when I needed them.  So I started using Evernote, which recently added a Blackberry extension.  Since I am never far away from my ‘berry, I can text, dictate, or send a picture to Evernote, and harvest those thoughts at the proper time.  I also am a heavy user of OneNote, so, on an as needed basis, I copy thoughts from Evernote to OneNote for a more permanent and well-organized record.  There is a good blog for OneNote, which talks about the many uses it has. 
2. I try to have a small camera near me all of the time.  At very least, I have my Blackberry camera. So, armed as such, it is easier to get illustrations of some of my topics and ideas.
3. Recently I have found that once I have decided on a thought or idea, it is productive to actively think about that idea while falling asleep.  During the night, somehow, the idea seems to be processed and examined from all sides, and, oddly enough, seems to be well developed my morning.

The next bottleneck is usually the development of the idea.  This is partially resolved by the sleep technique discussed above.  The problem with the sleep technique is that if you don’t capture it quickly, its persistence appears to be limited.  Use it or lose it.  So among the first tasks of the morning needs to be the capture of the previous night’s thought process.

I’ve had success with mind mapping techniques, especially for short blog articles.  For a longer treatise, a hierarchical outline, such as one might produce in Word would be more appropriate. Usually, I draw the mind map on a yellow college ruled pad, but I’ve gussied up the one for this article to make it visually appealing. See if it makes sense to you.

Once I have the ‘plan’, then it’s time to bring in the power tools to take care of the details.  I use BlogJet, a very convenient and inexpensive online blogging software, which lets me compose, spell check, insert artwork, links, tags, and virtually everything I need to do to produce the blog itself.  I felt that this piece of software reduced the amount to time I spend visually composing each article by at least 1/2 hour.  They also have a very nice feature which fits into Firefox, which allows me to copy and paste pieces of web pages into the text of my blog very easily.

Firefox is an extremely important piece of software for me.  It seems that no matter how hard Microsoft and Google try, they can’t quite come up to the standards of the Firefox Mozilla browser:  Mozilla always seems to be one step ahead of them.  There are two Firefox  plugins that are particularly effective for me: Taboo, and SEOQuake.

Taboo is a software tool which allows you to take many tabs open in Firefox, and store them in one tab as thumbnails.  So, instead of having 20 tabs opened, you might only have two: the one you’re immediately working on, and the other containing the Taboo ‘reservoir’ collection.  Taboo allows you to temporarily store dozens of pages in this thumbnail format, each image growing as you mouse over it.  It saves looking through the tab array, and saves scarce memory.  A very useful tool!  I have a Bookmark category called ‘Blogging Tools’, which I open in tabs  (about 10 sites), then annex them to the Taboo page.  Makes for very efficient research and fact checking.

SEO Quake, is another Firefox app that analyzes the keyword/ key phrase content of your article.  At the end of writing the article in BlogJet, you can save it to a temporary HTML file, and use SEO Quake to appraise it for links, key phrase effectiveness before you publish.  And it always helps to have a list of your Wordtracker phrases to rework your article against.

In the Pantheon of Tools I rank Delicious near the top.  It’s a very powerful tool which lets you save sites, software techniques and articles, like your Browser Bookmark feature.  Except that your Delicious sites are available to you whether you happen to be at your computer or not.  Available on client machines, Available on your Blackberry or iPhone. Available on your laptop.  Available in subgroups as RSS feeds of your tags, of as subscriptions to the tags of the rest of the universe.  An extremely powerful element in anyones toolkit, and a great contribution to social knowledge.

In addition to photographs, I try to include portions of scanned documents like software code where appropriate, to illustrate my point, as well as screen clips, video clips, and charts.  I find that Google charts allows one to create some very nice looking graphics with ones own data.  Those graphics, though not ornate in any sense,  are far more professional looking than those produced by Excel.

Finally, if you are not a strong typist, or are composing at the keyboard, while doing your research at the same time, Nuance’s Dragon Naturally Speaking is a good candidate for you.  I confess to using it sometimes to squeeze out a quick article when I am short on time.

Please leave a comment – comments help us do a better job!

These advances in technology bring benefits and challenges for small businesses like ours.  First, we much do the work for our clients for which they have hired us.  That’s how we get paid and earn our living.  Second, we must find new work, because when current projects are finished, we will need new ones to take their place.  Third, we must continue to learn so that we do not become obsolete and unable to compete.  Unfortunately, only one of these items is directly revenue producing.

Way back before the turn of the century, I remember reading that the amount of knowledge in the world was doubling every five years.  On a visit to my Uncle Joe, who was a senior chemist at duPont at the time, I asked him if he thought this was a true statement, how one could possible know this, if it referred to unique knowledge or many folks knowing the same thing, and all the things a curious kid might ask.  He allowed that he wouldn’t be surprised if it were true, and mused that perhaps it could be measured by the amount of ‘stuff’ written in scientific journals, etc.  The DNA molecule had been modelled, we had put a man on the moon, developed nuclear weapons, and so on.

If it were a true fact them, it is certainly doubling much faster now.  We have the Internet, digitized books and other media, catalogs – all manner of convenient and expanding knowledge and technology.  And there is every indication that these trends will continue- perhaps at an even faster rate.

Animal behaviorists have long been aware of the ‘Honey Dance‘ that bees to to alert other bees to the location of nectar.  A bee will find a source of nectar, then go back to the hive and do a dance that shows the other bees the distance and direction of his (there’s only ONE queen, and SHE doesn’t look for honey!) find.  The bees have developed a way to share information, and use communal intelligence and collective effort to maximize their efficiency in gathering food for the hive.

This brings us to the human honey dance!  Todays writings in blogs and websites in continuing to explode over the internet.  Now I am sure that there are many folks writing things out there that may be of great value to me and my company.  But I have no idea who or where.  That’s where the social networking sites come in.  The early entry was Facebook.  My daughter has a Facebook account since high school, and has about 1000 ‘friends’.  They’re not real friends, of course, but the culture of their tribe mandated that ones population of Facebook ‘friends’ was an indicator of popularity, and therefore success.  But they are almost 1000 people that she can correspond with, ask advice from, and, perhaps have commerce with.  And maybe that’s not so bad a thing!

Facebook was followed in the business worls by LinkedIn.  Now LinkedIn, though limited to 3000 invites per person, can theoretically give you access to 27 billion connections (not counting the folks whi invite you)!  Twitter is another source of these ‘loose connections’.  I’ve seen some folks on twitter with 50,000 plus followers.  The statistics are unclear about how all these connections translate into actual business.  And I do not see managing connections in numbers anywhere near these.  But maybe it’s not about managing connections.  I’ve met some very interesting and smart folks through LinkedIn and Twitter, and have given and gotten a fair number of ideas.  I suspect that in the longer run, within your network of followers, the folks who are able to share new ideas through their reliable ‘Honey Dance’ will eventually earn the respect of the other players in their network, whatever it’s size.