We are using the gauges image as a metaphor for thinking in classes and objects, which, in the software world, are the building blocks of abstract thinking.  We are doing that because it is easier to demonstrate an abstraction when it applies to something that you can see, rather than something that is invisible.  Notice the commonalities of the gauges objects: they all are processes which convey information from the automobile to the driver.  Although they differ in their detail, they all possess the property of being able to convert the voltage into usable information.  They all have scales, labels, pointers and the necessary equipment or functions to convey their respective information to the driver.

In a smaller sense they do have meaningful differences.  Their scales are different; their labels differ; and  their sizes differ.  And depending on the model of car, their shapes and colors can be different.  However these smaller differences can either be built in to the classes that define the gauge, or passed in as parameters.

It is the commonalities of these structures it would make our gauges and excellent example of a candidate for an abstract object.  Software designers refer to code that is a blueprint for these objects as ‘classes’.  Objects can also possess powers known as functions or methods which enable an object to perform its expected tasks.  In the case of the gauge object these methods are limited to those that receive and transmit information.  The gauge objects all receive information in the form all of voltage from a wire connected to them.  And they confirm that information and output it to move their needle pointer to an appropriate level all in the gauge’s scale.

Notice that in the case of the speedometer and tachometer, there are additional digital gauges which are used to convey the information of mileage.  However even in these embedded digital gauges, information is derived from the same voltage current sources.  It is also significant to note that these embedded gauges demonstrate another important principle of abstraction: that classes can be and frequently are composed from other classes, and can inherit and/or override properties from their more primitive ancestors.

It is this object oriented approach which is the hallmark of modern database and software design.  The practice of discerning, building and re-using practical and useful software objects creates synergies and efficiencies for both the software design process and the client, bring value to both.  It is also the best way to evaluate and test the integrity of the output, and maintain or modify the resultant code.

The ability and skill with which a software designer/consultant is able to visualize and perceive abstract components in the systems which the software designer is modeling is the most important and direct indicator of the web development consultant’s value and integrity in the quality of his work.

Long ago, before the PC had gained acceptance in the computing community, I was engaged in developing a well funded futures trading software platform using the Unix system which interacted with data stored (as I recall) on an Informix platform.  The workings of the program were in C language, and the most convenient method of writing the code was a program called Vi, which came as part of the Unix package.  It was, I thought at the time, a prodigiously inconvenient and non-intuitive piece of software.  Very soon thereafter, PCs gained acceptance within the organization, and we were able to port the C code over to the PC, adopt another more convenient tool for programming, called Personal Editor (PE) which had a much more friendly interface.  Although the C programs were written on the PC, they were compiled and run on a mainframe and we happily bid a fond farewell to the quirky Unix system.

In parallel to this migration, Linux came of age, and the Vi editor was replaced by Vim (Vi improved).  Unix and Linux acquired a GUI capability, and the developers and programmers grew in their expectations and level of sophistication.  I was recently re-introduced to the Vim editor, which has a graphics component.  And the quirky commands with which I grew so frustrated years ago, I quickly came to appreciate.

And I have spent the last several weeks experimenting and out-fitting my Vim editor with a set of new and powerful tools which have made my life a lot easier.

At Paladin, we do a lot of database programming to which access is usually gained through the web (either internet, or intra net).  Modern practice and convention now suggests that even what was previously the province of ‘desktop’ data-enabled applications are now run using industrial strength database management systems ( such as Oracle, Sql Server, MySQL, or Informix) communicated with and from through a browser (Internet Explorer, Google Chrome, Firefox, etc.)  This has occurred in no small part because of the reluctance of the IT professionals to cede to standards of internet and data processing and delivery to one company whose name begins with ‘M’.

So the state of the art today allows 6 or 7 different types of Browsers to request and receive information from dozens of mainstream industrial-strength database managers.  This configuration has given rise to a Tower of Babel in which the processing and parsing of data through these channels is typically handled by a variety of languages – each with different grammar, purposes, characteristics and strengths.  As a matter of fact, in any web based data application, it would not be at all unusual for as many as a dozen languages/technologies to be required.

For example:

1. All communications with the users’ screen is done through HTML, or HTML 5.
2. The screen appearance can be modified and enhanced using CSS
3. Variable information sent to the screen comes from the database and is called using PHP, or some similar language
4. Once the data arrives at the desktop, it is parsed and sorted using Javascript
5. Jquery encapsulates that data so it appears similar, no matter what browser the user is using
6. Depending on what data arrives, the screen can be changed using AJAX
7. XML and XSLT technology can allow runtime applications to dictate output and appearance
8. Database languages, either Ansi SQL or Javascript may be used to interact with the database proper.

For every one of the above languages, Vim provides support in syntax files, indenting formats, pairing of parentheses, braces, brackets, and control expressions.  There are plugins available in many of the languages which will test for grammatical errors and unclosed expressions.  There is support for many other languages, too: C, C++, Ruby, Python, and others in addition to its power macro and scripting languages.

There is command support within the editor for taking a file from a buffer (many files can be edited concurrently) and sending it to a browser to see if the results are as you expected immediately.  The software allows you to create your own dictionaries (context sensitive, of course), shortcut keys, abbreviations, and a whole host of other very productive stuff.  The product is quirky, to be sure, but it has the aroma of genius.

And here’s a list of Vim resources:

• The Vim Reference Manual
• Vim source and down;load download and repository of many plugins
• Vim Vim Help Files
• Vim Vim Introduction and Tutorial
• Graphical vi-vim Cheat Sheet and Tutorial

There are several distinct types of mashups – consumer and enterprise. Enterprise mashups usually occur over secure networks, and are processed on a server, while consumer mashups are more forgiving, and accomplished on the users’ desktops. Web-based mashups usually use the clients web-browser to do the combining and reformatting of data. The various data streams must be sent to the client device as individual data streams so that they can be processed. Server-based mashups will do the analysis and reformatting of the data on the server side and send the data to the client as a single data stream.

Early mashups were created manually by programmers. As with many programming break-throughs, they were first done by enthusiastic programmers just to see if it could be done. As mashups became more popular, there arose a market for tools to simplify and speed up the creation of mashups. Software companies responded by creating tools that allow designers to construct mashups in a visual environment. All the potential components of a mashup are presented in this visual environment and can be connected to create the desired mashup. While mashup editors have made the process of creating mashups much simpler, they have not done a lot to create new ways of accessing the data to be combined. Mashup Enablers address this problem by adding new types of data that can be added to a mashup. A mashup enabler is defined as “the service and tool providers that make mashups possible”.

This type of technology was impossible until recently because of the speed of the average consumer internet communication. The number of http requests needed to ‘feed’ one page of data necessitates a high speed connection to the internet. And the very nature of the requests makes pre-made dataset presentations unfeasible. Associated technologies such as XML, SOAP, JSON and evolutions in Javascript capabilities have also facilitated adoption of slick representations of this and other Web2.0 applications.

Here’s a mashup created using Yahoo pipes. Yahoo pipes is a graphic interface which allows querying, filtering, sorting manipulating and formatting output from sources across the internet. This particular one taps several RSS feeds using technology from Yahoo and Delicious. The combines them into a single feed which is sorted by dates so that the most recent articles appear at the head of the list. If left to its own devices, the pipe would return hundreds of thousands of articles! But for our purposes, we limited it to the 15 most current articles.
(add references to other Paladin articles above)

However, with the advent of readily available high speed processing on the desktop and high speed communications, this mashup technology is beginning now to come into its own. Web pages that before were static and ‘dusty’ and were outmoded before they were published are now beginning to give way to dynamic conversations between the provider and user. Mashups are one way of delivering custom tailored content to a hungry consumer with an ever shortening attention span and an ever expanding demand for information.

For the business owner, it is also a boon. Here is an excellent way to provide up to the minute content which can be tailored to the needs and desires of an interested customer – kind of like an MIRV missile (multiple independently-targeted re-entry vehicle). This is very powerful magic from a marketing and business development point of view. And even more exciting is that after the relatively small initial setup cost, the business owner can have his website updated and fresh, targeted to any individual customer or prospect by piggybacking on the technology of someone else!

Here’s a mashup using Flickr.com and Flickriver.com. Flickr is the repository of data containing the photographs, and Flickriver provides the technology that produces a randomly ordered list of up to 500 separate images from a pool of Flickr files. The list is different every day, so this blog article will show a different crop of images every day -automatically!

No, this is not plagiarism. It is perfectly legal and respectable. The providing companies expose their API (application programming interface) to the public exactly for that reason: they are anxious for other sites to share their technology because it establishes themselves as an industry standard, and makes it more likely that they will get a larger share of the business generated by the downstream customers .

If you feel that your company might benefit from such an enhancement to your web site and marketing program, please feel free to contact us. Additionally, we welcome comments on our articles in the space provided below.

A few months ago, I answered a question posed, i think, on one of the LinkedIn discussion groups (as MrPaladin) where someone wanted to know about Web2.0. My answer was roundly accepted as authoritative and descriptive. But in retrospect, it was probably provincial and pedantic. The answer I gave described the feature of Web2.0 applications that allows interaction with the user without reloading the page. I used as examples perhaps web pages with stock tickers, or news tickers which constantly update their content, or folks who put their latest tweets from Twitter on their web page to be updated the minute each gem of wisdom gets published.

That answer was correct, but one dimensional, because it focused only on the appearances and apparent technology. In a practical sense, Web2.0 is a much larger concept.

At Paladin Consultants, LLC we do web sites, and web services. Many of the technologies we use come from more sophisticated and mission critical applications. Over the past decade, developments in computer languages, processing speed, band width and transmission speed has grown, with an associated increment in demand and expectations for more sophisticated content, marketing, interaction, and entertainment in web sites. This article is an exposition of some of those methods and techniques.

Another article, which discusses the technology of XML (eXtentible Markup Language) was published a few weeks ago in our blog.    XML is another indispensable tool for creating and maintaining content in web2.0 pages.

The early web sites were billboards. They advertised their clients’ product for all who passed by. This progressed rapidly to shopping carts, so that the passers by could order and pay for products. But there were obvious limitations. In the first place, folks don’t like to read long web pages. So the amount of information offered on a page needed to be limited, or, even better, refined so that the user got served only what he was interested in.

Second, marketing grew more sophisticated. Instead of just offering products to a mass market, web sites not are able to establish a kind of a dialog, or relationship with the customer. This is accomplished through interaction over the web. You don’t want to force the customer to reload pages with every choice, so new technology now allows the user’s machine to go and request new data, choices, colors, or what have you while the customer is browsing elsewhere on the page. That new information is requested, received, and formatted behind the scenes, and inserted into the appropriate place while the customer reads on without interruption.

This Twitter widget is something I have resisted putting on our blog and web site. Not all the things I mention on Twitter are, strictly speaking, technology and business related. But in the overall context of things, perhaps that’s not all bad. People sometimes value that the folks they are dealing with are real people and have personalities just like them. And this Twitter widget is a good example of how you can be more in touch and interact with your customer. It puts a ‘human face’ on your company brand.

Web2.0 is said to be the embodiment of more exciting, interesting and topical content into today’s web by using new technology such as javascript, video, sound, and content from other sites. Interaction and content from Flickr, Twitter, Google, YouTube. The ability to embed this new technology into web sites and deliver them to users through their browser brings a welcome new dimension to web communication with the customer.

Story of my life from Paladin Consultants LLC on Vimeo.

Here’s a very short video a friend recently sent me. It suits me to a tee philosophically, and it is illustrative of the purpose and intent of this article in a very real way. See if you don’t agree that content of this sort will make a user more receptive. See if you don’t agree that it can go a long towards establishing a connection between the site’s owner and the prospective client/customer/user.

Here’s a demonstration of a web2.0 technique that combines the solution of economy of screen real estate with a clever way of ascertaining and then delivering content tailored to a user’s preferences.  Click on one of the dark subject words below.

Web2.0 is a great technology because it delivers on target to so many points:

Interest
By your choice of ‘Interest’ you have shown that this subject is a key hot button for you. You don’t need to be bothered by the other items, and clicking on the key word can be both recorded and rewarded. At the same time, screen real estate is conserved. Sometimes when a user is confronted with a huge page of text with no ‘candy’ he will move on before you get your point across.

Content
You are able to add interesting and rich content to your articles or pages in a way that didn’t exist until very recently. Sometimes you might want to add a video extolling the virtues and benefits of your products, or convey an idea in an entertaining way.

Marketing & SalesHere, as I suggested above, you are able to ferret out the user’s preferences, and deliver a targeted response. You are fmeeting his/her needs in a way that was impossible before. You are able to establish esily a relationship with the user which is non-threatening and helpful. And this technology will help you distinguish yourself from your competitors.

ROITechnology, properly implemented, is an effective and cost saving technique. You are able to use a veritable smorgasbord of other sites’ technology (such as in my Twitter example) including Google, Flickr, Yahoo, and many many others. Many sites encourage use of their content, and even provide maps, or hooks into them because they want to cooperate and establish a standard. This is good because you can use this free content to offer information and service to your clients. A prominent example of this is Google Maps, which are appearing on many retail sites today. And, although the technology is not ‘so simple a caveman can do it’, it is well proven, reliable, and effective.

The purpose of this article is to suggest various ways that one can improve his content and delivery to targeted and happy potential customers. The fast moving technologies embedded in web2.0 web pages is growing more powerful and pervasive every day. If I were to talk about all of the technology, the article would too long and uninteresting. So I have confined my remarks to suggesting business uses of this technology. If you would like to see more examples shown in a non-technical and non-threatening environment, the folks at the Academy of the Sacred Heart in Bloomfield Hills, Michigan have put together an excellent primer on the subject. Another, more technical source is the W3Schools Online Web Tutorials.

Please feel free to comment on this article below.

XML (eXtensible Markup Language) is a very important and key technology in today’s internet, and other applications involved in the smooth transfer of data between computers.  As a markup language, it bears a family resemblance to other similarly named languages, most notably HTML or HyperText Markup Language.  HTML is the language of the internet which allows web pages to be displayed and formatted in multiple browser configurations with the two-fold objectives of flexible formatting (to allow layouts to be presented adequately across multiple screen sizes), and the enabling of hypertext, or the links which allow the user to ‘jump’ to other related pages with the click of a mouse.

The operative genetic link between these two languages is the ‘Markup Language’ common component of their names.  The markup language means that content of the files using these languages are enclosed in ‘tags’ or labels, which define their context.  But there the similarity ends.  HTML recognizes a list of perhaps 200 words that when properly used as labels and descriptors can allow a web page to be rendered or interpreted into what passes as a well formatted document.

XML, on the other hand doesn’t DO anything.  It provides the mechanism to store reasonable amounts of data in a commonly accepted and defined format for transfer and communication between computers and systems.  Its vocabulary is theoretically unlimited in that tags may be made up and transmitted at will,provided that their punctuation is proper.  The files themselves are made up of elements which are delimited by these tags.  XML tags are case sensitive, whereas HTML tags are not.  The elements, or nodes may be multi-generational with some elements having lesser nodes describing characteristics or individuals of the larger nodes.  Here’s an example of a small XML document which simulates a group of  appointment book entries:

<?xml version=”1.0″ ?>

<appointmentplanner>

XML is truly an abstract language.  There are no specific words required, only that each element, including the root element of the document itself, be enclosed by matching tags, and that its first line declare that it is an XML document.

There’s more.  One can actually create an XML language which will act as a container for a specific application.  This involves an additional referenced file which contains those definitions.  These files are called schemas or DTD documents (Data Type Definition) which can actually define the permitted or required elements in an XML file.  One could invent an XML application language which would be specific to -say– chemistry (requiring definition of atomic numbers, or isotope variants), or medicine (perhaps a separate schema for each specialty, or even for each doctor within a practice)  The schemas or DTDs can define the type of data within an element – text, video, binary, file, or whatever.  This is very powerful technology.

There are several ancillary technologies related to XML which magnify its strengths.  For instance XPATH and XQUERY empower the user to parse an XML document to extract portions or elements of the document related to what he is seeking – much like Regular Expression and database query technologies.  XSLT uses the XPATH functionality to format an XML document.  And there is facility within modern day relational databases to receive data from XML, and deliver data from XML, distributing and gathering it appropriately from relational tables.

The flexibility and abstract nature of XML make the protocol extremely adaptable and useful for exchanging all manner of data between applications.

• Business to Business transactions are now typically (orders, confirmations, quotes) transacted using XML, or one of its variants or extensions.  This allows a buyer company using Windows machines to communicate and accept confirmation of orders to their supplier company who is running a Unix system
• RSS transmissions are standardized using XML, or its close cousin ATOM protocols.  This allows a light weight individually tailored package to be delivered to the user and interpreted successfully, whether the user is a Mac Windows, or Linux machine.
• XML allows websites to provide a web page to be displayed and tailored to each individual.  This is more complex than Amazon’s personal recommendations.  This technique is closer to the iGoogle page which allow you to construct your own home page.  The preferences and URLs you ask for, are formatted to the way you set them, then stored in an XML file.
• XML technology would allow you to construct a feed of your Twitter or Product Specials information to be constantly updated on your blog or website.
• AJAX (asynchronous Javascript and XML) techniques which use XML with javascriptis a technique which allows an interactive and dynamic web experience for the user by empowering the client side web application to update the web page depending on choices made real time by the user.

The best thing about these interactions is that they are done under the radar.  There is no need to take any action either as sender or receiver of this information, since the XML filesare created and interpreted without any human interaction.  This is a very easy to provide updated fresh content for a website or blog without expending additional resources. XML is related to and is basic to similar protocols like SOAP (Simple Object Access Protocol) and JSON (JavaScript Access Notation).

I read an article about 10 days ago which was critical of Microsoft’s efforts to gain a toehold on the web that got me to thinking.  The premise of the article was that Microsoft has thrown huge sums of money into developing a web presence, but has so far failed.  Kevin Kelleher, the article’s author advances an argument that Microsoft should throw in the towel, and get out of the web business, because Microsoft obviously doesn’t do ‘it’ well.

My own vantage is one of an early and ardent admirer of Microsoft.  My Company, Paladin Consultants, LLC ,was one of the original Solution Providers (later renamed Microsoft Partners Network).  Over the past 20 years, we have  designed hundreds of custom software projects for many big name clients using Microsoft Sql Server database, Visual Basic,  .Net Projects in C#, and extensive application development using the Microsoft Office Products (including programmed routines in Word, Excel, and full blown small database system in Microsoft Access).  A few of the larger projects are detailed on our website.

Gates & Co. has changed the world through their innovations, and they are deserving of every nickle they have made.  Microsoft’s software products set a standard for quality, ease of use, and innovation that the rest of the industry used as a standard.  And for years we have been a strong supporter of Microsoft products.

But something happened in the mid ‘90’s - a number of things, actually, that changed the character of Microsoft, and of the playing field.

First, Microsoft seems to have lost its spirit of innovation.  Their execllent ideas like Windows, Visual Basic, SQL Server, and Office have only been marginally improved since the mid ‘90’s.  Their PR machine would, perhaps vehemently disagree with that assertion, but, aside from the cosmetics, and being easier to connect with those programs, the functionality has been little changed for the good of the user.  Their foray into the browser market is a case in point.  They built Internet Explorer into their Windows software to preclude competition from other browser manufacturers like Netscape.  They were sued by the U.S. Justice department  and by the E.U for anti competitive practices.  The corporate culture for some reason elected to compete in the market with anti-competitive practices, rather than innovation.  The Netscape browser was, unfortunately a casualty.

Second, the terrain of the battle shifted dramatically.  Although Bill Gates paid lip service to the concept that ‘ the future of the information business is the internet’, in practice his actions showed that he didn’t get it.  For instance, take a look at this snippet of text taken from a word document:

In an effort to make IE and Word indispensible, Microsoft allowed Word users to save their documents as HTML files.  The only problem was that in order for those files to render properly, the user was required to use IE.   Here’s a snippet of the HTML generated by Word.  Notice how the tags are largely proprietary, and not able to be interpreted by other browsers:

The larger thought is that the very gestalt of Internet is to have as open a system as possible.  The internet is home to many different operating systems, and APIs.  The nature of the beast is that all participants must be able to share those resources freely.  Microsoft’s view of the world, however, is that they want everybody to use their products, and will do anything they can to ‘tie’ as many people as possible to their paradigm.  In a real sense, thse two competing philosophies and business models are incompatible.  Another example of this is the Microsoft .Net platform.  We develop on .Net for clients, but invariably, there are open products that are available that do just as good a job, if not better, faster, cheaper, and easier to maintain.  The same can be said when comparing Windows Internet servers with Apache Linux servers.

So, when it comes to Internet products developed by Microsoft, by the time the product comes to market, there are already superior products available from their competitors.  VBScript vs Javascript, MSN vs Yahoo, Bing Vs Google, IE vs Firefox, Frontpage vs Dreamweaver and the list goes on.

Third, Microsoft has an extremely poor record of loyalty to its ‘partners’.  They have shown this again and again that when it becomes advantageous for MS to desert their partners, they will do so in a heartbeat.  The loyal army of Solution Providers were presented with a more and more expensive and onerous set of hoops to jump through that it became not a good business proposition for most of them.  In the beginning, Microsoft was a good source of referrals for many of us.  It was advantageous for them to establish themselves as the default platform.  But after a few years, those referrals invariably went to the ‘partners’ who sold the most Microsoft desktop products.  So, like their own internal innovation, relations with their ‘partners’ also fell prey to the drive to stoke consistent quarterly earnings growth.  Their relationship with RealPlayer is another example of this kind of ‘partner’ dissolution.  RealNetworks ended up suing Microsoft and winning $3/4 billion, in addition to the $600 million fine levied against Microsoft by the E.U. related to the RealNetworks suit.

I am concerned that as the desktop market matures along with the growth of Internet and cloud computing that Microsoft will continue to suffer reversals.  Of course, it is possible that the company will experience a resurgence in its innovation, but at this point it is difficult to see the theater that will make this likely.  Contrast this to the buzz that frequently comes from the new product machine at Apple. Growth in Internet based computing shows no signs whatever of slowing down.

In 1949, Charles Goren published his seminal work, ‘Point Count Bidding’, in which he described a method of evaluating Bridge hands for playing and bidding.  Goren’s ‘Point Count Method’ revolutionized Bridge playing for the rest of the century, and was the basis for an entire contextual grammar that the best teams used to describe their hands to each other.  Elegant in its simplicity and powerful in its approach, the method evaluated cards by assigning ‘points’ to the highest cards, and additional points to short suits in a hand.  Players would then translate those point counts into the native language of the game so partners could in a concise and precise manner, describe their hands to each other and arrive at the best fit in the bidding.  The system made Goren the most successful and best known professional Bridge player of all time.

Bill James

Bill James had an early interest in baseball.  After graduating from the University of Kansas in 1973 with degrees in English and Economics, he set about publishing an annual book called The Bill James Baseball Abstract.  The annual was published from 1977 until 1988, when he stopped to write more pointed books and articles on baseball, as well as new annuals about baseball.

Both of these guys are notable for bringing numerical analysis to bear on importantly refining the knowledge of their business.  In Goren’s case, he collaborated with several others – another bridge player, and an actuary –  and built a system based on empirical observation and experience.  James used computers and statistical analysis to accumulate, evaluate and formulate his ideas.  He was not immediately accepted by the baseball cognoscenti.  While he was doing his work, he was working as a security guard at the Stokely Van Camp, and, after all, what could a security guard possibly know about baseball?

But his work gained acceptance because his conclusions were statistically verifiable.  See, Bill had data processing capabilities that allowed him to go back in time, and pose questions, and challenge assumptions that had long been held in the industry.  He was able to quantify, for instance he defined the concept of secondary average:

where

• TB =Total Bases
• H =Hits
• BB =Walks
• SB =Stolen Bases
• CS =Caught Stealing
• AB =At Bats

This measures a hitters total contribution by including his walks and net stolen bases.  Another concept he pioneered was the Runs created measure of performance, which, he suggests, is a better measure than the normally used RBI:

where H is hits, BB is walks, TB is total bases and AB is at-bats.  In both of these indicators, Bill reevaluated the contribution of walks in baseball offense.  This was significant in that a player who got relatively fewer hits, but drew a lot of walks were valued by the baseball market at a lower price than their contribution.  James demonstrated that these hitless wonders who drew walks were almost as valuable as the ones who hit a lot of singles, but could be had for far less salary.

He had a number of such axioms which applied to pitchers, formulae for equating minor league performance to major league expectations, and many others.

He was hired by the Redsox in 2004.  Many think he was a decisive factor in the Redsox winning the World Series in 2004 and 2008.  The previous Redsox victory in the World Series was 85 years earlier.  He advised the management on what mix of pitchers to hire, was a main proponent in hiring David Ortiz.  He’s concerned about such things as where players normally hit balls, who can take advantage of Fenway’s close-but-tall Green Monster Wall in left field, which pitcher/catcher team allows fewest steals.

Now those numbers were available to anyone for the past 100 years.  But until the computer age, they were available in an unusable form.  And it took Bill James to shuffle and deal those numbers and experiment with them to bring about a more meaningful and productive way of using them to benefit the industry, and various teams in it.  He had not an impressive resume, nor exceptional intelligence.  But he was a man with a passion for baseball with the statistical and computer skills he needed to apply to his favorite sport.

Many businesses have analogous situations to baseball before Bill James.  They have the data, but perhaps it is not in usable form.  Raw accounting reports are not very useful because by reporting on P/L and taxes accrued they are like the won/lost major league standings records: they report on what has already happened.

A proper managerial accounting system should allow you to collect data by product.  Which products are you selling and to whom? When to they sell best?  How do they respond to different advertising media and themes? How do sales of each item respond to news items? What industries buy your products?  Does your system allow you to get and process feedback from existing customers?  Can you segment and analyze your different products/markets?  Can you easily share pertinent information among different departments in your company?  Can your prospective customers find you on the internet?

If your business is typical, there are literally hundreds of questions and answers you should be mining from the data and computer records you already likely have!  And if you are not recording this information in a usable form, the gain from making a change will be paid many times over in a smaller or more productive workforce, better information, better sales responsiveness, happier customers, better product feedback, more effective advertising and marketing, and on and on and on.

How about interacting with your customers?  Can they order online?  Can they get return authorizations? Can they make suggestions for improving the product?If you have a large company, can you effectively communicate with your employees?  (Not by sending an email to a few, but broadcast changes in policy?)  Changes in individual retirement, or other benefit plans?

We have a long history of doing similar projects with a wide variety of companies.  Solutions typically involve a capable database management system, user friendly software tailored to the demands of the individual business, and a consultant who can listen and quickly understand the demands of your particular business.  Adoption of techniques such as these in your business plan allow your business to compete more effectively, even though your competitors are larger in size and capitalization.

Shortly after my article was published, I got a tweet, from Eugene Mandel, who invited me to discuss the whole concept of data mining on Twitter.  That tweet led to a very collegial telephone conversation between us, in which we exchanged ideas which will provide the basis for this article.

Eugene is a very engaging young entrepreneur in Silicon Valley, who has been involved with a number of startups.  His current plans integrate very nicely with my concept of Twitter data mining.  My prediction is that his newest venture, mustexist.com, is destined to become an important tool for Twitter users.  He recently started the company with his pal and co-founder, Alex Sherstinsky, a Ph.D. from MIT.  These guys are very serious players.

Mustexist’s current product is called list tags. It allows one to take a twitter name, and creates a ‘cloud’ of lists in which that Twitter name appears, roughly signifying the importance of keywords in that person’s list audience by the size of the type in the cloud.

Here’s a copy of Glenn Beck’s tag cloud.  I shrunk it down to accommodate the page size, and its contextual importance in this article.  Beck is on some 3800 lists.  Politics, News, and political head the parade, but further down the list are lists named things like ‘nutcase’, and other less desirable things to be associated with.  Mandel goes through the Twitter API, and finds all the lists on which Glenn Beck is mentioned.  Then, he queries those lists and finds the most popular keywords in their title.  Then he ranks those keywords by their frequency of appearance in Beck’s universe of lists, and performs a simple statistical conversion to interpolate the list keyword frequency to the type size in the cloud.

Pretty neat trick, eh?

But there’s more: list_tags enumerates some of the lists on which the Twitter member appears.  Here’s a picture of the TopLists for “Politics” in which Glenn Beck appears.  Now think, for a moment how valuable this information can be.  Many of these “top” lists contain 400–500 members.  These are lists for the top topic which have the largest number of members – regardless of whether or not the subject appears on those lists.  That’s valuable information for marketers, or for researchers and even job seekers.  But more than that– everyone who is maintaining such a list is someone who has enough interest in your subject to go to the trouble of maintaining a list of people who he/she considers are the leaders in the field.  And if that’s not enough, the lists editors is another valuable source of fertile relationships to cultivate.

The second. longer list is a list of which Glenn is a member.  So, in the case of a celebrity, it will not be unlikely that the same names will be on both lists.  I’ll not reproduce the Lists for ‘politics’ because it is too long – and because in appearance it strongly resembles in format the ‘Top List’.

So the stage is set now with a resource now to empower Twitter users with access to many lists.  A little resourcefulness will enable you to harvest those lists, and, by extension, to knowledge of all of their membership.  This will give you first hand access to communicating with the most well-regarded people in any given field!  And with the LIstimonkey resource, you can monitor the conversations of those experts for keywords.  With a little software jiggering and less than an hour’s effort, using our Glenn Beck example, we could identify, harvest and communicate with thousands of political junkies.

Mandel, however, is talking about taking Mustexist to a whole other level.  Using the database he has assembled for the list-tag project, he is planning to offer in the near future an interactive, iterative way of ‘surfing’ the tweetstream of these lists, in much the same was that Google lets you surf for articles.  He made the analogy of a newspaper in our conversation.  Each newspaper has several sections: sport, business, local, etc.  If you accept the proposition that a list’s tweetstream is like a newspaper, then it would be very productive to skim through the newspaper, looking only for articles that interested you.  You could skip the car ads, and dating club ads, and focus on baseball, or international news.

Similarly, over a period of time, a list’s tweetstream will have a lot of chaff: someone’s opinion of last night’s American Idol, inclement weather, mother-in-law coming for a visit…  But what if you could adopt a list, or amalgam of similar interest lists?  What if you could in real time and on an ad hoc basis query the stream for stuff like: climategate?  cap and trade?  world health organization?  BTU content of coal?  And how powerful would it be if you could pursue your line of inquiry immediately, based on feedback you got from the previous question?

I think that MustExist is on the verge of somthing quite important that will add a new dimension of empowerment to the serious Twitter user.  There are currently some 23 million Twitter users, 75% of which are either classified as ‘addicts’ or ‘regulars’.  That’s a reliable 18 million pairs of eyeballs scanning and interacting with Twitter several times a week – sometimes several times a day.  Any tool that can reliably and easily segment relevant portions of that population and then address them is bound to be very successful.

Please add your comments.  It is helpful to hear the opinions of our readers!

Facebook and Twitter were innovative at the time, and helped businesses form a relationship with clients with a more personal approach, allowing a subtle transition from a purely passive approach to their prospective clients to a more pro-active and personal one.  A lot of folks missed the point of Twitter and Twitter and came late to the party.  The same can be said for the late adopters of Facebook, who didn’t understand their kids’ fascination with the social aspects of Facebook.  College kids live on facebook.  Andif you had any doubt about the centrality of Facebook to the zeitgheist of our society, one need look no further than Michaele and Tarek Salahi’s crashing the Whitehouse state dinner late last month.  Pictures of them and Obama and them and Biden were on their Facebook page before dawn.

I wish I had a nickle for everyone who commented on my embracing the Twitter idiom with a snide comment like, ‘What do I care what people had for breakfast.’  A lot of folks are missing the point of both of these social sites:  they provide a way to form quasi social relationships, and so called weak friendships.  Their point is that people are more apt to do business with people with whom they have some sort of connection.  If you’re looking for someone to re-do your kitchen, you’re much more likely to choose someone from your church, or even someone who is friendly with the butcher whom you lilke.  I am not a big Facebook user because it is too time intensive, but I have met quite a few friends, professional colleagues, and business partners on Twitter.

But Twitter is still a quasi passive platform, though the lines are blurry.  I do know that the easiest way to turn someone off on Twitter is to come onto him with a hard sell:  ‘I’ve made $15,000 this month on Twitter, and I can show you how to do it, too!!!!!’

But thee are now on the horizon two very important developments which promise to significantly enhance the power of using Twitter.

But first a small diversion.  Earlier this year, an enterprising your Australian named Chris Duell developed a site called Twitterhawk.  Now the idea of Twitterhawk is that the site sifts through the Twitterstream looking for key words that you set that occur near a geographic location that you specify.  So if you were a carpenter in Peoria, you might set up a search that looks for kitchen remodeling in Peoria.  Twitterhawk would return to you a list of those tweets,  And, for a few cents a tweet, Twitterhawk would send a series of rotating tweets to those whose tweets you designated as appropriate targets.

Back to those new Twitter innovations.  The first was the implemenjtation of lists on Twitter.  Now you can create lists of folks on Twitter that you follow.  And you can categorize them with common traits, for example software architects (like Paladin Consultants, LLC ), or economists, or photographers.  It helps you keep track of people, especially when you have lots of followers.  You can see the Twitterstreams of these lists, and even get an RSS feed of them.  Better still, in most cases everyone has access to virtually all of everybody else’s lists!

And where can you find and mine these lists?  Listorius is one of the first resources dedicated to managing the new feature.  Mashable also has a growing arsenal of Twitter Lists.  Unless I miss my guess, new sites dedicated to this new tool will be popping up like mushrooms after a spring rain!

The second innovation comes from a site called Listimonkey, a site designed by a very clever young Belgian expatriate named Xavier Damman, now living in the Bay Area.  He has also managed to secure a domain name in his own name!  Anyway, his wrinkle on Twitter is that if you send Listimonkey the name of a twitter list, a key word or phrase, and your email, Listimonkey will email you with a set periodicity the tweets from members of that list contain your key word or phrase!  Now, although there is some SEO juice and pride in having a large list, this is not necessary to work with Listimonkey.  In fact, you can use anybody’s public list!  (Most lists are public.)

Now think about that.  If you are a real estate agent in Chicago, you might watch somebody’s Chicago list for the key word ‘moving’.  If you were a photographer in Atlanta, you’d watch somebody’s Atlanta list for the keyword ‘wedding’ or ‘getting married’.  And, Listimonkey allows you to create as many of these list/keyword combinations as you like!  Very powerful, and a dimensional abstract improvment on Twitterhawk’s idea!

But Listimonkey is good not only for finding new customers and clients.  It’s great for finding out about new products, techniques, people, trends- wealth of information from one site that harnesses the communal information it mines from the Twitterstream!

I’m afraid Xavier’s innovation will spell bad news for Chris Duell and Twitterhawk.  It’s a shame, but the market marches on.  One can only wish Chris Duell luck, and hope that he has another great idea.  But with talent like he’s shown, I am sure that his next idea is right around the corner.

But getting back to Listimonkey, as those emails come rolling in from Listimonkey, it would be pretty easy for a good software developer to come up with a program which would parse those emails as they came in, and then tweet appropriate messages to the originators of those tweets mentioning the keyword(s).  Xavier has such a good idea, that I predict will soon overrun his mailserver.  But that will be an happy problem for him to solve!

Please leave a comment, so we know whether or not we are on target!  It helps us find out what our readers think.

I have developed a system for fitting our blog writing into our daily routine, and it involves the use of technology by gaining efficiency from productive software and streamlining the thought processes.  This helps get around the press of servicing clients in our software development and web business at Paladin Consultants, LLC .

Inertia and the lack of a clear idea of what to write about has been the number one blogging stumbling block for me, along with a cumbersome production process of the blog article itself.  I have solved this in several ways.

1. I found I had a lot of ideas, but didn’t have a way of recalling them when I needed them.  So I started using Evernote, which recently added a Blackberry extension.  Since I am never far away from my ‘berry, I can text, dictate, or send a picture to Evernote, and harvest those thoughts at the proper time.  I also am a heavy user of OneNote, so, on an as needed basis, I copy thoughts from Evernote to OneNote for a more permanent and well-organized record.  There is a good blog for OneNote, which talks about the many uses it has. 
2. I try to have a small camera near me all of the time.  At very least, I have my Blackberry camera. So, armed as such, it is easier to get illustrations of some of my topics and ideas.
3. Recently I have found that once I have decided on a thought or idea, it is productive to actively think about that idea while falling asleep.  During the night, somehow, the idea seems to be processed and examined from all sides, and, oddly enough, seems to be well developed my morning.

The next bottleneck is usually the development of the idea.  This is partially resolved by the sleep technique discussed above.  The problem with the sleep technique is that if you don’t capture it quickly, its persistence appears to be limited.  Use it or lose it.  So among the first tasks of the morning needs to be the capture of the previous night’s thought process.

I’ve had success with mind mapping techniques, especially for short blog articles.  For a longer treatise, a hierarchical outline, such as one might produce in Word would be more appropriate. Usually, I draw the mind map on a yellow college ruled pad, but I’ve gussied up the one for this article to make it visually appealing. See if it makes sense to you.

Once I have the ‘plan’, then it’s time to bring in the power tools to take care of the details.  I use BlogJet, a very convenient and inexpensive online blogging software, which lets me compose, spell check, insert artwork, links, tags, and virtually everything I need to do to produce the blog itself.  I felt that this piece of software reduced the amount to time I spend visually composing each article by at least 1/2 hour.  They also have a very nice feature which fits into Firefox, which allows me to copy and paste pieces of web pages into the text of my blog very easily.

Firefox is an extremely important piece of software for me.  It seems that no matter how hard Microsoft and Google try, they can’t quite come up to the standards of the Firefox Mozilla browser:  Mozilla always seems to be one step ahead of them.  There are two Firefox  plugins that are particularly effective for me: Taboo, and SEOQuake.

Taboo is a software tool which allows you to take many tabs open in Firefox, and store them in one tab as thumbnails.  So, instead of having 20 tabs opened, you might only have two: the one you’re immediately working on, and the other containing the Taboo ‘reservoir’ collection.  Taboo allows you to temporarily store dozens of pages in this thumbnail format, each image growing as you mouse over it.  It saves looking through the tab array, and saves scarce memory.  A very useful tool!  I have a Bookmark category called ‘Blogging Tools’, which I open in tabs  (about 10 sites), then annex them to the Taboo page.  Makes for very efficient research and fact checking.

SEO Quake, is another Firefox app that analyzes the keyword/ key phrase content of your article.  At the end of writing the article in BlogJet, you can save it to a temporary HTML file, and use SEO Quake to appraise it for links, key phrase effectiveness before you publish.  And it always helps to have a list of your Wordtracker phrases to rework your article against.

In the Pantheon of Tools I rank Delicious near the top.  It’s a very powerful tool which lets you save sites, software techniques and articles, like your Browser Bookmark feature.  Except that your Delicious sites are available to you whether you happen to be at your computer or not.  Available on client machines, Available on your Blackberry or iPhone. Available on your laptop.  Available in subgroups as RSS feeds of your tags, of as subscriptions to the tags of the rest of the universe.  An extremely powerful element in anyones toolkit, and a great contribution to social knowledge.

In addition to photographs, I try to include portions of scanned documents like software code where appropriate, to illustrate my point, as well as screen clips, video clips, and charts.  I find that Google charts allows one to create some very nice looking graphics with ones own data.  Those graphics, though not ornate in any sense,  are far more professional looking than those produced by Excel.

Finally, if you are not a strong typist, or are composing at the keyboard, while doing your research at the same time, Nuance’s Dragon Naturally Speaking is a good candidate for you.  I confess to using it sometimes to squeeze out a quick article when I am short on time.

Please leave a comment – comments help us do a better job!