Tuesday, September 18, 2012

How to: Side-loading a pdf into your Android Kindle app

This took me a while to figure out, so thought a blog post on the topic might help other people who have similar issues.

Let's say you have a pdf or other file that you want to read on your Andtroid phone in the Kindle app - admittedly, there are several other applications in the Android eco-system that are arguably better for reading pdfs (e.g. Aldiko, Cool Reader etc) and the like, but let's posit that this is what you want to do for sake of argument. (if you already use the Kindle app on your phone to read ebooks, using Kindle to also read your personal documents leads to uniformity of interface and convenience of use). Currently, Amazon requires that for the Android platform, in order to access personal data, you first email it to them at your <email>@free.kindle.com address, so they can convert it and make it available to you from within the Android ecosystem. This is nice, and somewhat convenient, but has two major (in my view) drawbacks:
1. you need to load your personal documents into the Amazon cloud, and need to manage them within the cloud using the Amazon interface thereafter,
2. you need to have a wifi or other network interface instead of simply transferring your document from the PC to your Android phone.

A rather convenient means exists to bypass the Amazon ecosystem while utilizing the Kindle app to read self-generated pdf documents like your office memos that you do not necessarily want to entrust to the cloud. To do do, you can follow the steps below:
1. download and install an Android application like FileExpert. This permits you to navigate the file system on your phone and locate files of interest
2. connect your Android phone to your computer, and turn on access to the phone's internal storage via USB.
3. create a folder in the phone (preferably on the SD card within it) to store the files you want to transfer. Transfer the files over into that directory via USB.
4. Disconnect your Android phone from the computer, next, run FileExpert (or equivalent) application. Navigate to the folder where you have stored your files. Click on the file you want to read. The OS asks you what application you would like to launch to read this file. Select the Kindle application, and you're set.

Wednesday, June 20, 2012

How (not) to conduct a Case Interview

There are several books that advise students and busy professionals on how to prepare for a Case Interview. An excellent one is Case In Point. However, having conducted and been interviewed as a candidate several times in Case Interview settings over many years, I note that many interviewers tend to lack skill in conducting Case Interviews. This is sad, quite embarrassing and many a time a dis-service to the organization the interviewer works for. A poorly conducted case interview can hurt an otherwise worthy candidate's career prospects, and cost your company an attractive hire. If a Case Interviewer does a bad job, it is unlikely she will permit the candidate to advance even if he performed well given the interview constraints. The interviewer's ego will simply not permit it. And a candidate's protests are worthless since he'll be viewed as incompetent and disgruntled because he did not advance.

Of course, the most typical Case Interview setting still remains one where a prepared interviewer meets a not-as-well-prepared candidate, with both sides being prepared for the encounter being something of an exception rather than the rule. 


What happens when an unprepared Case Interviewer meets an unprepared Case Interviewee? This falls into the realm of questions like, "when a tree falls in a forest with no one there to hear it, does it make a sound?" And we ignore it in this post.


So what are some key "rules of thumb" for Case Interview interviewers? We present a brief, and not exhaustive list here in what follows:



  1. Do your homework. Know the case question you are asking. Know it well. Know the context/setting, the problem, possible solutions, and paths to get there. Most importantly, know what "a correct answer" looks like.
  2. Know what direction(s) you are willing to let the candidate take, guide him in those directions. Do not perform this function haphazardly, pushing one way and then another, because you yourself do not know how one should proceed from start to finish.
  3. Scope the problem beforehand. You can have multiple objectives, but you must decouple them and ask the candidate to solve each piece first before going on to the next. Do not cut back and forth between multiple questions. This confuses both yourself and the candidate.
  4. Know what questions you expect the candidate to ask, what data you want to present them when they do, and how you want to respond if they ask questions you did not anticipate.
  5. Be reasonable with the case problems you make up. There has to be a way to get from your question or premise to an acceptable answer. Well prepared candidates can crack reasonable questions in half the time you think needs to be allotted. By the same token, if the question is poorly defined, or you base it on some vague idea in your mind, no candidate will be able to finish it in time.
  6. Know what assumptions make sense, and which ones you are willing to accept. If the prepared candidate asks "I am assuming there are 300M people in the US, and the average life-span in the US is around 80 years, is that reasonable?". Feel free to say no, and ask for clarifications, but once you say yes, and the candidate does all the math based on these assumptions, don't backtrack 15 mins into the problem telling him the base assumptions are incorrect.
  7. Last, but most important, do not penalize the candidate for your lack of preparation. If the "answer" the candidate comes up with, based on assumptions you approved earlier does not look right, gently ask the candidate to redo the calculations with more reasonable assumptions, but acknowledge that your approved assumptions were somehow incorrect.
The above are only some ideas on how prepared interviewers might conduct a Case Interview. Note however that there are exceptions to every rule. An interviewer is justified in breaking one or more of the above rules if she is conducting a "confrontational" or "hostile" interview to test the candidate's mettle in a final round. These are designed to see how the candidate handles pressure, uncertainty and difficult situations. But such interview scenarios evolving naturally (instead of deliberately) due to lack of preparation is unforgivable. Good interviewees spend many weeks of effort mastering the Case Interview technique. Hurting someone's career potential because you are unprepared is criminal. Do not conduct Case Interviews if unprepared.



Reverse Inboxes

In this post, the author examines his social network by analyzing his email. Email headers - when both the sent and inbox folders are considered together, give strong indications of who sends email that is inbound, and who one sends email out to. This, when taken together with the number of times a particular unique identifier appears in the To: or the CC: or the BCC: fields of the email messages - perhaps entries in these three different types of fields are weighted differently - gives a pretty good indication of the strength of one's professional or personal relationships with other people.

We can fine-tune the mechanical classification described in the earlier paragraph a little bit further if we were to perform either sentiment analysis or some other text data mining in order to determine if a. the content is of a personal or a professional nature, and b. whether the email document is positive, negative or neutral in terms of expressed sentiment. These things together can be used to construct a network or graph of the relationships between the various people in one's communications sphere.

One can, if one performs this exercise with various people's email boxes, also determine if one person's network is more dense and more completely connected than another's and whether the span or expanse of one person's social network is greater than another's. This might also permit us to differentiate between people's personal and professional networks, and the set of people (or nodes) that straddle the two domains.

We posit that it is possible to have a nodding acquaintance with lots of people, and know some people really well - it's hard to know many people really well - and this is something we can confirm by performing this "reverse inbox analysis" for various people's inboxes.

To perform this analysis effectively while maintaining the author's privacy, rather than present the email addresses of individual users from the author's mailbox, an MD5 hash of the email address concatenated with a fixed random string of data is used instead. This hash value is repeatable but randomized, so provides an individualized marker by email address though the identity embedded in the email address is not visible.

If one were to compute the reverse inbox based social networks for different users, and then connect them together, a very complete view of the organization's or community's social network as a whole would emerge. We discuss this in a related post on "Mining your Social Network". We content ourselves with providing a simple reverse inbox implementation in this post.

[code to follow shortly]

Saturday, June 2, 2012

The Itsy Bitsy Spider

In nature, spiders are beautiful things. Yes, they can be classified as creepy things when we look at particular specimens, with their long spindly legs and abrupt quick movements. But viewed in the abstract, they do some very unique things - for one, the fibers they emit to build spider-webs are considered to be stronger and more resilient than steel fibers of the same diameter. But in this blog we talk about a different kind of spider altogether.

The WWW is a mesh or web of hyperlinks. What better creature to crawl this abstract space and extract information from this structure of data and knowledge than a spider - and indeed, programs that perform this function are called spiders. In this post, we present a simple Python spider program that crawls the web. (a snake and a spider in one sentence ...)

Let us look at some applications of this kind of a program.

  1. let us say you are trying to scrape all content that hangs off a single web-page across a set of multiple hyperlinks. One way of gathering all this content would be to write a program that iteratively processes each link on the page and collects the data it reads, organizing it into separate files. Freeware programs like HTTrack, the website copier, do just this. But this is a severely limited spider since it is restricted to just those links that either a. hang off the specified web-page, or to links that are hosted by a particular domain or sub-domain of the internet e.g. www.my-company.com
  2. the more exotic application - document retrieval for web search. For Google to be able to respond to search queries, it must first build an indexed data set of various documents from the web. To do this, Google unleashes multiple multi-threaded spiders onto the web, and they collect all documents it makes sense to collect (using some intelligent criteria), following links from one document to the next, until a large part of the web is gathered. Google then indexes these documents to construct a data set that can be rapidly queried to respond to user queries.
  3. Avinash Kaushik, in his excellent books, distinguishes between internal (as in, within a company's intranet, for employees and other internal users) and external search applications with a focus on web analytics. These two perhaps do merit special discussion even in this simple classification which is why we include this bullet item here. However the mechanics for (3) are very similar to those for (2) from a web-crawler perspective, with some of the limitations from (1) imposed on the spiders in question.
  4. desktop search - except here the spider crawls through the directory tree structure of one's file system instead of the Internet, but the operational principle is the same.
We focus our implementation on a very simple instance of (2) above. Also, there is etiquette one needs to follow for building spiders (e.g. the robots exclusion protocol), and since we are not building an industrial strength implementation but one just for fun, we gloss over these details and "hobble" our implementation, constraining it to not explore more than a pre-set maximum number of links before it halts.

Also, for the layperson reading this, while it is somewhat glamorous to visualize this process as the spider crawling the WWW, what really happens of course is that the spider program is resident on the host computer running it, it is simply downloading html documents from the specified starting site, following hyperlinks wherever they may lead, then downloading those other documents,... and so on.

A comparison of how many web-pages were indexed by the various web search engines shows how up to a point, increasing the index size improves search quality. We say "up to a point" because a web search engine  called Cuil (2008-2010), indexed slightly over 120B web-pages at one time, but an unfortunate inability to produce search results of comparable quality to other leading search engines led to its early demise. To be a popular web search destination, it is not just important to index as many meaningful, non-repeating, non-machine-(auto) generated, non-interstitial-screen pages from the web, it is also important to return results that provide the kinds of information the user is looking for.

Code is below. Sample output follows the code.


import os, sys, urllib2, re; 


u=sys.argv[1]; # the url to start with
lynx=[u]; # store the start url into the download set
for u in lynx: #process each link in turn
 r=urllib2.urlopen(u); # get the requested object
 x=r.read(); # read the document from the website
 xlines=x.split(" "); # split and process the lines
 c=[i for i in xlines if i.lower().find("http://")>-1]; # collect all hyperlinks
 for i in c: # process each hyperlink, ignoring some words
  if i.find("CALLOUT|")>-1: continue;
  stpos=i.index("http://");
  if i.find(".html")==-1: continue;
  if i.find("\n")>-1: continue;
  enpos=[]; # code segment that follows parses the retrieved document
  enpos+=[k.start() for k in re.finditer(".html",i)];
  #print stpos,enpos;
  enpos=[j for j in enpos if j>stpos+13];
  if len(enpos)==0: continue;
  enpos=min(enpos);
  candidate=i[stpos:enpos]+".html"; # store retrieved urls
  if candidate not in lynx: lynx+=[candidate];
 if len(lynx)>100: break; # hobble the spider so it doesn't go haywire


g=open("t.txt","w"); # write crawled urls to file
for i in lynx: g.write(i+"\n");
g.close();




Sunday, May 20, 2012

On Robust Hyperlinks

As the Internet grows and the WWW expands with it, users are becoming more savvy - young people today do not even have a memory of a time when it wasn't as prevalent. Savvy users tend to be more demanding of services they use. Also, as web servers become fronts for more critical business applications (e.g. store-fronts), there is the potential for revenue leakage if errors like a 404 Not Found message are returned in responses to user requests. In addition, many a time one comes across dangling hyperlinks (that point to void) which make the browsing experience less pleasant than it could otherwise have been. These kinds of scenarios make the case for Robust Hyperlinks [1].

What are they?
Robust hyperlinks are hyperlinks or URLs designed with a few extra words relating to the underlying web-page's content within them, say separated by dashes. (There are lots of FAT URLs floating around encoding all kinds of parameters in all kinds of applications, so why frown on the use of a few words to make hyperlinks more robust?)

Why use them?
Sunny Day Scenarios
Now, when the client (user sitting at a browser) makes a request for a page by clicking on the text label for the URL with a robust hyperlink, things work as usual, and the html document associated with the hyperlink is downloaded and rendered into the user's browser. So far so good, and it appears all that extra work was for naught.

Rainy Day Scenarios
What happens if Bob, the web-master for the site, carelessly updated the web-page and left that hyperlink dangling (e.g. by moving the underlying document to a different directory, or somehow renaming it)? The user at the browser receives an ugly "404 Not Found" or similar message from the server, and potentially has to start looking for other sources of information, such as, for example, by Googling for it.

Robust hyperlinks beg the question, is this second user step really necessary? If the client, say Google Chrome, Internet Explorer, Mozilla Firefox, or Opera, or whatever other browser the user uses, were able to:
1. read the server-returned error message,
2. parse it,
3. extract the five "keywords" from the request tied to that interaction,
4. query a Web Search engine for similar documents, and
5. finally present the result of such search queries back to the user seamlessly with a note indicating the original request were not fulfilled,
would that not make for a much nicer, more seamless user experience?

Alice, who makes the original html document request now has multiple links to choose from, all similar in content to the original hyperlink she clicked on, in terms of content.

How to get them to work?
In previous posts, we already explored the notions of TF-IDF or term-frequency/inverse document frequency, a mechanism that enables us to determine statistically improbably keywords or phrases from any particular document given a corpus or body of documents. If the keywords for a robust hyperlink were constructed from the top five statistically improbable phrases for the document in question, considering their TF-IDF ranking given a corpus of general web-pages (yes, this might take some work on the part of the organization hosting the web-pages, but is mostly amortized across the large number of hyperlinks they host, so the incremental cost per link is likely minimal), then this set of keywords is a statistically relevant expression of what makes this web-page more unique or less general when compared to most of the other web-pages out there.

What this also means is that if the client were to reformulate a query using these five keywords as search terms in a Web Search engine, the documents returned are likely to be good matches for the content Alice wants to consume.

Why five keywords? It appears the authors who proposed the scheme in the paper listed in the references have performed empirical tests and five was an ideal sweet spot between having too long or cumbersome URLs and too sparse URLs that might throw off the search operation to follow.

What's in it for me?
The organization hosting the original URL might argue against doing all this extra work at the cost of an additional fraction of a cent per hyperlink if this only serves to drive web-traffic away, potentially to competitor sites. A small incremental investment on the server side could be made to have the server conduct an internal search on the hosting organization's web-space to locate other documents that might serve as substitutes for the now broken link. This way, the robust hyperlink implementation serves to bolster the business case for the hosting organization's added spending as well.

Issues with Trusting Trust (with apologies to Ken Thompson)
For this scheme to work, one must necessarily trust the people who build the hyperlink to encode it only with words that are truly reflective of the content. This is also true today, but since links today are not automatically parsed by browser add-ons to query for additional content, there is a lower likelihood that an innocent click on a dead link might lead the user to see a web-page with links that are not-relevant, offensive or plain undesirable. This happens mostly because users do not typically read the structure of the link they click on. Perhaps over time, if robust hyperlinks really take off in a big way, intermediation services that verify the keywords and their relevance given the content of the underlying delivered documents will become more prevalent. Such services might be able to safeguard users' browsing experiences.

A second issue might be one of a hijacked search. If Alice starts with a certain set of search keywords that led her to the original hyperlink, she may be led off track by the keyword searches that result from the set of hyperlinks associated with the first failed document retrieval. To ensure usability considerations like this are addressed, it might make sense for the browser based implementation to give Alice a clearer sense of when a search query was done on her behalf using robust hyperlink keywords, so she can manage her Internet searches more effectively.

Examples:
The authors of the paper in the references provide several complete examples with nice, readable descriptions here.

References:
[1] http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1091.pdf, Thomas Phelps, Robert Wilensky, UC Berkeley.

Friday, May 11, 2012

Mining your Social Network

Stream of Consciousness Ramblings on Social Media...

On Being Human
No sentient being exists in isolation. Beings coexist together with others of their kind drawing on a sense of community, with the potential to work together to build things bigger than any one among them could even conceive of, alone. And perhaps this characteristic isn't so much restricted to just sentient beings. Are ants sentient? Bees? Both have fairly well-developed colonies.

Some leading technology companies [citation needed] work on the notion of Hive Intelligence - where reconfigurable sub-components of a systems can combine together in different ways to best adapt themselves to solve problems in some fairly different settings. Artificial intelligence has developed to a point where adaptive robots can be constructed where small robots collaborate and configure themselves into a larger automaton to solve new and difficult problems. Adaptive machines are fascinating - the Borg were arguably the most interesting race from Star Trek: The Next Generation. But we digress... The things that make societies and civilizations interesting, is mostly the collected knowledge of the species - knowledge that is accumulated and grown over millenia - an idea nicely discussed in the H.G. Wells novel "Christina Alberta's Father".

Humans are among the most complex of beings to ever exist. Environmental adaptation is almost second nature to us. Unless we are completely asocial, as some of our species sometimes are, we build networks of relationships wherever we go. And these networks evolve over time. In this post, we examine one aspect of the power of social networks, from the standpoint of network establishment, evolution, degradation, and regeneration, with a view to studying the power of this medium, and why it is perhaps unlikely that a single social networking platform will continue to predominate the global conversation.

The Times, ... They Are A Changin'....
"Know yourself and your enemy. You need not then fear the result of a thousand battles."
                                                                 -- Sun Tzu, "The Art of War"

People born into the digital age are grow up in a world where email, mobile phones, and texting are commonplace. This world also serves as an interesting laboratory to study (social) network evolution. How do relationships change over time? Mining historical email data can provide valuable clues. What bonds were stronger at what age? How does the relative strength of these bonds change? How does age of the individual, time (which generation the individual belonged to), and place (geography, and associated cultural context) define what bonds are "important", which media are used, for which aspect of social interaction, and what, and how much data is shared?

The Lonely Network
In other words, every human being is a DG (Directed Graph from Graph Theory, which may not necessarily be Acyclic in this case) in social network terms, with a node representing themselves at the center and other nodes representing their "friends". Social media connect these DGs together, "decentralizing" or democratizing them. But we can of course, learn a lot from even a single individual's graph, and studying that is perhaps much easier anyway.

In related posts, we start our exploration of this idea. In the post titled "The Reverse Inbox", we present a simple prototype that de-constructs a sample gmail mailbox, uniquely anonymizes every contact to protect the innocent, then presents a view of the mailbox owner's social network. An advantage we have with experiments of this nature is that given the large amounts of storage now available on free Internet email services, no one ever has to delete all their email, so histories going back several years can be easily extracted from even such a simple exercise.

We can see what links are important, in which directions, mine the network to infer the relative strengths of people's relationships, and construct a snail trail graph showing their evolution - the strengthening and weakening of different links, over time. We hypothesize that links tied to family, close friends, and co-workers grow stronger, with the latter fading as one moves from one job or career to the next.

When these lonely networks are joined however, the decentralizing process adds a new capability. Nodes that were hitherto unconnected in each individual's network now have paths between them. This is a concrete example of Metcalfe's Law in action: "The value of a network is proportional to the square of the number of nodes within it". You are node A, you want to get to node C, but don't know the way. The network tells you how to reach C via B, D, ...

How Social Are Social Media?
This also brings us to another idea. In Facebook and LinkedIn, all links between nodes (i.e. people) are bi-directional by default. In other words, friends are only friends if both people acknowledge the relationship, even if one does this somewhat reluctantly in some cases. Social networking seems to have misplaced the idea of "polite blocking" that was widely supported and prevalent in the Instant Messaging space. Twitter is somewhat different though, with people collecting followers like a non-rolling stone gathers moss, without necessarily pro-actively accepting each new follower into the fold. Arguably, these lead to different social dynamics.

Even more interesting is the fact that in Facebook it is relatively easy to "unfriend" someone - break the link. Can't do this so easily in LinkedIn - why, you might even need to call the company to get someone off your network. So connections on LinkedIn are perhaps worth more than those on Facebook - harder to break must mean a higher standard to forge - or people will eventually migrate to this behavior. For Twitter, we have the notion of "Social Capital": the number of one's followers. Links are cheap for individuals, but have greater value to the people they follow - since they contribute to social capital. Perhaps there is a business model here - celebrities, products, or groups being followed can set a limit to the number of links they will accept, based on certain criteria, and monetize the power of social networking per link accepted. If nothing else, perceived scarcity increases perceived notional value. (see other blog post on "Monetizing Internet Services")

A Heaven for Dead Networks?
What happens when networks die? What is death exactly? Without waxing philosophical, we can learn some lessons from examples of "death" in the social space. Two examples that leap to mind here are Friendster (now seeing a revival in Asia?) and MySpace. But to die, one has to be alive first.

Networks gain a measure of life through their expanse (number of nodes) and connected-ness (number of links). Smaller, strongly linked networks are more cohesive and perhaps provide better "value" to their constituent nodes than larger, more sparsely connected ones. Some combined measure of these two metrics would define the value that individuals connected on the network can derive from their association with it. It also stands to reason that perhaps each link in the network DG should also be weighted according to its importance given some of the discussion in the previous section.

To complete the anthropomorphic comparison: As the total metric for a network grows, it springs to life. As the rate of change of these metrics increases, the network transitions into adolescence and adulthood. As growth slows, it transitions to maturity and old age, and as nodes leave and links dangle or get deleted, the total metric decreases, and the network slowly atrophies and then dies.

Dr. Jure Leskovec at Stanford University does some interesting work with social networks including the study of the diffusion of information e.g. viral marketing and inferring social connectedness from the spread of disease within a population. His very interesting talk is here.

The Fountain of Eternal Youth
So how to stay younger longer? Since value builds from network effects and positive externalities, the best way to retain users is to a. grow value through "sticky" network services, b. evolve individual users' view of their immediate "vicinity" in their social graph to meet their social needs, and c. make it harder for them to leave (i.e. increase switching costs).

As an example, Facebook does (tries to do) all three. (a) by building their network as a social platform on which other companies like gaming firms Zynga can build and host interactive multi-player games, Facebook credits - its own currency, etc. (b) by enabling support for various kinds of interactions (professional vs. college vs. ...) and (c) by holding on to users' data and encouraging them to share more through features like "timeline".

As more users sign on, unless they can move taking their networks with them, it gets harder and harder for them to go elsewhere, especially since they have invested a lot of time building up their "presence" in the social graph, and all the connections they have established. This ensures survival of the network for a while - at least until the majority tire of it.

Forward the Federation
By no means is the battle for supremacy in this arena over. There are features Facebook lacks. Google+, Path and others are building these into their core DNA rather than as adaptations or mutations into an existing animal, and it will be interesting to see what the future brings.

At 900M users and counting, Facebook is likely not to get too many new users given almost 1 in every 6 people on the planet are already on it, and to a large extent, geography dictates social network choice (there are other social networks that are growing faster in Asia). Perhaps the next step in the evolution of social network technology is "federated social networking", but we leave that for discussion on another day.

If federated social networking becomes a reality however, the impact of "stickiness" is reduced and walls between networks become more porous, and a new dynamic will come into play.










Monday, April 16, 2012

Monetizing Internet Services

In this post we start out with a brief discussion of a simple taxonomy of service delivery models for Internet-based applications and then go on to discuss different business models that can be used to support these services. Finally, we put these together to discuss how the different kinds of Internet services may be monetized. The goal is to present several examples to drive the points home and make the discussion useful in a practical, real-world, setting.


Classes of Services
Services may be classified in different ways using different criteria. Here, we explore a couple of simple ones, and use these as a basis for building ideas that follow.


By Underlying Protocol or "Connectedness"
Different services have different characteristics and rely on different protocols. Looking at things from a protocol perspective... Some use best-effort routing underneath, with the UDP protocol where the loss of a small number of packets does not significantly degrade the user experience. Others utilize a protocol (TCP) that layers timers, packet sequencing, and a re-transmit mechanism on top of the underlying pipes, to ensure higher quality of the user experience, albeit with greater overheads. ... and new protocols are constantly in the works, crafted over time by hard-working engineers at the IETF and the IRTF.


"Sessions"?
From a more pedestrian standpoint, we may classify services in simpler ways - those that require a "session" vs. those that do not. For instance, voice and video calls require a session over which to transmit and receive data, while a simple web download does not (though strictly speaking, this latter service is provided over a TCP "connection" which we distinguish from the notion of session in this case). However, websites may utilize other mechanisms like cookies, web bugs etc to track different interactions between a user and a website, and construct a session context from these.


Modes of Interaction
Some services follow the "client server" paradigm. A user, sitting at a client machine, makes a request that his browser connects to a server on the Internet to fulfil, downloading data onto the user's screen in the process. 


Sometimes, the user (and his client) may be in a sheltered network like his company's intranet, at work, and the server may be in a different company network so there are intermediate "intelligent" nodes the user's request, and the server response must transit between the two endpoints. 


Other services, such as the file-sharing ones hosted by Torrents, are "peer-to-peer" services where computers are treated somewhat more democratically as equals, and these "equal" nodes are able to connect and communicate once they find each other using a network provided "directory".


Yet other services follow the "interception" or "intermediation" paradigm. In these cases, the server providing the service interjects service related information into the communication stream. This can be done effectively at the boundary between two domains at a server that all packets must transit, assuming the communication between end-points is not encrypted (if it is, there is no obvious way for interception-style services to work in those settings).


This taxonomy will be useful as we examine how we might offer, and bill for, Internet services.


How to generate revenue?
Lord Kelvin is reported to have said  "[...] you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind [...]".


To bill someone for something, unless you intend to collect a flat fee for unlimited access on a per-time-period basis, you need to be able to measure use. But how does one go about doing this? This is where event logging, accounting, and security controls come into play. Let us now briefly look at these three ideas.


Event logging refers to the process by which we gather statistics for various pre-defined network occurrences or events. You may want to collect information for a wider variety of events than the set you need to bill for the service because billing models may change over time, and you never know what information you might need to validate your current billing model, or perhaps even design a new one. Having more data at hand is usually (though not always) better than having less. Of course, collecting more data puts more of a burden on the service and might impact performance adversely. So there is a trade-off here (so typical in engineering).


Accounting or billing (not strictly synonymous, though we use them inter-changeably here) is the process that tells you what to do with the data you have so painstakingly collected. Collating the data together, and generating bills falls into this bucket.


Security controls are needed from a billing perspective to ensure authentication, authorization and accounting functions (sometimes referred to as AAA) enable you to ensure that services are provided to parties who are who they say they are, are authorized by the parties using the service, who can then be billed with non-reputation guarantees i.e. they cannot later claim the service was delivered to someone else or that it was not delivered to them at all. These are supported in different systems in various ways.

... and the $$$ comes rolling in... (with examples)
The key to being able to generate revenue is to provide some service that adds value, and then get as large a user-base to sign up as possible, and then either bill the users themselves, or some third parties that benefits from the user population that logs into your portal (think advertising revenue), or some combination thereof. Start-ups these days tend to build a multi-platform model (sometimes also called a multi-sided market) where you first give away service for free to attract users, then attract advertisers to generate revenue, then change the provided service to a tier-ed structure where higher levels of service require payment from the user base as well. With the advent of cloud technologies, as user data gets stored more with the service provider (who can rent large amounts of storage space in the cloud from companies like Amazon), this raises user switching costs, making it more likely that users will stick with you, and you have a growing and captive user base. This last factor is a major contributor to the success of services like Facebook, Twitter, and LinkedIn. 

In Social Media based services, users value the networks they build, but these are hosted on the Facebook servers and are not things the users can pick up and leave with. The same goes for media about the users that the users themselves, or others inject into the system. Twitter users cannot take their followers onto a competitor's system unless a big mass of other users moves with them as well. All these services are successful primarily because of network effects. Metcalfe's law ("the value of a network is proportional to the square of the number of users connected to the system") holds strong sway over the services here. LinkedIn is a third example of a successful service - this one focused on somewhat more serious professional networking applications. One difference between LinkedIn and Facebook is that the former is somewhat more rigid. You can "unfriend" people on Facebook, but if you allow someone into your network on LinkedIn, you cannot take them out. This may have consequences to how these two services grow over time.


Does anyone remember Second Life from Linden Labs? They even used to have their own currency (called Linden Dollars) in the Virtual World. Somehow, Second Life went from all the rage a few years ago to a somewhat more quiet existence now. Guess there is a cycle to everything - turn, turn, turn.


We note some different ways in which mobile applications are monetizing value today. The first two models are obvious ones - 1. through advertising leveraging platforms like AdMob, and 2. by selling downloads at say $0.99 each. Lately though, more and more apps are selling "usage rights" - you can play the game so many times, or read so many documents with our software each time you pay a $0.99 fee. And perhaps periodically free usage "turns" can be made available through special promotions or at first download, to gain in popularity and drum up interest.


 This is similar to the "revenue sharing" model telecom equipment vendor firms sometimes use: give away the platform (hardware) for free, then have the service provider pay you a certain percentage for each transaction they bill the subscriber for.  This is also similar to how Apple bills application developers. You pay a fee to sell your app via the iTunes platform, then get a percentage of the total money you make from downloads.

[This is a place-holder for a work in progress.]