Archive for August 2009

In 1965 then a co-founder of Fairchild Semiconductor, Gordon Earl Moore predicted that the number of silicon components on a single integrated circuit chip will double every 12 months. He later went on to start Intel with Robert Noyce and worked to make that prediction a reality.

True to his prediction the number of transistors on a microprocessor has been doubling at roughly 18 months for the past 45 years and this trend has come to be known as Moore’s law.  It’s estimated, for example, that the semiconductor industry produced more transistors in 2005 than the number of grains of rice produced that year.  For a deep look at the remarkable history of semiconductors explore this exhibit at the computer history museum web site.

Unlike Newtonian laws of motion or laws of electromagnetism, Moore’s Law is not one of nature but of human endeavor. In a symposium honoring the 40th anniversary of that prediction Moore himself stated that the law is based more on economics of the industry than on improvements in any particular material or process and has room to go on for another decade at least.  In 2009, Intel CEO Craig Barrett said “We can scale it down another 10 to 15 years. Nothing touches the economics of it.”  There has been ample speculation and analysis (like this one by author and blogger Andrew Curry) on when Moore’s law will end but another decade of it will surely lead to lot of creative destruction.  Former chief editor of Wired magazine Kevin Kelly goes so far as to ask was Moore’s Law inevitable?

Fundamentally Moore’s law upends status quo by doing two things. In every cycle (18 months to 24 months) the cost of silicon components falls by a factor and the performance improves by a factor. Such an exponential decrease in costs and increase in performance offers ample opportunities for nimble and new players to outsmart the slow and old ones and win customers. 

What are the implications of Moore’s Law for web search?

Web search as practiced by Google, Yahoo and Bing today essentially boils down to building this stack : crawling, indexing, ranking and presenting web sites as search results to a keyword search.  Each layer of this stack is affected by Moore’s Law differently and offers different opportunities.

1. Crawling : Crawling  is inherently a massively parallel task. You maintain a huge distributed crawl queue and manage several processes pulling the data from the web based on that queue.  Cheaper hardware and bandwidth lowers the cost of doing such a crawl every cycle.  But unfortunately the amount of data on the web has also been growing exponentially. The web is said to have grown from 5 exa bytes of data in 2002 to 281 exa bytes of data today. One would need enormous cash reserves to fund a crawl that big despite cheapening hardware. Newcomers without access to such cash reserves are at a huge cost disadvantage.
2. Indexing : Indexing is the task of organizing all the crawled data in ways that influence ranking and presenting.  Modern day search engines build what is called a reverse index that maps keywords to websites they are present in. The process of indexing is a massive, resource intensive task, and newcomers are usually at a cost disadvantage, despite cheapening hardware.
3. Ranking : Ranking is the task of retrieving relevant results and ordering them.  The secret sauce of each search engine is said to reside here. It involves proprietary algorithms and is carefully controlled. Google is said to use about 200 signals that influence ranking.  In some ways Moore’s Law does not directly affect this layer even though most of the analysis that one would need for ranking is done during the indexing phase. Fortunately offerings like Yahoo Boss give newcomers the chance to improve upon the ranking without building a huge crawl or index and the cost of such services are coming down with each cycle.
4. Presenting : Presenting is the task of rendering search results in a manner that’s efficient, useful and easily consumable. Despite the heavy lifting that happens at the backend in the previous layers what a typical web search consumer sees is one text box and a list of ten blue links on a sparse search results page.  Moore’s Law offers the biggest opportunity for newcomers here as consumers upgrade their personal computers each cycle.  Studies show that users expect to see results to their keyword searches in half a second. And –with Moore’s law– what one can do in that half a second improves exponentially every cycle, both on the client side and on the server side. What was not possible to do in the last cycle suddenly becomes possible.  With the rekindling of the browser wars tomorrow’s web browser will be vastly more powerful taking advantage of the next generation of multi core processors. They will offer much richer rendering and new forms of interaction.  Newcomers to web search will have the best opportunity to disrupt the status quo by riding that wave. One need not limit the interface to keyword search. With the explosion of touch and voice interfaces in the last year or so, new ways of input will emerge that will make today’s keyword search seem archaic.

posted by Manyam Mallela

AMA Social Media Event

I was in San Francisco last night attending a social media event entitled, “Advertising, PR, and Marketing Suck! Now What?” hosted by the American Marketing Association. The event had a good line-up of speakers including Guy Kawasaki, Renee Blodgett (President and Founder of Blodgett Communications), Louis Gray (from louisgray.com), Loic Le Meur (Seesmic founder), and Steve Patrizi (VP of Sales, LinkedIn). Guy Kawasaki did a great job as the moderator and made the event fun, interactive and informal with his witty questions. Here are some of the highlights:

What would you do if you had a great product in a niche and zero dollars in marketing?

Reach out to your personal network such as friends, family, and colleagues and possibly get some thought leaders to use your product. Focus on the second tier of bloggers, make them fans of your product and they will market the product for you at no cost. Invest in building a community by being honest and transparent; however be aware that this takes time. Release your product as soon as possible, learn from mistakes, and be honest to your consumers about fixes. Another idea is to find your consumers on Twitter, follow them, and engage in conversations. However, be careful, since you are walking a thin line between good marketing and spam here.

What would you do if you now had $10,000 dollars in marketing? How would you put it to use?

Hire a community relations intern to listen to your customers, actively respond to every piece of feedback, and turn customers into advocates. You can also try creating a fun video of your product and distribute it virally. Thousand dollar press releases are an option too, but only for communicating major changes. One should also invest in targeted online marketing with Google AdWords. Louis Gray has seen success selling even $350,000 products using just AdWords.

What is role of an advertising agency specializing in social media for big companies? Should they just do it on their own?

Everyone agreed that advertising and PR firms add significant value. Especially since big companies need to change old ways and adapt to social media. Often CEOs come from a technical background, and an agency plays a critical role in getting the messaging and branding right. However, simply hiring a firm is not enough  –a company has to commit resources to working actively with the agency on responding to the feedback received from customers.

What would you do if a small handful of people are attacking your company on the web (e.g. Twitter, Facebook, etc.)?

If the complaint is genuine then acknowledge and communicate what you are doing to fix it. However, if this is some aimless ranting, then learn about who they are. Often such people have few followers and it is not even worth your time to respond.

In summary, the big difference between traditional marketing and social media is trust. No one trusts a brand, but we trust what our friends say. Word-of-mouth and viral marketing in the social media world have an unprecedented opportunity. However, social media is not end-all. Instead, it is new infrastructure, which augments traditional marketing in a way never seen before. Before one had twenty influencers to reach, now we have thousands! Nevertheless, marketing’s main purpose, like before, is still to get the messaging and branding right, identify target customers, and act on feedback.

One of the very first members of the Kosmix team, Vijay Chittoor, has a regular bi-monthly column on Inc.com.  Every other month, Vijay shares insights about technology to help small and medium business owners improve their operations.  This month’s piece offers advice on how smart companies are using social media tools to outsource work to their customers.

Here is Vijay’s latest column, reprinted from Inc.com:

One of the biggest challenges for entrepreneurs is to scale up their business, and to manage the growth by hiring more people in every function. But what if you could achieve growth by just letting your community of users do most of the work?

Several creative companies have used social media tools to get their customers involved in core aspects of their business, all the way from marketing to product design, product testing, and customer service.  Here are some great examples of organizations that are using social media to drive sales and efficiencies, while still connecting with customers:

Effective marketing using social media

By now, most people know that social media provides many tools for creating brand awareness, as well as for generating sales leads. Fiskars, a Finland-based manufacturer of scissors realized that scissors are very popular among scrapbookers, and set out to reach this community. After identifying four Fiskars users who were extremely passionate about the brand, the company set them up with a website and a blog, and made them consumer evangelists. The “Fiskateers” program has since then grown to more than 5,000 Fiskateers across 70 countries, each actively blogging and evangelizing the brand. Having so many “marketers” on its payroll would certainly have been unsustainable for the company, but by leveraging the power of its community, and using online tools like blogging, Fiskars has created a strong brand identity among its target audience.

Blogging isn’t for you?  Try Twitter to connect with your audience. Naked Pizza of New Orleans, which prides itself on making the “world’s healthiest pizza,” has latched onto Twitter as a means of promoting its fresh ingredients and offering promotional deals. Twitter has been so effective that they’re now using billboards to drive more people to the Twitter account. More and more restaurants are finding Twitter to be an effective way to boost their sales.

Finally, no discussion of social media marketing is complete without talking about viral videos. Blendtec, a division of the Utah-based K-TEC, manufactures high powered, durable, commercial blenders. In 2006, Marketing Director George Wright had the unenviable task of creating a brand campaign with a budget of $50. When Wright saw CEO Tom Dickson and some engineers testing the blenders with heavy duty chunks of wood, he hit upon an idea and used the $50 to buy the domain http://www.willitblend.com. Since then, the “Will it Blend” series of videos has seen more than 80 million views on YouTube and increased Blendtec’s sales by more than 700 percent.

Involve customers in product design

How can you add value and create customer loyalty if you don’t even control your product design process? Threadless, an online T-shirt store operated by the Chicago-based skinnyCorp, has found the secret to that, selling more than a million T-shirts a year, none of which were designed by the staff. All the designs are submitted and evaluated by the community of users on its website. Hundreds of artists submit their designs, and users vote on them. Every week, the best designs are selected for printing, and the winning designers get $2,000 in cash, $500 in gift certificates, and another $500 for every reprint. According to some reports, the company generates more than $30 million in revenue and $10 million in profits.

Muji, a Japanese retailer, has latched onto a similar concept through its website muji.net, where it invites submissions for innovative furniture designs. Muji, which means “without brand,” has a community of half a million people who submit and evaluate designs.  Shortlisted designs are then sent to professional designers, who polish them before sending them off for production.

Web companies often launch products in a “beta” state and invite selected users to test the product. Joffrey’s Coffee & Tea Company took this idea and applied it to coffee. It invited bloggers to beta-test its coffee by sending them free samples. More than 1,500 bloggers participated, and generated enormous buzz for Joffrey’s on the Web. Based on feedback from these bloggers, Joffrey’s launched Coffee 2.0 with many “bug fixes and improvements.” Even the name Coffee 2.0 came from one of the beta testing bloggers. Not only did Joffrey’s use social media effectively to do product testing and improvements, but it also created enormous buzz around the product.

Get customers to help with customer support

Customer support is one of the most difficult things to scale as the business grows. Consumers are increasingly logging on to social media sites to express their frustration with poor service. For example, the consumer complaint video “United Breaks Guitars ” has had close to 5 million views on YouTube.

Innovative companies are using social media in a couple of different ways to provide customer support. eBay has outsourced almost its entire customer support function to its users from its very beginnings. In his book The Perfect Store, Adam Cohen writes about eBay in 1996: “Omidyar did not have time to explain to each individual user how to write a listing in HTML, or to give advice on bidding strategy.” The solution was to launch a Bulletin Board where users could “gather, share information and ask for help.”  Later, eBay ended up hiring some of the people who were the most active and helpful on the forums to work for it, answering customer emails and providing additional support.

A different model of support treats social media as another channel for the in-house customer support team. Frank Eliason, Comcast director of digital care, has a following of more than 25,000 people on his “Comcast Cares” Twitter account, where he answers user questions. The real-time nature of Twitter and its search functionality allow Eliason to even reach out to Comcast users who haven’t actively sought help.

By applying a bit of imagination to social media tools like blogs, Twitter, Facebook, and YouTube, these forward-thinking companies have grown their businesses by leaps and bounds. Take cues from these examples of the power of community, and you’ll avoid some of the growth pains that arise from controlling and managing all of your business functions in-house.

www.tvclassic.net

Startups lead very exciting lives. They are born of a core group of people coming together to give shape to an idea and go through many transformations before they reach an outcome. While many times the outcome is dissolution or sale, some companies make the leap to big company land.

During this process, there is a key transition step where they are no longer a small startup, but haven’t fully made the transition to a established company. This is similar to adolescence among humans, where we begin to form our identity and understand our place in the world. Successfully making this transition involves active guidance and support from the founder and management.

Although the strongest leaders in the start-up world have very different personalities and management styles, they tend to share the philosophy that communication and staff development are key factors in building a strong company. Here are a few management pointers from the pros:

1. Keep in touch with your employees: It’s easy during the growth phase to lose touch with your employees as management levels are added. While this distance is natural, bridging it by talking to employees regularly will give you insights into how the transition is proceeding
2. Communicate clearly and often: When you are small, communication via hallway and water cooler conversations might work, but as you grow, you need to evolve strong communication channels that work both ways.
3. Develop Mentors: In addition to the founders and execs, develop other senior leaders into mentors. Take steps to ensure that any employee who needs help or has growth aspirations has an advocate to talk to and learn from.
4. Hire carefully: There is usually a lot of pressure to grow quickly, and that typically leads to the pressure to staff up quickly as well. However a few bad hires can really cause a lot of pain. So pay close attention to your new hires and assist people in making alternative career choices if they don’t fit your organization anymore.

Let me know what you think of these and feel free to chime in with your own!

Marissa Mayer (image from Wikipedia)

A bunch of us from Kosmix went to hear Marissa Mayer speak at PARC on “Innovation at Google: The physics of data”. We were not alone: the auditorium was packed beyond its capacity. For Google data is everything, and, rightly so, given that it has the coveted and ever-growing “Database of Intentions”. Marissa sees Speed, Scale, and Sensors as the three pillars driving the next quantum leap in this space.

Speed

Marissa explained how the real-time web and services like Twitter, Facebook, and YouTube are creating data at a staggering space. Every time Google crawls the Web, it finds 20% new data, not seen in the last 90 days. Online activity like searching on Google and buying products on Amazon creates data, which is used by applications such as Google Trends, and Amazon Product Recommendations. This real-time data can be applied in creative ways to solve real-world problems. For instance, Google uses search activity to visualize flu trends and can use it to predict epidemics.

Scale

All this data is useless if you can’t analyze it quickly, which is why Marissa puts great emphasis on scale. Google has always had a formidable infrastructural advantage here, both with huge server farms and software innovations such as MapReduce. However, innovations outside Google such as Amazon Web Services and Hadoop also provide great alternatives for companies that need to do some serious number crunching.

Marissa demonstrated various Google products such as Google Squared, Google Public Data, Fusion Tables, Flu Trends, Google Trends, Similar Image Search, Gapminder, and Google Map Maker to illustrate the amazing data visualization systems possible once data is available in a structured format. Google Fusion Tables, for example, combine scientific data to provide a more visual way to analyze health trends like the leading cause of death among children in different regions around the world. Similarly, Google Public Data, which makes government data more accessible, charts statistics like unemployment rates by county. Another tool, Gapminder, lets you build visual correlations between carbon footprint and income per capita. She also stressed upon the importance of user-generated content. For instance, by making map-editing simple, Google Map Maker enabled the community to create detailed maps in countries such as India.

Sensors

As the Web starts to meet the real-word, sensors play an increasingly important role. Your cell phone has eyes, ears, touch, location, orientation, and can connect with others using the web. This is exactly why Android is a strategic bet for a company like Google, which values data immensely.

Love your Data

At every turn, Marissa’s talk emphasized that Data is King. But you don’t have to be Google to use data to make a difference. Every company–big or small, web-based or offline—that generates data should treat it like gold: Save it, analyze it, visualize it, standardize it, share it, and think how you can generate even more.

We’ve all heard it. Websites and web brands need to always provide instant gratification to their consumers. And who better exemplifies this than the brand that is practically synonymous with the internet: Google Search. Enter in anything, whatever broad or specific string of words you want, and you will get back a list of information to instantly satisfy you.  A list of information that is “relevant” to what you searched for. But is “instant gratification” and “relevancy” really going to make you smarter? Is it really going to make you more informed? More worldly? Really give you the ideal experience to give you the information you need?

Sometimes, yes, it works. Need directions? Need an official homepage? Need an article where you know the exact title? Sure, Google rocks for these things. But want to know about Ancient Greece? Need to plan a trip to Napa? Want to know about the history of Vogue magazine? How about why Erykah Badu grooves so well? Or why Tesla Motors is a big deal? There is no one right answer. When information exists in a plurality, and not a singularity, the format of Google Search we’ve come to rely on really fails us. Because what you really need is something to engage you. Just as the editors and writers of books, magazines, and newspapers engage you. You’ll find some things in Google Search to feed your hunger for knowledge, but doesn’t it just strip away all the things you would typically get from traditional learning tools? Instantly telling you an answer is vastly different from actually taking the time to learn.

I love to read. I love to learn. And I truly believe most people also like to read, learn, and know more about the world around them. I am an extreme case however. The library was my favorite place as a child. To go in and get lost, see all the books on the shelf, scan through them, read about the authors, ask the librarian questions. In all of these acts, there is context to the learning. Not just a flat page of links. Google search is akin to ripping all of the pages out of all the books, turning the lights off, and scrambling around to find a page you can use. The process of learning takes time. Getting to know about something takes context. And when it really matters to you, it should take more time, and should have the same context you would get in the real world.

So yes, there are sites that give you some context, and encourage you to learn and be informed. Wikipedia for instance. However we don’t know if the articles are true, and it is merely an online equivalent of a print encyclopedia. Mahalo sets humans to the task of making search results pages, thereby making pages more accessible. None of these encapsulate going to your local library, going to the right Dewey Decimal shelf, and getting the right context and the most informed you possibly can. And with the explosion of new information types on the web: conversations, social networks, videos, images, blogs, how-to, reviews, and more; no one website has encapsulated these things while bringing the same context of learning in the real world as if in the library, until now. This is what we strive to do at Kosmix. To be that editor that decodes search results, brings context to what is behind the links, and give you the multi-dimensional, rich information as a reference tool that works. And, hopefully, Kosmix Topic Pages will work so well that maybe we can get beyond instant gratification, and learn how to love learning again: with time, patience, and imagination.

The internet is abuzz today with Google’s announcement on a next generation search project that the internet has fondly dubbed “Google Caffeine”. I read the news along with my own morning shot of caffeine, and it immediately increased my sense that exciting things are happening again in the search space.

Take the last few months. The launch of Bing, the MicroHoo deal, a search box taking prominent place on the twitter.com homepage, Google launching the “Wonder Wheel”, Facebook turning on Real-Time search, and today’s news on Google Caffeine. Clearly, the search industry itself seems to be on a heavy dose of caffeine recently.

Google Caffeine is probably a major backend release for the Google search engine,and, like any other release, it will be faster and better. After all, search engines need to constantly innovate and update to keep up with the competition, and Google has always done that better than anyone else.  You can try it here.I definitely see an incremental improvement in quality.

Things get more interesting when you consider that Google clearly must have several big releases for their search product every year. However, lately Google not only has to improve its product, it must also communicate these improvements and be “seen” to improve its product. After several years of stability, search is heating up, and the market leader must maintain its brand or lose it forever.

This is a typical cycle in any industry reliant on innovation.A short period of rapid innovation, followed by a longer period of consolidation and stability again… followed by a rapid period of innovation. After years of the traditional search model, search seems to be entering its next period of rapid innovation, and all the big players, as well as the smaller ones, must innovate or be left behind.

So to me, in my moments of caffeine-induced clarity, I see “Google Caffeine” as another indication that we are about to see a revolution in search. My hope is that this will deliver on search’s many possibilities!

All of us at Kosmix were very excited to see the story about Facebook’s acquisition of FriendFeed. It was just last week that FriendFeed co-founder Bret Taylor dropped in at the Kosmix HQ for a Lunch 2.0 panel with Kosmix & Mozilla.

FriendFeed has been a pioneer in the social networking space, introducing many features that were quickly copied over by Facebook, e.g. the “Like” button. FriendFeed is also a great illustration of the potential of data portability. Interestingly, Facebook has been one of the first major backers of the Data Portability project, and one of the first comments made by the 2 companies after the acquisition affirmed their shared belief in openness & portability.

Everyone at the Lunch 2.0 panel would remember Bret’s succinct definition of what success means to him: 6 billion users on Friendfeed! Now, with the power of Facebook behind them, we hope the Friendfeed will be able to nail that soon. For more insights and advice from Bret, check out the highlights video posted on Kosmix’s about page or on Vimeo.

(Iceberg image ©Ralph A. Clevenger)

Experts estimate that search engines can access less than 1% of the data available on the Web, only the tip of the iceberg. Where is the rest of Internet’s data? It’s lurking in the Deep Web.

What is the Deep Web?

The Deep Web is defined as dynamically generated content hidden behind HTML forms that is inaccessible for search engines’ crawlers. Deep Web is also referred as the hidden or invisible Web.

The Deep Web consists of three key elements: (1) pages and databases accessible only through HTML forms; (2) disconnected pages not accessible to crawlers; and (3) password protected and subscription only sites. Some people also include real time Web data as a part of the Deep Web, since it’s changing so fast that traditional search engines are not able to surface it in their results.

How Vast is the Deep Web?

According to one study by Michael K. Bergman in 2000, the Deep Web accounted for 7,500 terabytes of data. At that time, search engines could index only 10s of terabytes of data. By 2004, a subsequent study by Kevin Chang and his colleagues estimated that the Deep Web had grown to more than 30,000 terabytes of data. At this rate, one can only imagine how vast it must be today, particularly given the ubiquity of the Internet over the past five years. Such an enormous amount of data has huge wealth of information—the key is figuring out how to access it.

Is it Possible to Access the Deep Web?

Absolutely—though it’s not easy. There are two main approaches to accessing Deep Web data:  run-time integration, and off-line indexing.

In the run-time integration approach, one has to build a system that performs the following tasks: first, figure out the appropriate forms that are likely to have results for the given query terms; second, map the query terms suitably to search those forms and integrate the results from various forms; and third, extract relevant parts of results to display. This approach enables richer experience for users, and sites like Cazoodle.com seems to rely on this method.

But there are some drawbacks to run-time integration. It’s extremely difficult to figure out appropriate forms for the given query terms. In addition, mapping query terms to search those forms and extracting information from the results is highly labor-intensive tasks and not very scalable.

In the off-line indexing approach to access Deep Web data, one has to construct a set of queries to search through forms, process the queries through forms while off-line, and index the result. Once the query set is constructed, this approach can reuse the search engine infrastructure for crawling, indexing results, and index serving.

Google has taken this approach to surface Deep Web content. However, algorithmically constructing input values for forms is a non-trivial task. Furthermore, this approach cannot be applied to HTML forms that use HTTP POST method, since the resulting URLs are the same, and form inputs are part of HTTP request rather then the URL.

The Kosmix Approach to the Deep Web

At Kosmix, we surface Deep Web content by using a combination of run-time integration and off-line indexing approaches. At the core of Kosmix technologies are (1) a sophisticated categorization engine that enables mapping of query to appropriate category; (2) a highly scalable fetching and run time integration system to fetch data from various sources, integrate, and provide rich experience; and (3) an off-line crawling and indexing systems that enables scalability.

For example, for a query like “Ravioli”, we show nutritional values from Fatsecret.com. Our categorization technology enables us to identify Ravioli as a food query, and enables us to surface Deep Web content from Fatsecret.com.

The Next Hurdle

While invaluable treasures are hiding behind the Deep Web, there are significant challenges to solving the problem of reaching this information. The next step for search engines will be to find an easier way to tap into the Deep Web, and to keep up with the Real Time Web.

My prediction? The Deep Web will force a drastic change in how traditional search engine systems are designed and built.

References:

[1]. Michael K. Bergman. White Paper: The Deep Web: Surfacing Hidden Value. http://brightplanet.com/index.php/white-papers/119.html, 2000.

[2]. Kevin Chen-chuan Chang, Bin He, Chengkai Li, Mitesh Patel, and Zhen Zhang. Structured Databases on the Web: Observations and Implications. In SIGMOD 2004.

[3]. Cazoodle. http://www.cazoodle.com/docs/Press_Kit.pdf, 2009.

[4]. Jayant Madhavan, David Ko, Lucja Kot, Vignesh Ganapathy, Alex Rasmussen, and Alon Halevy. Google’s Deep Web crawl. In VLDB 2008.

Thanks again to all of you who came out our Lunch 2.0 panel yesterday with Mozilla and FriendFeed.  It was an awesome afternoon, and we have the photos to prove it!  Click on the thumbnail to see the larger image.

And don’t miss this five minute video with highlights from the panel!