Search will remain the most exciting aspect of the web experience. Content creation and search will keep feeding on each other ad infinitum. Communication is important, but not the number one function on the web. It is content consumption. Search, search and more search.
Step one is conceptual. What would be the best possible formula? Twitter does not feel the need to index the entire web. The idea that because Google has the largest index of webpages and so it will always be number one has been challenged in ways small and big recently.
Twitter said no thanks absolutely to indexing. Wolfram Alpha said forget the 10 blue lines, let me get you straight to the answer. Bing said maybe we can't do better at search, and we are almost as good, but how about trying to beat Google on the presentation of search results? We are not a search engine, that would be Google. We are a decision engine, we will help you make better, faster decisions.
Disclosure: I have yet to visit the new Bing page, but I hope to shortly.
If you think about it, Google is like Twitter. Most links placed on the web are human decisions. I decide what sites and blogs and news articles and videos to link to from my blog. A purely machine oriented search engine would not care about who links to whom. It would be about how often those links are clicked on to generate visits to your site, to your page. But then depending solely on visits to rank a site might also not work. The most popular are not necessarily the best. Or we would all end up on the CNN site to learn about the latest in quantum physics.
If we could come with a great formula then we could ask, do we have the technology to deliver? Can we get it if we don't have it?
And that is not even getting into the niche search engines. You can limit to show your AdWords ads within a certain geographical region. You should be able to localize your search similarly. Localize in terms of space and time. Show me only pages that were created during the past hour on this topic. The past minute. The past 10 seconds. The past minute in Queens. On this topic.