In the world of Web search, you’re fairly limited as to what you can actually use. Sure, there are more than a handful of search engines out there, but a vast majority of them have to lean on Google to supply results when their index fails to deliver a useful result.
Some of these search engines boast enhanced privacy, such as Startpage, an independent search engine promoted widely by privacy advocates for its policy on never storing results and/or IP information from users. Alternative roots of Startpage, including Startingpage (Google Powered) and a proxy service that uses the engine’s proxy as a middle man to the sites you’d rather not have on your ISP record.
Recently, a new contender in the world of search has emerged, and this one turns the fundamental principals of search as we know it on its head. According to SMR Digital, YaCy is a peer-to-peer search engine that can power search across a corporate intranet or a larger internet. Peers on YaCy download a small app that powers a localized search engine you can access using your browser of choice. Searches made using YaCy are handled locally, queries are made to the greater hive of peers, and results are displayed in lightning-fast speed.
How YaCy Works
The index pool is dependent a number of factors. An intranet application of YaCy pools search results using a localized index made by computers existing on the network. Each peer contributes to the index through crawling and contributing their own search results to the community.
When you’re using YaCy for Internet search, you’re combining your own local index with the index of peers all around the world. Instead of a centralized server initiating and controlling site crawls, your client acts as a miniature version of this service.
There is no central body of control in YaCy. Each peer is an island within itself, and you have the ability to customize your level of participation, experience, and use of the search engine. You could opt to simply pull from the global pool of indexes without contributing your own, or you may opt to contribute more and enjoy the full capabilities of YaCy.
YaCy is run on an integrated NoSQL Database. This database stores a Reverse Word Index, Metadata, and source documents pertaining to the sites being indexed. YaCy peers are constantly exchanging index fragments using a Distributed Hash Table. This allows Index data to reach the peer before a query for that information is even submitted. In a sense, your peer has a miniature version of the whole search engine index stored locally, making it accessible even if you’re not actively connected to the global peer group. If something you search for isn’t present in your local index, YaCy will call out to peers to fill in the gaps.
Why is YaCy Important?
In a time where governments are censoring search, restricting communication, and even outright denying their citizens access to important information, it’s important that there be some form of peer-accessible Web interface available. TOR routers have changed the way oppressed civilizations experience the Internet, though this doesn’t completely solve the issue of major search engines such as Yahoo! and Google occasionally bending to the will of these governments in order to maintain their bottom line.
Google, the predominant force in the world of Web search, is arguably the most powerful company in the world. If your company doesn’t exist on Google, it might as well not exist on any search engine. By taking the power of indexing out of the hands of these corporations, and placing it back in the hands of users, YaCy has proposed an interesting shift in the way the Internet is catalogued and searched.
Think about it, how many companies were decimated by Google’s Panda upgrade? This update single-handedly ripped business out of the hands of honest and hardworking Web entities with little warning. While Panda may have been intended to hurt impostor and malicious sites, many honest ones also received a harsh drop in search ranking as a result. Otherwise honest businesses were forced to invest large amounts of money to combat the damage done by this update, and this all in the wake of significant losses resulting from the loss in traffic caused by lower page ranks. Many sites have yet to fully recover.
YaCy has no centralized search server or control environment. That means that rank is determined purely by the peers through keyword results and indexing. Censorship of search would be extremely difficult (if not impossible) as YaCy peers store so much locally.
YaCy is not without its drawbacks. As a peer-oriented search engine, it relies on an active and large pool of diverse users to build a large enough data pool to make search results more relevant and optimized for a wider variety of users. Currently, YaCy is built on a foundation of early adopters and tech-minded individuals, each with a similar search pattern and usage experience.
Below is a side-by-side comparison of my search results using YaCy and Bing. By clicking for a full-size image, you’ll notice a clear difference between the two, as one appears to rank more relevant links while the other reflects more primitive ranking.
Search results are hit and miss. For example, a search for “LockerGnome” revealed LockerGnome.com as the first result, but a mislabeled font site sits directly below it. YaCy appears, in its current form, to be capable of falling victim to older search spamming techniques that Google has mostly eliminated.
As with any open, community-powered platform, it will take time to grow and mold into something worthwhile for the average user. The tricky part about this particular platform is that in order to reach that point, it needs those average everyday users to fill in gaps and make searches that its current user pool would never try. Indexing is based on a combination of crawling and peer-generated results. Time and population are two factors that must be present in order for YaCy to truly succeed.
Is YaCy the Future of Search?
Chris Pirillo, the founder of LockerGnome and an advocate for user experience on the Web since the mid 1990s, is optimistic on YaCy’s general structure and what projects like this could mean to the future of search: “This is the future of search. If I were Google, I’d be nervous. Too many legitimate content publishers were hurt by Google’s round of ‘Panda’ updates.”
Google may be the current giant in the world of Web search, but in the world of business, it’s still the new kid on the block. How many more of these legitimate content publishers are going to continue to support and promote a search engine that basically drives them out of business through updates that have no obvious or recognizable advantage to the consumer?
Peer-to-peer search may be in its infancy right now, but as a global market continues to seek out new ways of finding information that are free of corporate or governmental censorship, the shift to a decentralized search engine is all but inevitable.
What do you think? Is YaCy the future of search?