Apple Web Crawler Stokes Search Speculation

Developer Jan Moesen recently spotted an Apple Web crawler written in Go. He detailed the discovery on his website, where he said it is requesting HTML pages only, ignoring image files, CSS and such. He posited the question: is this a sign of an official Apple project, or is it just an Apple worker tinkering with some personal project? No one knows at this point, but it has stoked quite a bit of speculation, particularly that Apple may have a search-related project in the works, the likes of which are also speculation.

According to Moesen, the first spotted web crawler hit happened on October 15, with it identifying itself as Mozilla/5.0 (compatible; Fetcher/0.1). He goes on to point out what he believes to be a bug concerning setting a User-Agent header, leading to the discovery that it is written in Go.

That's the extent of what is known, which isn't much. The big question is whether this points towards some official project Apple has going on, and if so, what? Speculation has it this could be tied to Spotlight and its new ability to directly access web search content.

That, of course, is purely speculation and there is very little upon which it is hinged. Far less likely are those gossiping about possible Apple search engine aspirations. It could be a sign that Apple is working on some way to further distance itself from its competitors, but until something more concrete surfaces, there's no way to know.

SOURCE: Apple Insider