News

Network size

25 May 2009

The network size is now featured on the front page once again. When the new crawler was implemented that statistic had to be dropped because it was too resource intensive to calculate with how the new crawler worked. But now that issue has been resolved. Some background The number of leaves on the network isn’t a good [...]
A Quick Update

12 February 2009

I haven’t made a post in awhile so I thought I should. Not much is going on with the crawler right now. I’ve been pretty busy lately and haven’t had any time to spend on improving the crawler. However there were a few subtle updates to many of the webpages. More detailed descriptions were added to [...]
The Architecture of a Crawler

1 November 2008

I’m going to explain how crawlers work. There are three main tasks that a crawler has to take care of. Find new hosts to crawl. Request data from a host that is being crawled. Display to the user the data gathered. This design lends itself well to being distributed. Several host crawlers (those that perform task 2) can all [...]
Recent Updates

19 October 2008

My focus lately has been on hub uptimes. There is a new page showing hub uptime distribution graphs. It gives a visual representation of some of the categories on the uptimes page. The overall hub uptime distribution graph also features two vertical lines. The red line shows where the average hub uptime is and the [...]