http://crawler.trillinux.org/news G2 Crawler News Sat, 01 Nov 2008 15:25:35 +0000 http://backend.userland.com/rss092 en I'm going to explain how crawlers work. There are three main tasks that a crawler has to take care of. Find new hosts to crawl. Request data from a host that is being crawled. Display to the user the data gathered. This design lends itself well to being distributed. Several host crawlers (those that ... http://crawler.trillinux.org/news/2008/11/01/the-architecture-of-a-crawler/ My focus lately has been on hub uptimes. There is a new page showing hub uptime distribution graphs. It gives a visual representation of some of the categories on the uptimes page. The overall hub uptime distribution graph also features two vertical lines. The red line shows where the average ... http://crawler.trillinux.org/news/2008/10/19/recent-updates/ The crawler has been running pretty well with only minor tweaks from day to day which sometimes show up as blips in the graph. It was also down for a few days due to a failing hard drive. Yesterday the crawler got into the Foxy network again which uses the same ... http://crawler.trillinux.org/news/2008/07/10/quick-g2paranha-update/ Anyone who has read through this blog knows that the crawler has tended to crash fairly often. In recent times it was crashing to much to even continue running it. But rather than give up entirely I decided to write my own crawler. Five weeks later and g2paranha has emerged. ... http://crawler.trillinux.org/news/2008/06/23/g2paranha/ I was reading the Gnutella2 article on Wikipedia today and I noticed both entries in the External Links section point to my sites (crawler.trillinux.org and g2.trillinux.org). The latter being the new home for the G2 specs after gnutella2.com was allowed to expire. This got me thinking that it looks like ... http://crawler.trillinux.org/news/2008/02/28/the-state-of-g2/ I spent last weekend replacing my router with another computer. The transition was a bit bumpy but things are starting to get sorted out. More extended periods of downtime are possible over the next few weeks as I get things completely transitioned and working reliably. http://crawler.trillinux.org/news/2008/01/30/more-crawler-downtime/ The crawler has been down since Friday because I'm doing hardware work on my router. It is also the computer that does backups and it has had a slowly failing hard drive for the last few months. I finally bought a new hard drive and have been deciding how to ... http://crawler.trillinux.org/news/2007/10/08/crawler-downtime/ I added some new graphs back in April on the hub density page. They show the percentage of hubs with a certain number of leaves. This way the capacity of hubs can be tracked more granularly than just the average leaves per hub statistic. Let me know about other improvements you'd ... http://crawler.trillinux.org/news/2007/05/24/new-graphs/ You may have noticed that the number of "Unknown"/"??" countries has been increasing since the graphs came back in June. This is because the country is determined by using MaxMind's GeoLite Country database which maps IP addresses to countries. The version the crawler was using hadn't been updated since September ... http://crawler.trillinux.org/news/2006/10/11/country-database-update/ Today a new best hub uptime was established beating the old one of 209h 10m 45s. This is because for some reason unknown to me the crawler has decided to run for over a week continuously without dying. I did however check system logs recently and it does look like ... http://crawler.trillinux.org/news/2006/10/06/new-best-hub-uptime/