1. Find new hosts to crawl.
2. Request data from a host that is being crawled.
3. Display to the user the data gathered.

This design lends itself well to being distributed. Several host crawlers (those that perform task 2) can all be working in parallel and independently. All the host crawlers need is a coordinator (the one that performs task 1) to feed them lists of hosts that aren’t duplicated. The host crawlers then send their responses back to the coordinator which finds new hosts from the responses and then stores the responses. Lastly the aggregator or statistics generator (the one that performs task 3) periodically runs through all the data collected and creates useful ways for the user to view this information.

That’s how a crawler works in general terms. But the fun begins when a crawler has to actually be built. One of the most important decisions is how to store the data collected and how to store and distribute the list of hosts that need to be crawled next.

Relational Database Approach

This is the approach that g2paranha takes. It’s an easy and straightforward way to store data for anyone trained on these traditional databases. But the data that a crawler needs to store is for the most part not relational except for the links between hosts. Another problem with relational databases is that many of them lock all of the data whenever data is being written or read. This creates a huge bottleneck in a distributed environment where lots of both of these operations are being performed. So while it is easy to implement it may not be an optimal solution. On the positive side the extremely powerful SQL language is available for extracting statistics from the data.

Non-relational Database Approach

This is the direction I have been heading in for the crawler. It seems to be a good fit for the type of data that the crawler needs to store. But I have very little experience in this are. So far my only forays  into this field have been with CouchDB which is still in heavy development. CouchDB looks promising but I haven’t had much luck with getting it to work.

So if anyone has experience in non-relational databases or in creating distributed crawlers I’d like to hear from you.

The other major addition is to the uptimes page. The second table of information is new and expands on the information in the first table. The new table shows for each grouping/category:

• average
• median
• minimum uptime
• maximum uptime
• the total number of hubs that fit this category
• the number of hubs below the average for this category
• the number of hubs above the average for this category
• the ratio of hubs under the average and over the average

On the vendors page I added back the showing of the GDNA data. But for now no hubs will be showing up as GDNA. GnucDNA does not send any vendor code at all so the current crawler just shows them as UNKN. The old crawler looked at the hub’s User-Agent to figure out if it was GnucDNA and then set the vendor code appropriately. So while new data won’t be logged the same way the trend of GDNA can at least be seen again on the yearly vendor graph.

Lastly, there is an experimental feature that shows where each hub is on a geographic map if they provided that information in their profile. Of course most users do not reveal this information and some lie about their location but it can still be a lot of fun. You get to see that the G2 network has users all over the world and they’re all interconnected. This mapping feature is experimental because it requires a fast computer in order to work smoothly. So give it a try but understand that it may not work well for everyone. In my experience Chrome did really well while Firefox and IE did poorly so if you already have Chrome installed you might give this feature a try in that browser even if you don’t use it for anything else.

Green lines are used to show hubs that are connected together. Clicking on a marker will reveal additional information about that hub:

• username if provided
• name and version of the software they are using
• number of hubs and leaves they are connected to
• the actual country they are in based on their IP address and MaxMind’s GeoIP
• the exact coordinates they provided

I hope you like the additions. Keep the suggestions coming.

Yesterday the crawler got into the Foxy network again which uses the same protocol as G2 but is a private network separate from G2 probably using GnucDNA’s authentication scheme. Their network is significantly larger than the G2 network as can be seen from the spike history graph. Since it is so much larger it presents problems for crawling and since it doesn’t represent the true open network of G2 I have filtered it out from being crawled.

Over the next few weeks expect there to be a few bumps as any bugs get ironed out. I’ll also be rounding out the feature set provided by the original crawler. Right now that means adding country flags and keeping track of unique nodes. This last feature is the best estimate of the network size so until it is implemented the network counter won’t be put back on the front page. The leaf count on the history page gives an over count of the network size because each leaf can connect to multiple hubs.

I’ll try to keep the news updated as I progress.

This is partly to benefit others and partly out of self-interest. I don’t think the G2 protocol as a whole is all that spectacular anymore, if it ever was. But parts of the protocol can be reapplied to accomplish other things. For example, at its core is a specification for a compact, extensible tree structure for communication. This could be made generic and used for all sorts of applications outside of G2. The search mechanism of a random walk is not original or unique but it’s the largest P2P network I’m aware of that still makes use of it so from that perspective it could be interesting to study.

I run the crawler out of self-interest. I like data, statistics, and graphs. I never turn down the opportunity to collect raw data and turn it into graphs and make inferences from the data.

I started maintaining the G2 website sometime in (late?) 2005 and moved it to its current home in October 2007. The crawler has similarly been running since late 2005. Here’s to many more years to come.

Let me know about other improvements you’d like to see and I will try to make them happen.