me, 2.0: graphical and passive browsing history

me, 2.0: jose nazario beauty and the street

graphical and passive browsing history

http://monkey.org/~jose/figs/http-graph-sm.png

one of the many things i've wanted my browser to do is to build a tree or directed graph based model of my browsing history. since i have yet to see a widget do this, i whipped up a small tool that does this: http graph. all of this data can be gathered passively, and it builds the graph based solely on the HTTP request header sent by the client:

GET /favicon.ico HTTP/1.1
Host: www.packetfactory.net
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.5a) 
            Gecko/20031030 Mozilla Firebird/0.6.1
Accept: image/png,image/jpeg,image/gif;q=0.2;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.packetfactory.net/projects/libnids/

from this you can get the URL (Host + GET string) and the referrer, meaning you can infer with high confidence how the user surfed there. this still leaves you with some islands, but you can now graphically see website structures and information.

the graph output format is in dot/neato format (via graphviz). while not always perfect, it's standard, decent and should work for most people. it writes the output to a file, /tmp/http-graph.dot. a Makefile line like this to produce a PS file output would look like this:

http-graph.ps: http-graph.dot
        neato -Tps -o http-graph.ps -Nstyle=filled -Nshape=box \
          -Ncolor=grey -Elen=3 -Ew=2 http-graph.dot

those weights and lengths balance between graph size and readability.

http-graph uses pynids, which is obviously built on top of python and libnids.

next Saturday, Oct 02, 2004 @ 06:53am | previous Thursday, Sep 30, 2004 @ 08:26am | archives

Last modified: Friday, Oct 01, 2004 @ 06:44am

Your Ad Here