How do you deal with robots?

Requests made by robots (aka crawlers, spiders, bots) are filtered out from the usage statistics. We detect robots based on a standardized list of robots provided by the COUNTER and Making Data Count projects.