|
Years
ago, we bought a website log analyzer, known as WebLog Expert. It downloads
the raw logs from our webhost server, and provides several different kinds of reports
about the activity of our website.
Our webhost updates the logs at about
7:00am, so they're slightly out of sync, because the logs don't start and
stop at exactly midnight.
These
are the Summary page and Activity pages
for April, 2013.
Here
are the Summary and
Activity pages for June 9, 2012 through part of June 22, 2013.
(the log for June 22, 2013 is incomplete until the June 23, 2013
update runs, as mentioned above)
It's far too much work to do
all the individual
months, so the pages shown are as much as I can do now.
Use
your backspace key to return here from the above links.
We also use the
AXS Visitor Tracking
program, which shows visitor activity in real time. Here is our
AXS log for June 22, 2013 through part of June 23, 2013.
This is less than one full day. (warning: it's huge and it's pretty boring, too)
Sometimes, spiders or
robots scan this site, and I discover hundred
(occasionally thousands) of hits from these "crawlers" in the logs. Some
of them are legitimate search engines, such as Yahoo, MSN, Google, etc.
Others
are "scumbag scrapers" - they steal whatever content they want. They steal ALL your content.
Why are my pictures, music, and videos on websites in Russia and China?
So far, I've discovered over a dozen Chinese and Russian "scumbag scrapers".
I
researched ways to block many of them, but I doubt
anyone could block them all. Sometimes they are hitting pages, pictures, and videos, so they are
"hits" - even though they're "Definitely Not Wanted Hits".
The blocks are immediate, and they
aren't even logged in AXS, but are
logged in the website logs.
I wrote a few batch files to extract just the "scumbag scrapers" from
the logs, and it looks like some of the blocks are working pretty well.
Can you
believe this? It's ONE "scumbag scraper" in Russia. I did a "FIND" on the website log, and isolated "Baidu" to a little text file.
What's amazing is it's only 8:09am through 9:05am! ONE HOUR! This happens a
lot. These "scumbag scrapers" are NOT GOOD.
180.76.5.195 - - [26/Jun/2013:08:09:01 -0400] "GET
/error/error403.htm HTTP/1.1" 302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0;
+http://www.baidu.com/search/spider.html)"
180.76.5.137 - - [26/Jun/2013:08:09:02 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.64 - - [26/Jun/2013:08:09:02 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.176 - - [26/Jun/2013:08:09:03 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.151 - - [26/Jun/2013:08:09:03 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.98 - - [26/Jun/2013:08:18:28 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.87 - - [26/Jun/2013:08:18:28 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.169 - - [26/Jun/2013:08:18:29 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.97 - - [26/Jun/2013:08:18:29 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.19 - - [26/Jun/2013:08:18:30 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.23 - - [26/Jun/2013:08:27:55 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.142 - - [26/Jun/2013:08:27:56 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.154 - - [26/Jun/2013:08:27:56 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.25 - - [26/Jun/2013:08:27:57 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.151 - - [26/Jun/2013:08:27:57 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.54 - - [26/Jun/2013:08:37:22 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.26 - - [26/Jun/2013:08:37:22 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.62 - - [26/Jun/2013:08:37:23 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.149 - - [26/Jun/2013:08:37:23 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.21 - - [26/Jun/2013:08:37:24 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.57 - - [26/Jun/2013:08:46:49 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.197 - - [26/Jun/2013:08:46:49 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.26 - - [26/Jun/2013:08:46:50 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.15 - - [26/Jun/2013:08:46:50 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.6.213 - - [26/Jun/2013:08:46:51 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.194 - - [26/Jun/2013:08:56:16 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.6.35 - - [26/Jun/2013:08:56:16 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.189 - - [26/Jun/2013:08:56:17 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.195 - - [26/Jun/2013:08:56:17 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.6.26 - - [26/Jun/2013:08:56:18 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.150 - - [26/Jun/2013:09:05:43 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.6.222 - - [26/Jun/2013:09:05:43 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.89 - - [26/Jun/2013:09:05:44 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.65 - - [26/Jun/2013:09:05:44 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.6.35 - - [26/Jun/2013:09:05:45 -0400] "GET /error/error403.htm HTTP/1.1"
302 228 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" |
