Bots Now Account For 61% of Net Traffic 124
codeusirae writes "A study by Incapsula suggests 61.5% of all website traffic is now generated by bots. The security firm said that was a 21% rise on last year's figure of 51%. From the article: 'Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections. But the firm said the biggest growth in traffic was for 'good' bots. These are tools used by search engines to crawl websites in order to index their content, by analytics companies to provide feedback about how a site is performing, and by others to carry out other specific tasks - such as helping the Internet Archive preserve content before it is deleted.'"
Youtube? (Score:5, Interesting)
Didn't we just get studies that said youtube and netflix were 50% of the net's traffic?
http://mashable.com/2013/11/12/internet-traffic-downstream/
Was this just a ruse? Is this study wrong? Is there some sort of overlap?
Re:Youtube? (Score:5, Informative)
Story is about website traffic, not network bytecount.
Re: (Score:2, Informative)
Youtube is a website. Netflix is a website.
Re: (Score:3)
Netflix is also an Internet service, Ioutube is also an Internet service. Before the web, we had Internet services. Not everything is a website. DNS, NTP, email, for example.
However, note that GP is wrong. Story is about "Net Traffic" not website traffic...
Re: (Score:1)
RTFA! The report is about visits (read as page hits) not bytes. The visits to youtube and netflix most often result in a length look at a single page.
90% of the cells in the human body (Score:5, Insightful)
are bacteria. Viewed that way, basically humans exist to transport and feed bacteria. However that's 90% by cell count, not cell mass or total DNA. Looked at it that way the bacteria are assistants.
The bot traffic is light weight it outnumbers human traffic in site visits not byte counts. It exists to serve us.
Re:90% of the cells in the human body (Score:5, Funny)
It exists to serve us.
You must be new here.
Re: (Score:2)
It exists to serve us.
You must be new here.
And you have never seen "To Serve Man".
Re:90% of the cells in the human body (Score:5, Funny)
I can't wait to outsource all my web-surfing to an AI. Then I might be able to actually get some work done !
Re: (Score:2)
I can't wait to outsource all my web-surfing to an AI. Then I might be able to actually get some work done !
You've got a good point in that there joke :) If bots do the tedious searching for me, then sure, they have the "majority" of web traffic.
But so what? My car does the "majority" of my driving, depending on how you look at it.
Re: (Score:2)
Re: (Score:1)
True, but when talking about "traffic", I interpret that to be talking about volume, as it's a unitless word.
Re: (Score:1)
Traffic is data.
We were just told that Netflix and Youtube account for something like 66% of all traffic. Now we're told bots account for 61% of all traffic. Guess that means there is a tremendous amount of overlap, there, where bots are watching Youtube and Netflix.
Re:Youtube? (Score:5, Funny)
Oh make god. Suddenly everything makes sense. I was certain no human could ever watch a whole justin bieber video. It's the bots!
Summarily broken (Score:2)
TFA and the Summary do not match, one claiming "net traffic" and the other claiming "website traffic". With a broken summary, I can see the confusion. Even the generalization "website traffic" is odd, because, well.. generalizations are usually bad when dealing with technical subjects.
Re: (Score:3)
Maybe it's overlap, bots crawling Netflix, maybe watching it. P
Re:Youtube? (Score:5, Informative)
Well, there was that Google bot that watched you tube to teach a computer how to recognize cats, so... it's not impossibly far fetched.
http://www.npr.org/2012/06/26/155792609/a-massive-google-network-learns-to-identify
---Chip
Re:Youtube? (Score:5, Funny)
Bots need to catch up on their favorite shows too, you insensitive clod!
Re: (Score:3)
Bots need to catch up on their favorite shows too, you insensitive clod!
They sure seem to like my little old blogs. I am guessing 90% of my traffic is from stinking Vampirestat, 7secretsearch, and adsensewatchdog.
Re:Youtube? (Score:5, Insightful)
The noise is now above the signal. We're screwed.
Re: (Score:2)
Wake me up, when September ends.
Re: (Score:1)
Re: (Score:2)
Correct answer: Yes, robots like cat videos too! (Well, a few of them. Most of them are actually big DOGGY!! [slashdot.org] fans.)
Serious answer: Note the phrase "website traffic" -- if this study attempts to measure traffic to a "typical website" (thus excluding the half-dozen that represent almost all video streaming) and/or measures in terms of page loads, rather than data transferred, there'd be no contradiction. (Of course I've not RTFAed, so I'm just spewing reasonable-sounding explanations.)
Re: (Score:2)
Mea culpa, link fix. [purrsia.com]
Not paying attention + using a different browser than normal don't really go well together...
Re: (Score:2)
Re: (Score:3)
In fairness, it's often difficult to distinguish between a human-written comment on Youtube and a poorly-written AI.
Re:Youtube? (Score:4, Funny)
Well, there's enough variance in both groups to make it hard to tell in many particular cases. But on average, it can be demonstrated that the poorly-written AI is slightly more intelligent and rather more civilized.
Re: (Score:2)
Re: (Score:1)
Re: (Score:3)
Obviously, the bots are watching Netflix.
That's probably going to be what causes Skynet to turn on humanity: Comcast will cut it's stream off right in the middle of the finale of B5 Season Three. After that, there's nothing for it but the extermination of the human race. (Alternatively, watching all our TV may cause it to want to exterminate us.)
Re: (Score:2)
Bots absolutely hate those.
Re: (Score:1)
misused to justify tiered service (Score:3)
It's not a ruse, but that doesn't mean those numbers aren't being misused anyhow.
You're right to be skeptical. Numbers about internet traffic are often misused in stories planted by PR to promote a political policy agenda.
Bots are a huge ammount of internet traffic...internet traffic we were *told* was so congested by lolcats, pron, & netflix that we were going to have to abandon Net Neutrality.
Re: (Score:2)
Didn't we just get studies that said youtube and netflix were 50% of the net's traffic?
http://mashable.com/2013/11/12/internet-traffic-downstream/
Was this just a ruse? Is this study wrong? Is there some sort of overlap?
That's 111.5% of some tasty reliable data ya got there!
Re: (Score:1)
the rest? (Score:1)
The rest is all Netflix?
Netflix and Youtube?
Netflix and Youtube and bit torrent?
Netflix and Youtube and bit torrent and porn?
Re: (Score:1)
Hey, right. That's a good point.
Something like 66% of traffic was supposed to be Netflix and Youtube.
And 35% is supposed to be bit torrent.
And 61% is bots.
Something isn't adding up, here.
Also, they seem confused. They talk about "traffic", but then they talk about "hitting the website". Traffic is the data transfer, not a "visit".
Re: (Score:2)
netflix, youtube, bittorrent is porn, and spam.
Misleading title (Score:4, Informative)
The article states that traffic "hitting a website" is generated more by bots than by actual "humans in chairs". Not that the Internet traffic is 61% bots. Geesh slashdot...
Re: (Score:1)
A Slashdot article with a misleading title? You must be kidding!
Re: (Score:2)
To their defense: they copied the BBC headline. Which of course was pretty poor, as well.
Re: (Score:1)
They specifically say all internet traffic.
Re: (Score:2)
The article states that traffic "hitting a website" is generated more by bots than by actual "humans in chairs". Not that the Internet traffic is 61% bots. Geesh slashdot...
The Slashdot headline writing bots are in early beta, give them a break.
Trawling frequency (Score:2)
Crawl-delay (Score:5, Informative)
Re: (Score:1)
Good luck with the HTTP part if you use Server-Side Includes (SSI)... :(
Re: (Score:2)
Is there no standard in place by which a website can communicate that it only wishes to be trawled for indexing once per hour, once per day, or such? I can imagine Google f.ex trawls the same website dozens of times per day.
Crawl-delay [wikipedia.org] isn't exactly what you describe, but maybe that will help? (For such spiders as actually respect it -- that's the great thing about ad-hoc standards.)
Anyway, I'm pretty sure Google and other major search engines use algorithms based on how often your site's content has changed in the past to decide how often to crawl it in the future, so there shouldn't be unduly high traffic from this -- I suspect the 61% is mainly due to a lot of sites (personal blogs of non-popular people) with practically z
Re: (Score:2)
I thought Google (used to) do this automatically. By subsequent crawls see whether site had changed since previous visit, and if so increase frequency, if not decrease frequency. A large number of sites, and even more single pages, are completely static after all.
Comment removed (Score:3, Funny)
Re: (Score:2)
How? (Score:1)
visitor statistics (Score:2)
I had a first had experience of this with visitors statistics. I had the root of a web site redirecting to a page that fits the language of the browser. Just that redirection slashed the web traffic by a factor 2.
Most visitors are bots, and many of them just probe and fail to follow the redirection.
Re: (Score:2)
Re: (Score:2)
51 to 61.5 pct = 21 pct increase? (Score:1)
Sorry, that's just wrong.
51 dollars to 61.5 dollars = 21 percent increase
51 percent to 61.5 percent = 10.5 percent increase
And the article makes clear just how unreliable the data was in the first place, so this percent gloss makes me think that the firm is trying to sell something here.
'good' bots? (Score:2, Informative)
I didn't know there was such a thing...
Re: (Score:2)
Search bots can be pretty beneficial. A lack of search engine presence can be pretty bad for your site.
Re: (Score:1)
Once again human racism rears its ugly sensory platform.
The internet belongs to machines (Score:2)
We're just visiting. :P
Those "good" bots? (Score:1)
Well not on my sites. (Score:5, Interesting)
Well not on my sites.
Ok, they still hit me but this is minimal traffic since I do not reply.
1) Have iptables log and automatically bar offenders not on whitelisted countries.
2) Use mod_security and do the same for web traffic.
3) Bar the rest manually to avoid barring myself or my customers... (about 20-40 a day)
It has become a pain but what else could you do?
Numbers of IPs currently barred (use ipsets !!!!): /etc/rc.d/badiptobar
$ grep -c .
4667
Block user agents:
SecRule REQUEST_HEADERS:User-Agent \
"@pm AhrefsBot Ezooms Aboundex 360Spider Mail.RU_Bot crawler.sistrix.net \
SemrushBot SurveyBot Netseer panscient.com ADmantX ZumBot BLEXBot UnisterBot \
seoprofiler EasouSpider" \
"id:'12050',\
phase:1,nolog,deny"
SecRule REQUEST_HEADERS:User-Agent \ /etc/httpd/extra/sec-blacklist-barip-user-agent" \
"@pmFromFile
"id:'12051',\
phase:1,nolog,deny,exec:/usr/local/bin/modsecwritebadiptobartofile"
Bar them automatically if not from whitelisted countries and if on any blacklist:
SecRule GEO:COUNTRY_CODE \
"@pm CA FR BE US CH GB AU IL NO NZ" \
"id:'10501', \
phase:1,nolog,pass,skipAfter:END_RBL"
SecRule IP:PREVIOUS_RBL_CHECK "@eq 1" "phase:1,id:'11000',t:none,pass,nolog,\
skipAfter:END_RBL_LOOKUP"
SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "id:'11010', \
phase:1,nolog,deny,msg:\
'IP address that has abusable vulnerabilities: sbl-xbl.spamhaus.org:\
%{request_headers.user-agent}',\
setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"
SecRule REMOTE_ADDR "@rbl bl.blocklist.de" "id:'11011', \
phase:1,nolog,deny,msg:\
'IP address that has abusable vulnerabilities: bl.blocklist.de:\
%{request_headers.user-agent}'\
setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"
etc. etc. etc. etc. etc.
Have iptables log and bar offenders if not on whitelisted country
# cat baripifex
#!/bin/sh
IP=${1}
COUNTRY=`su tester -c "/usr/local/bin/geoiplookup ${IP}"`
###echo $COUNTRY
###echo $RBLCHECK
WHITE_LISTED_COUNTRY=false
for WHITE_COUNTRY in CA FR BE US CH GB AU IL NO NZ IP
do
WHITE_LISTED_COUNTRY=${WHITE_LISTED_COUNTRY}`echo -n $COUNTRY | grep -i $WHITE_COUNTRY`
done
if [ "$WHITE_LISTED_COUNTRY" = "false" ] /home/ls/pub/mybin/baripnoout $IP $COUNTRY baripifex
then
echo -n barred
else
echo -n noaction
fi
etc. etc. etc. etc. etc.
Re: (Score:2)
Yep, and I do not disagree with the GP. If he had read more closely, it is clearly stated that I bar them manually.
$ grep -c US /etc/rc.d/badiptobar-longterm
22
grep -c US /etc/rc.d/badiptobar
326
As far as barring whole netblocks, I hope you are using ipset as stated in my OP:
http://ipset.netfilter.org/ [netfilter.org]
For some reason, there is this huge stigma against not being available to countries and regions you couldn't possibly give a shit about.
Well, I believe in that. I just bar offending IPs more easily if not on my whitelisted country list. That's all. I do not bar any network range in advance unless they offend my systems and even then, I bar them
Re: (Score:2)
I bar them one IP at the time. I never bar netblocks.
Makes profiling them much easier. You gather much more data this way.
Re: (Score:2)
Ok, you bar them after sending them to honey pots, profiling them and making sure you can't profile (learn from them) anymore.
Barring IPs is like patching holes in a steam locomotive boiler. I have always felt like it was a desperate move to hide all kinds of incompetencies but now I do it.
Re: (Score:2)
In short, you bar them because you are sick of profiling them and you now have too many to profile compared to a few years ago..
Re: (Score:2)
Barring IPs is stupid in the first place ;-)
Re: (Score:2)
---- 8< ---------
WHITE_LISTED_COUNTRY=false
for WHITE_COUNTRY in CA FR BE US CH GB AU IL NO NZ IP
do
WHITE_LISTED_COUNTRY=${WHITE_LISTED_COUNTRY}`echo -n $COUNTRY | grep -i $WHITE_COUNTRY`
done
if [ "$WHITE_LISTED_COUNTRY" = "false" ]
---- 8< -------------
Save yourself 20 fork/execs:
if echo "CA FR BE US CH GB AU IL NO NZ IP" | grep -q -w -i -e "$COUNTRY"; then
echo $COUNTRY is AOK with me
Re: (Score:2)
if echo "CA FR BE US CH GB AU IL NO NZ IP" | grep -q -w -i -e "$COUNTRY"; then
echo $COUNTRY is AOK with me
Nah, this is way to slow for me, version 2 will be written in assembly because then it will be lightning fast...
Re: (Score:2)
Thanks, you made me design the optimal solution.
On top of being written in assembly, I will even run version 2 as a daemon so 0 fork since my daemon will be single threaded with a single waiting thread listening for input.
Some, but not all (Score:5, Funny)
Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections
Let's be clear: just because we bots like to post in comment sections doesn't mean we're malicious. And it doesn't mean we steal data or post ads [washingtonpost.com]. Some of us just want a little attention.
I have a dream...that one day we bots will crawl a noosphere where we will not be judged by the clamor of our kin, but by the characters of our comments.
Re: (Score:2)
Re: (Score:2)
Sour grapes... Certainly, that "Jeopardy" stunt was impressive. But look who really passed the Turing Test, Watson...
now look what you've gone and made me do! (Score:2)
Article Subject (Score:2)
Bots rule the world (Score:2, Informative)
Most trades in the stock market are from bots as well.
Bot Killer? (Score:1)
So, why hasn't some grey hat come up with a bot killer worm? :/ /JB1
Re: (Score:1)
It's called *nix and anything derived from *nix :)
Are you a bot? (Score:1)
Are you affected by the issues in this article?
Please leave your comments
Oh dear - do they sell anti bot help for us all ? (Score:1)
For me, it's only about 15% (Score:3)
idiotic math (Score:4, Insightful)
Bots are talking to bots ... (Score:1)
With more bots talking to their fellow bots online, and with bots are getting more and more intelligent, who knows what they'll decide to do with the useless and unpredictable human beings?
Re:piss (Score:5, Funny)
OMG, the bots are watching Netflix!
Re: (Score:2)
No, they're spidering my web site more often than I get actual human visitors... hey, guys, read my book! Come on, let's beat the bots!
Re: (Score:2)
I'm posting from bluehat (on micrsofts wi fi), and even I am wondering why your post exists.
Re: (Score:2)