So, in effect, robots and spiders and...bots, oh my, come in and shake signs at us, going, "Ooga Booga!"
And then we hold up a sign that says, "Sorry spider-bot-things, we're blocking you."
This is how it works?
Niska: Mr. Reynolds? You died, Mr. Reynolds. Mal: Seemed like the thing to do.
'War Stories'
Buffistas Building a Better Board
Do you have problems, concerns or recommendations about the technical side of the Phoenix? Air them here. Compliments also welcome.
Beverly - Sep 22, 2003 6:55:23 pm PDT #5068 of 10000
Days shrink and grow cold, sunlight through leaves is my song. Winter is long.
Typo Boy - Sep 22, 2003 7:00:48 pm PDT #5069 of 10000
Calli: My people have a saying. A man who trusts can never be betrayed, only mistaken.Avon: Life expectancy among your people must be extremely short.
Umm - yeah. But google and the legit ones will. It is like a no-trepassing sign. Most people won't enter the posted property. But a sign, by itself, keeps out nobody who chooses to ignore it.
Kristen - Sep 22, 2003 9:08:08 pm PDT #5070 of 10000
So we could theoretically be getting spidered by spiders of ill repute. Which is amusing but probably only inside my head.
Speaking of inside my head, I'm getting another tooth yanked tomorrow so I will be out of touch most of the day. Hopefully, the board will still be here when I get back.
Typo Boy - Sep 22, 2003 9:19:10 pm PDT #5071 of 10000
Calli: My people have a saying. A man who trusts can never be betrayed, only mistaken.Avon: Life expectancy among your people must be extremely short.
Not entirely funny. The most common spiders and bots of ill repute are sent by spammers harvesting e-mail addresses.
Matt the Bruins fan - Sep 22, 2003 11:33:38 pm PDT #5072 of 10000
"I remember when they eventually introduced that drug kingpin who murdered people and smuggled drugs inside snakes and I was like 'Finally. A normal person.'” —RahvinDragand
I've noticed an oddity tonight - when any given page finishes loading, it then goes blank and reloads a second time. Is that good, bad, or just annoying?
DXMachina - Sep 23, 2003 2:43:39 am PDT #5073 of 10000
You always do this. We get tipsy, and you take advantage of my love of the scientific method.
So we could theoretically be getting spidered by spiders of ill repute. Which is amusing but probably only inside my head.
Are spiders of ill repute anything like robots of unusual size?
I've noticed an oddity tonight - when any given page finishes loading, it then goes blank and reloads a second time. Is that good, bad, or just annoying?
I'd go with unique. I haven't seen anything like that happening. Anyone else?
Leigh - Sep 23, 2003 3:01:58 am PDT #5074 of 10000
Nobody
Not me, and I've been reading sporadically over the past couple of hours.
§ ita § - Sep 23, 2003 4:02:28 am PDT #5075 of 10000
Well not canonically, no, but this is transformative fiction.
The most common spiders and bots of ill repute are sent by spammers harvesting e-mail addresses.
All they can get here are e-mail addresses in posts, and donations and admins.
Typo Boy - Sep 23, 2003 7:39:28 am PDT #5076 of 10000
Calli: My people have a saying. A man who trusts can never be betrayed, only mistaken.Avon: Life expectancy among your people must be extremely short.
Don't they generally follow all links? That would mean they do the equivalent of clicking on the user which does the query that gives your e-mail address if you made it public? Oh but that is not visible if you login as guest.
Hmmm - still they could be searching even if they did not find anything. Anyway to tell if we were spidered/botted?
§ ita § - Sep 23, 2003 8:00:36 am PDT #5077 of 10000
Well not canonically, no, but this is transformative fiction.
If we make two (possibly incredibly invalid) assumptions, there was no undue activity Sunday night. Four robots stopped by and asked for robots.txt -- inktomi grabbed some filk, and the rest went straight home. There are no suspicious client agents that are taking up a great deal of traffic.
In fact, there wasn't a great deal of traffic, period.
Now, there's no reason a bogus spider would a) ask for robots.txt or b) not pretend to be MSIE6.0.
I just don't see anything odd. But I'm searching for a more granular analysis tool. Webtrends has gone all weird on me.