See other Science/Tech Articles
Title: Internet geeks here! Who can determine how many web spiders/crawlers are on 4um??
Source:
none
URL Source: http://none
Published: Nov 20, 2009
Author: X-15
Post Date: 2009-11-20 03:04:08 by X-15
Keywords: None
Views: 2815
Comments: 170
"Web spiders/crawlers: programs that search websites looking for specific words or patterns to compile into a database." A popular gun website I visit had 20 running, if 4um has less then I assume it has a lower profile in the eyes of FedGov.
Post Comment Private Reply Ignore Thread
Top • Page Up • Full Thread • Page Down • Bottom/Latest
#1. To: X-15, Pinguinite, christine (#0)
You'd need access to christine's server logs to get a good idea. However, there are many kinds of spiders, some quite difficult to detect. Here is a quickie spider that I wrote. It runs on the Mac, OS X 10.5 Leopard. However, it is a standard Bash script and should work easily on Linux or Unix systems, probably in a Cygwin setup on Windows too. The script uses Lynx, a venerable text-only browser, to fetch my Comments page to a file called htmlsource1. It then uses the stream editor Sed to parse this captured HTML file by scanning the right column for news stories, capturing the thread names and URLs at 4um to a file called htmlsource2. It then uses Lynx to capture each thread to a separate file by thread number in a subdirectory called '4um'. You could build a database or use text search tools like grep to mine the stored threads for info. No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings. I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country. # # lynx directory LYNX=/usr/bin/lynx # IP host HOST='freedom4um.com' DIRNAME='4um' if [ ! -d "$DIRNAME" ]; then $LYNX -source "http://$HOST/cgi-bin/latestcomments.cgi?SNSearch=1&EM=on&Fm=&To=TooConservative" > lynxhtmlsource1 sed -n "/ /,/ FETCHCOUNT=0 rm -f lynxhtmlsource1 lynxhtmlsource2 if [ $FETCHCOUNT -ne 0 ]; then exit
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country. Thank you for the reply and info!
_________________________________________________________________________ thank you, TC.
BTW, by changing only a few lines in the above code, I could slowly download your entire database and reconstruct each poster's remarks. Essentially, your mySQL database could be replicated by downloading all the threads and parsing the user comments into a new mySQL database. I'm sure Neil could point this out as well, probably better than I can. This is why watching your server logs for an IP that downloads every thread or an IP that downloads every thread in the database sequentially is good to do. Anyway, this seemed a good thread to point this stuff out.
can that be done on any forum?
Yes. You only have to parse for the HTML tags and CSS classes. Not at all difficult. I once wrote a Firefox extension that allowed me to entirely replace the look and feel of TOS, add backgrounds, insert YouTubes to replace the YouTube links, implement my own browser-based bozo filter, etc. It's quite easy. You have to have good server log analytic software to find out if your site is being mined. Now, 4um isn't really high traffic so you can probably get a good idea by looking at IP addresses. You should watch for IP addresses that only read threads (and that read every thread) and never post. Those lurkers can just as easily be spiders for ADL, FBI, SPLC, NetNanny, Google, etc. In fact, you should assume that you are being spidered this way until you can prove otherwise. And the spidering of your threads could just as easily come from multiple IP addresses. You can deter some of this by requiring the use of cookies but a competent programmer can fake that too. You should assume that every word you put online will be recorded. The Feds are building huge new datacenters to store the content of the entire internet and all cell phones and landlines. They may make no use of that unless and until they detect (or need) to chase a domestic threat for terrorism or hate crimes. It is at that point that you will receive a subpoena for your server logs to obtain IP addresses (if they don't already have them by being on the backbone and sniffing everything) and an included order that you will not tell anyone that they are assembling evidence. After that, the ISPs for those requested IP addresses will get national security directives and subpoenas to provide their logs to identify the IP address and they would also be silenced. This is is the new Soviet Amerika. Welcome to the gulag, comrade.
I was wondering how a certain poster appeared on a thread so quickly where the word heebs was used. I suspect a spider looking for certain keywords.
Obama's health care plan shall be written by a committee whose head says he doesn't understand it, passed by a Congress that hasn't read it, signed by a president who smokes and has no birth certificate, funded by a treasury chief who did not pay his taxes, overseen by a surgeon general who is overweight and financed by a country that is nearly broke. What could possibly go wrong? - buckeroo Not only did the PC Avenger appear, but so did another defender of Obama and big government. The Rats are spiders.
Pray for Obama: Psalm 1984
In addition, you can google a deleted thread and download it from "cache". It will live forever once it has been posted.
_________________________________________________________________________ It all happened way too quick to be coinkydink, no?
Obama's health care plan shall be written by a committee whose head says he doesn't understand it, passed by a Congress that hasn't read it, signed by a president who smokes and has no birth certificate, funded by a treasury chief who did not pay his taxes, overseen by a surgeon general who is overweight and financed by a country that is nearly broke. What could possibly go wrong? - buckeroo It was a partial infestation of leftists, which is now under control. After the last set of Marxists were booted, I no longer believe in coinkydinks when dealing with them :P
Pray for Obama: Psalm 1984
Who dat?
AKA Stone. I think he's in love with you:
_________________________________________________________________________
_________________________________________________________________________ _________________________________________________________________________ war
Pray for Obama: Psalm 1984
Oh nooooooooooooooooo! Another reason to hide under my bed and think up more scapegoats!
Turning and turning in the widening gyre The falcon cannot hear the falconer; Things fall apart; the center cannot hold; Mere anarchy is loosed upon the world, The blood-dimmed tide is loosed, and everywhere The ceremony of innocence is drowned; The best lack all conviction, while the worst Are full of passionate intensity. .... Yeats
Thanks, but no thanks on the link. I don't go to those kind of websites, but I can imagine. ZioNutterBuddy needs to learn when someone is pulling his leg. When he goes on his tirades, it does more to further the cause of anti-semitism than Adolph Hitler or Mahmoud Iran-Dude could ever hope to achieve. If he's not careful, Abe Foxman is gonna have a price on ZioNutterBuddy's head one of these days. Sometimes I wonder if he's not a clever StormFronter playing a rabid ZioNutter and going on about kill, kill, killing the goys to wake people up about the jews. Kook or Klever, too klose to kall.
Well, there is always the possibility of coincidence. But then again... You shouldn't post anything assuming that it is somehow anonymous. That includes email and all internet activity. And don't assume your internal network is absolutely secure either. I don't think anything has been truly anonymous for at least the last five years, maybe before. And certainly, the entire future will be recorded and used against you in a court of law. Words to the wise.
What do you think of this?
Packrat posted a link to to an LP post. When I clicked on it, it led me to this website. I didn't realize it wasn't libertypost.org until after I had to sign in with my password in order to reply back to him.
not found???
That's odd, it was there when I posted it. It said the site had changed hands 3 times in the last year and had been registered since 2008.
Try it from here?
Here's what we know about liberty-post.net: View 6 ownership records archived since 2008-04-07 . libertypost.net was around before 2008 if I remember right.
No doubt whatsoever.
What do you think of this? I think that libertypost.org is not related to those various liberty-post domains. Why, what do you think of it?
Just for fun, I ran my script again for the first time in months. Since I didn't have any recent threads in my cache of 4um threads, it processed the entire news article sidebar. You can see the time stamp so you can check your logs and find this access. Then I ran the script again a few minutes later. Since no new articles had been posted, it found nothing new to cache. As I said, this crappy little Bash script isn't really even a spider, just a webserver capture script. If you wrote in a higher level language and sniff headers and such, it would be quite easy to become extremely sophisticated about this. I could churn out a real spider in very short order as could Neil or tons of script kiddies. As for the posters here at 4um, one should assume that FBI and other agencies may employ agentes provocateur in the classic style they used against the Klan and others. Assuming that you had, say, a half-dozen posters here that post the most vitriolic content on the site, they could greatly raise the profile of 4um with SPLC/ADL/FBI quite easily. After all, what would they do if they couldn't point to the dire threat of rampant political incorrectness online? And how else could they assemble the "evidence" that there are vicious racists out there which justifies the usual begging letters sent out to donors by SPLC/ADL or the begging to Congress for staff and vast new computer systems by FBI/NSA/etc.? Hey, if that well isn't producing enough, you just have to prime the pump a little, baby! So even if you're running a legit free speech forum (and I have no reason to believe anything else though I consider the possibility), that doesn't mean that your forum might not be used as a honeypot for the race hustlers like ADL/SPLC or by various letters of the alphabet like F, B, or I. As I've demonstrated, it's quite easy to do.
website downloader
Apparently, I didn't escape the double-quotes properly so that is why my script didn't download all the threads. Neil would not make such an elementary error. My fault. :) Now, I have to decide if fixing it is worth the time. I really don't like Sed. And doing this stuff from a Bash script really is tedious.
Ping to #22.
No biggie, but please allow a bit more time for it, say one thread every 2 seconds. Going too fast, at least at a busy time, could cause the server to bog down. Mainstream search spiders from google and such hit sites much less often.
Oh, I make that one often enough!
I always assume that to be the case. If some invites, directly or indirectly, comments which could be incriminating, incitement to violence, or libelous I assume, provisionally pending further data, that I am dealing with a provocateur or shill of some kind. I also assume the government and their filthy vermin traitorous minions know exactly who I am and where I live. So, they can go fuck themselves (pardon the French). Since I do not make personal threats against even politicians and figures whom I despise as long as they have to maintain a pretense that they are obeying the Constitution I'm safe. Once they try to dispense with it I head for the hills. ;-)
"An education isn't how much you have committed to memory, or even how much you know. It's being able to differentiate between what you know and what you don't. ~ Anatole France
It's rather amazing that in every recent case where someone is caught plotting "terrorism", the accused was found trying to buy weapons or bombs of whatever sort from an FBI agent. Seems the modern terrorists never have their own sources of such things, and they instead have to rely on sources given them by these FBI provocateurs. I don't recall any so-called terrorist plots being busted that didn't involve an FBI agent being proactively involved in the mix somehow.
That's why they call it Home Grown Terrorism.
Comments (41 - 170) not displayed.
Top • Page Up • Full Thread • Page Down • Bottom/LatestInternet geeks here! Who can determine how many web spiders/crawlers are on 4um??
[Apologies. Neil's 4um code won't let me post this spider code accurately. It's screwing my Sed command and turns the text red.]
#!/bin/sh
mkdir "$DIRNAME"
fi
/ s//http://$HOST1/p" lynxhtmlsource1 > lynxhtmlsource2
for URL in $(cat lynxhtmlsource2); do
ARTICLE="${URL:49:6}"
# echo "#$ARTICLE URL: $URL"
if [ ! -f "$DIRNAME/$ARTICLE" ]; then
$LYNX -source "$URL" > "4um/$ARTICLE"
echo "fetched $ARTICLE..."
let FETCHCOUNT+=1
fi
done
echo "$HOST: $FETCHCOUNT fetched"
else
echo "no new articles on $HOST"
fi
#2. To: TooConservative (#1)
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#3. To: TooConservative (#1)
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
#4. To: christine (#3)
#5. To: TooConservative (#4)
I could slowly download your entire database and reconstruct each poster's remarks.
#6. To: christine (#5)
can that be done on any forum?
#7. To: christine, TooConservative (#5)
(Edited)
can that be done on any forum?
Let me get this straight.
#8. To: Critter (#7)
#9. To: TooConservative (#6)
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#10. To: Jethro Tull (#8)
Not only did the PC Avenger appear
Let me get this straight.
#11. To: Critter (#10)
#12. To: Jethro Tull (#8)
...the PC Avenger...
Mike Flaherty: Yeah, it's been there a long time. I wish I knew which corner.
My Man Godfrey (1936)
#13. To: Esso (#12)
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#14. To: X-15 (#0)
#15. To: TooConservative (#6)
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#16. To: TwentyTwelve (#14)
"This man is Jesus,” shouted one man, spilling his Guinness as Barack Obama began his inaugural address. “When will he come to Kenya to save us?”
-Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
#17. To: Esso (#12)
the PC Avenger
#18. To: X-15 (#15)
#19. To: X-15 (#0)
Who can determine how many web spiders/crawlers are on 4um??
#20. To: X-15 (#13)
(Edited)
Mike Flaherty: Yeah, it's been there a long time. I wish I knew which corner.
My Man Godfrey (1936)
#21. To: Critter (#7)
(Edited)
I was wondering how a certain poster appeared on a thread so quickly where the word heebs was used. I suspect a spider looking for certain keywords.
#22. To: TooConservative, christine, Pinguinite, all (#6)
(Edited)
whois.domaintools.com/liberty-post.net
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#23. To: All (#22)
(Edited)
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#24. To: TooConservative, All (#23)
libertypost.net
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#25. To: wudidiz (#22)
#26. To: christine (#25)
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#27. To: christine (#25)
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#28. To: christine, All (#27)
Wiki article on Liberty-post.net
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#29. To: All (#28)
1 registrar has maintained records for this domain since 2008-04-04
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#30. To: TooConservative (#1)
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
#31. To: wudidiz (#22)
whois.domaintools.com/liberty-post.net
#32. To: Pinguinite, christine, X-15 (#30)
It only took 30 seconds to capture 68 full threads to my hard drive. I note that about ten threads are missing, did you remove some perhaps? Or maybe there's some little bug in my script; it wasn't like this code was highly important to me so I didn't go nuts over it. Sat Nov 21 05:29:23 CST 2009
fetched 108669... (this one is your sticky thread)
fetched 110723...
fetched 110722...
fetched 110721...
fetched 110720...
fetched 110719...
fetched 110718...
fetched 110717...
...
fetched 110650...
fetched 110649...
freedom4um.com: 68 fetched
If I used a cron job to schedule this script to run regularly, say every hour or even every 12 hours, it would capture every thread posted at 4um. You'd have to revisit the thread, perhaps sniffing the headers to detect whether the thread had changed, in order to capture all the comments because those can come in days, months or even years later. You'd capture 99% of these chat forums' content if you just wait one month to capture a thread, parse it, and store it.
Sat Nov 21 05:34:29 CST 2009
no new articles on freedom4um.com
#33. To: TooConservative (#32)
#34. To: christine, Pinguinite (#32)
It only took 30 seconds to capture 68 full threads to my hard drive. I note that about ten threads are missing, did you remove some perhaps?
#35. To: TooConservative (#23)
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
#36. To: TooConservative (#32)
It only took 30 seconds to capture 68 full threads to my hard drive.
#37. To: TooConservative (#34)
Neil would not make such an elementary error.
#38. To: TooConservative, Pinguinite, christine, X-15 (#32)
As for the posters here at 4um, one should assume that FBI and other agencies may employ agentes provocateur in the classic style they used against the Klan and others.
#39. To: Original_Intent (#38)
If some invites, directly or indirectly, comments which could be incriminating, incitement to violence, or libelous I assume, provisionally pending further data, that I am dealing with a provocateur or shill of some kind.
#40. To: Pinguinite. all (#39)
I don't recall any so-called terrorist plots being busted that didn't involve an FBI agent being proactively involved in the mix somehow.
.
.
.