Title: Internet geeks here! Who can determine how many web spiders/crawlers are on 4um?? Source:
none URL Source:http://none Published:Nov 20, 2009 Author:X-15 Post Date:2009-11-20 03:04:08 by X-15 Keywords:None Views:2525 Comments:170
"Web spiders/crawlers: programs that search websites looking for specific words or patterns to compile into a database."
A popular gun website I visit had 20 running, if 4um has less then I assume it has a lower profile in the eyes of FedGov.
Internet geeks here! Who can determine how many web spiders/crawlers are on 4um??
You'd need access to christine's server logs to get a good idea. However, there are many kinds of spiders, some quite difficult to detect.
Here is a quickie spider that I wrote. It runs on the Mac, OS X 10.5 Leopard. However, it is a standard Bash script and should work easily on Linux or Unix systems, probably in a Cygwin setup on Windows too.
The script uses Lynx, a venerable text-only browser, to fetch my Comments page to a file called htmlsource1. It then uses the stream editor Sed to parse this captured HTML file by scanning the right column for news stories, capturing the thread names and URLs at 4um to a file called htmlsource2.
It then uses Lynx to capture each thread to a separate file by thread number in a subdirectory called '4um'.
You could build a database or use text search tools like grep to mine the stored threads for info.
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
Thank you for the reply and info!
_________________________________________________________________________ "This man is Jesus, shouted one man, spilling his Guinness as Barack Obama began his inaugural address. When will he come to Kenya to save us?
The best and first guarantor of our neutrality and our independent existence is the defensive will of the people and the proverbial marksmanship of the Swiss shooter. Each soldier a good marksman! Each shot a hit! -Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
BTW, by changing only a few lines in the above code, I could slowly download your entire database and reconstruct each poster's remarks. Essentially, your mySQL database could be replicated by downloading all the threads and parsing the user comments into a new mySQL database. I'm sure Neil could point this out as well, probably better than I can. This is why watching your server logs for an IP that downloads every thread or an IP that downloads every thread in the database sequentially is good to do.
Anyway, this seemed a good thread to point this stuff out.
You only have to parse for the HTML tags and CSS classes. Not at all difficult.
I once wrote a Firefox extension that allowed me to entirely replace the look and feel of TOS, add backgrounds, insert YouTubes to replace the YouTube links, implement my own browser-based bozo filter, etc.
It's quite easy. You have to have good server log analytic software to find out if your site is being mined. Now, 4um isn't really high traffic so you can probably get a good idea by looking at IP addresses. You should watch for IP addresses that only read threads (and that read every thread) and never post. Those lurkers can just as easily be spiders for ADL, FBI, SPLC, NetNanny, Google, etc. In fact, you should assume that you are being spidered this way until you can prove otherwise.
And the spidering of your threads could just as easily come from multiple IP addresses. You can deter some of this by requiring the use of cookies but a competent programmer can fake that too.
You should assume that every word you put online will be recorded. The Feds are building huge new datacenters to store the content of the entire internet and all cell phones and landlines. They may make no use of that unless and until they detect (or need) to chase a domestic threat for terrorism or hate crimes. It is at that point that you will receive a subpoena for your server logs to obtain IP addresses (if they don't already have them by being on the backbone and sniffing everything) and an included order that you will not tell anyone that they are assembling evidence. After that, the ISPs for those requested IP addresses will get national security directives and subpoenas to provide their logs to identify the IP address and they would also be silenced.
This is is the new Soviet Amerika. Welcome to the gulag, comrade.
I was wondering how a certain poster appeared on a thread so quickly where the word heebs was used. I suspect a spider looking for certain keywords.
Let me get this straight.
Obama's health care plan shall be written by a committee whose head says he doesn't understand it, passed by a Congress that hasn't read it, signed by a president who smokes and has no birth certificate, funded by a treasury chief who did not pay his taxes, overseen by a surgeon general who is overweight and financed by a country that is nearly broke.
In addition, you can google a deleted thread and download it from "cache". It will live forever once it has been posted.
_________________________________________________________________________ "This man is Jesus, shouted one man, spilling his Guinness as Barack Obama began his inaugural address. When will he come to Kenya to save us?
The best and first guarantor of our neutrality and our independent existence is the defensive will of the people and the proverbial marksmanship of the Swiss shooter. Each soldier a good marksman! Each shot a hit! -Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
It all happened way too quick to be coinkydink, no?
Let me get this straight.
Obama's health care plan shall be written by a committee whose head says he doesn't understand it, passed by a Congress that hasn't read it, signed by a president who smokes and has no birth certificate, funded by a treasury chief who did not pay his taxes, overseen by a surgeon general who is overweight and financed by a country that is nearly broke.
It was a partial infestation of leftists, which is now under control. After the last set of Marxists were booted, I no longer believe in coinkydinks when dealing with them :P
Godfrey Smith: Mike, I wouldn't worry. Prosperity is just around the corner. Mike Flaherty: Yeah, it's been there a long time. I wish I knew which corner. My Man Godfrey (1936)
_________________________________________________________________________ "This man is Jesus, shouted one man, spilling his Guinness as Barack Obama began his inaugural address. When will he come to Kenya to save us?
The best and first guarantor of our neutrality and our independent existence is the defensive will of the people and the proverbial marksmanship of the Swiss shooter. Each soldier a good marksman! Each shot a hit! -Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
_________________________________________________________________________ "This man is Jesus, shouted one man, spilling his Guinness as Barack Obama began his inaugural address. When will he come to Kenya to save us?
The best and first guarantor of our neutrality and our independent existence is the defensive will of the people and the proverbial marksmanship of the Swiss shooter. Each soldier a good marksman! Each shot a hit! -Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
_________________________________________________________________________ "This man is Jesus, shouted one man, spilling his Guinness as Barack Obama began his inaugural address. When will he come to Kenya to save us?
The best and first guarantor of our neutrality and our independent existence is the defensive will of the people and the proverbial marksmanship of the Swiss shooter. Each soldier a good marksman! Each shot a hit! -Schweizerische Schuetzenzeitung (Swiss Shooting Federation) April, 1941
Who can determine how many web spiders/crawlers are on 4um??
Oh nooooooooooooooooo!
Another reason to hide under my bed and think up more scapegoats!
Turning and turning in the widening gyre The falcon cannot hear the falconer; Things fall apart; the center cannot hold; Mere anarchy is loosed upon the world, The blood-dimmed tide is loosed, and everywhere The ceremony of innocence is drowned; The best lack all conviction, while the worst Are full of passionate intensity. .... Yeats
Thanks, but no thanks on the link. I don't go to those kind of websites, but I can imagine.
ZioNutterBuddy needs to learn when someone is pulling his leg. When he goes on his tirades, it does more to further the cause of anti-semitism than Adolph Hitler or Mahmoud Iran-Dude could ever hope to achieve.
If he's not careful, Abe Foxman is gonna have a price on ZioNutterBuddy's head one of these days. Sometimes I wonder if he's not a clever StormFronter playing a rabid ZioNutter and going on about kill, kill, killing the goys to wake people up about the jews.
Kook or Klever, too klose to kall.
Godfrey Smith: Mike, I wouldn't worry. Prosperity is just around the corner. Mike Flaherty: Yeah, it's been there a long time. I wish I knew which corner. My Man Godfrey (1936)
I was wondering how a certain poster appeared on a thread so quickly where the word heebs was used. I suspect a spider looking for certain keywords.
Well, there is always the possibility of coincidence. But then again...
You shouldn't post anything assuming that it is somehow anonymous. That includes email and all internet activity. And don't assume your internal network is absolutely secure either. I don't think anything has been truly anonymous for at least the last five years, maybe before. And certainly, the entire future will be recorded and used against you in a court of law.
Packrat posted a link to to an LP post. When I clicked on it, it led me to this website. I didn't realize it wasn't libertypost.org until after I had to sign in with my password in order to reply back to him.
"The trouble with people is not that they don't know but that they know so much that ain't so." ~ Josh Billings
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
Just for fun, I ran my script again for the first time in months. Since I didn't have any recent threads in my cache of 4um threads, it processed the entire news article sidebar. You can see the time stamp so you can check your logs and find this access.
Sat Nov 21 05:29:23 CST 2009 fetched 108669... (this one is your sticky thread) fetched 110723... fetched 110722... fetched 110721... fetched 110720... fetched 110719... fetched 110718... fetched 110717...
It only took 30 seconds to capture 68 full threads to my hard drive. I note that about ten threads are missing, did you remove some perhaps? Or maybe there's some little bug in my script; it wasn't like this code was highly important to me so I didn't go nuts over it.
Then I ran the script again a few minutes later. Since no new articles had been posted, it found nothing new to cache.
Sat Nov 21 05:34:29 CST 2009 no new articles on freedom4um.com
If I used a cron job to schedule this script to run regularly, say every hour or even every 12 hours, it would capture every thread posted at 4um. You'd have to revisit the thread, perhaps sniffing the headers to detect whether the thread had changed, in order to capture all the comments because those can come in days, months or even years later. You'd capture 99% of these chat forums' content if you just wait one month to capture a thread, parse it, and store it.
As I said, this crappy little Bash script isn't really even a spider, just a webserver capture script. If you wrote in a higher level language and sniff headers and such, it would be quite easy to become extremely sophisticated about this. I could churn out a real spider in very short order as could Neil or tons of script kiddies.
As for the posters here at 4um, one should assume that FBI and other agencies may employ agentes provocateur in the classic style they used against the Klan and others. Assuming that you had, say, a half-dozen posters here that post the most vitriolic content on the site, they could greatly raise the profile of 4um with SPLC/ADL/FBI quite easily. After all, what would they do if they couldn't point to the dire threat of rampant political incorrectness online? And how else could they assemble the "evidence" that there are vicious racists out there which justifies the usual begging letters sent out to donors by SPLC/ADL or the begging to Congress for staff and vast new computer systems by FBI/NSA/etc.? Hey, if that well isn't producing enough, you just have to prime the pump a little, baby!
So even if you're running a legit free speech forum (and I have no reason to believe anything else though I consider the possibility), that doesn't mean that your forum might not be used as a honeypot for the race hustlers like ADL/SPLC or by various letters of the alphabet like F, B, or I.
It only took 30 seconds to capture 68 full threads to my hard drive. I note that about ten threads are missing, did you remove some perhaps?
Apparently, I didn't escape the double-quotes properly so that is why my script didn't download all the threads. Neil would not make such an elementary error.
My fault. :)
Now, I have to decide if fixing it is worth the time. I really don't like Sed. And doing this stuff from a Bash script really is tedious.
It only took 30 seconds to capture 68 full threads to my hard drive.
No biggie, but please allow a bit more time for it, say one thread every 2 seconds. Going too fast, at least at a busy time, could cause the server to bog down. Mainstream search spiders from google and such hit sites much less often.
#38. To: TooConservative, Pinguinite, christine, X-15 (#32)
As for the posters here at 4um, one should assume that FBI and other agencies may employ agentes provocateur in the classic style they used against the Klan and others.
I always assume that to be the case.
If some invites, directly or indirectly, comments which could be incriminating, incitement to violence, or libelous I assume, provisionally pending further data, that I am dealing with a provocateur or shill of some kind. I also assume the government and their filthy vermin traitorous minions know exactly who I am and where I live. So, they can go fuck themselves (pardon the French). Since I do not make personal threats against even politicians and figures whom I despise as long as they have to maintain a pretense that they are obeying the Constitution I'm safe. Once they try to dispense with it I head for the hills. ;-)
"An education isn't how much you have committed to memory, or even how much you know. It's being able to differentiate between what you know and what you don't. ~ Anatole France
If some invites, directly or indirectly, comments which could be incriminating, incitement to violence, or libelous I assume, provisionally pending further data, that I am dealing with a provocateur or shill of some kind.
It's rather amazing that in every recent case where someone is caught plotting "terrorism", the accused was found trying to buy weapons or bombs of whatever sort from an FBI agent. Seems the modern terrorists never have their own sources of such things, and they instead have to rely on sources given them by these FBI provocateurs.
I don't recall any so-called terrorist plots being busted that didn't involve an FBI agent being proactively involved in the mix somehow.