See other Science/Tech Articles
Title: Internet geeks here! Who can determine how many web spiders/crawlers are on 4um??
Source:
none
URL Source: http://none
Published: Nov 20, 2009
Author: X-15
Post Date: 2009-11-20 03:04:08 by X-15
Keywords: None
Views: 2760
Comments: 170
"Web spiders/crawlers: programs that search websites looking for specific words or patterns to compile into a database." A popular gun website I visit had 20 running, if 4um has less then I assume it has a lower profile in the eyes of FedGov.
Post Comment Private Reply Ignore Thread
Top • Page Up • Full Thread • Page Down • Bottom/Latest
Begin Trace Mode for Comment # 113.
#1. To: X-15, Pinguinite, christine (#0)
You'd need access to christine's server logs to get a good idea. However, there are many kinds of spiders, some quite difficult to detect. Here is a quickie spider that I wrote. It runs on the Mac, OS X 10.5 Leopard. However, it is a standard Bash script and should work easily on Linux or Unix systems, probably in a Cygwin setup on Windows too. The script uses Lynx, a venerable text-only browser, to fetch my Comments page to a file called htmlsource1. It then uses the stream editor Sed to parse this captured HTML file by scanning the right column for news stories, capturing the thread names and URLs at 4um to a file called htmlsource2. It then uses Lynx to capture each thread to a separate file by thread number in a subdirectory called '4um'. You could build a database or use text search tools like grep to mine the stored threads for info. No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings. I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country. # # lynx directory LYNX=/usr/bin/lynx # IP host HOST='freedom4um.com' DIRNAME='4um' if [ ! -d "$DIRNAME" ]; then $LYNX -source "http://$HOST/cgi-bin/latestcomments.cgi?SNSearch=1&EM=on&Fm=&To=TooConservative" > lynxhtmlsource1 sed -n "/ /,/ FETCHCOUNT=0 rm -f lynxhtmlsource1 lynxhtmlsource2 if [ $FETCHCOUNT -ne 0 ]; then exit
thank you, TC.
BTW, by changing only a few lines in the above code, I could slowly download your entire database and reconstruct each poster's remarks. Essentially, your mySQL database could be replicated by downloading all the threads and parsing the user comments into a new mySQL database. I'm sure Neil could point this out as well, probably better than I can. This is why watching your server logs for an IP that downloads every thread or an IP that downloads every thread in the database sequentially is good to do. Anyway, this seemed a good thread to point this stuff out.
can that be done on any forum?
Yes. You only have to parse for the HTML tags and CSS classes. Not at all difficult. I once wrote a Firefox extension that allowed me to entirely replace the look and feel of TOS, add backgrounds, insert YouTubes to replace the YouTube links, implement my own browser-based bozo filter, etc. It's quite easy. You have to have good server log analytic software to find out if your site is being mined. Now, 4um isn't really high traffic so you can probably get a good idea by looking at IP addresses. You should watch for IP addresses that only read threads (and that read every thread) and never post. Those lurkers can just as easily be spiders for ADL, FBI, SPLC, NetNanny, Google, etc. In fact, you should assume that you are being spidered this way until you can prove otherwise. And the spidering of your threads could just as easily come from multiple IP addresses. You can deter some of this by requiring the use of cookies but a competent programmer can fake that too. You should assume that every word you put online will be recorded. The Feds are building huge new datacenters to store the content of the entire internet and all cell phones and landlines. They may make no use of that unless and until they detect (or need) to chase a domestic threat for terrorism or hate crimes. It is at that point that you will receive a subpoena for your server logs to obtain IP addresses (if they don't already have them by being on the backbone and sniffing everything) and an included order that you will not tell anyone that they are assembling evidence. After that, the ISPs for those requested IP addresses will get national security directives and subpoenas to provide their logs to identify the IP address and they would also be silenced. This is is the new Soviet Amerika. Welcome to the gulag, comrade.
What do you think of this?
Noone wants to touch this stuff. Disconcerting.
What is your problem anyway? She owns both domain names and points both of them to her site. If you use libertypost.net as your URL, it mostly works but you won't have the same browser cookies (it seems) so it may look or act a little differently on a few screens. You might have to sign in again or whatnot. Libertypost.net points to the server for libertypost.org. Libertypost.com and libertypost.us are owned by the USPS and that isn't sinister either. Pointing multiple domains at a single server or server farm is a very common practice. This is how places like Amazon and Newegg work too.
Because noone would ever think of something so outrageous as duplicating a website. The idea is simply preposterous. For anyone to even think of such a thing or question it is clearly and indication of well, that they have a problem. Forgive me Maam for my ignorance or paranoia or stupidity or whatever my problem is, but isn't the fact that there are two different urls mean that there are two different websites? Regardless of how similar they look?
How snarky you are, wudi. Nevertheless, I'll answer you, noting the irony of you calling me rude. All website servers and all computers (and devices like addressable routers) occupy a dotted-IP address on the internet, just as your own home network has its own range of addresses. So your home network might use the common reserved range of addresses 192.168.0.1 - 192.168.255.255. In other words, a range of 65536 addresses (roughly) are reserved from any server or other LAN using them. So you might use an IP address of 192.168.0.1 for your DSL modem, the router you connect to it might use 192.168.1.1 and then all your home network computers could use addresses in the range 192.168.1.2 - 192.168.1.254. So a place like libertypost.org has a server with a dotted-IP address too. The dotted-IP address is what you access, the name is just a convenient way to access the actually address. You can point one domain name to a webserver. Or you can point a million names to a webserver. Or you can point different domain names to different subdirectories of the same server. Or whatever. Domain names don't care what you point them at. And the domain registrars are happy when you buy multiple names so they make more money. So, using common utilities, you can look up the actual dotted-IP address of LibertyPost. So you can see for yourself that the two names are the same. You can also just use the dotted-IP address since that is what you always end up using anyway. The name only translates to the number. So try clicking it: 69.55.11.240. Once again, you arrive at LibertyPost. Unless Goldi really is the supersecret hacker of all time. You can verify these domain names are at these numbers as I've told you by using your Command Prompt on Windows XP (or its equivalent on Vista or Win7). It's found in the Programs->Accessories folder. Once you open it, you can type nslookup to find your own primary, probably a DSL modem or router in your house. You can then enter libertypost.net or libertypost.org or whatever to check for yourself what the IP address really is. Type 'exit' to leave nslookup and then type 'exit' again to dismiss the command prompt and return to Windows. You can also google for 'nslookup' and use the online versions of these but you can't really trust those nslookup sites either because it might be Goldi running them all on her secret Evil Servers. Obviously, you can trust no one. It's really sad. Of course, Goldi's site, like 4um, runs from a single mySQL database using a bunch of Perl scripts and PHP pages, the most common kind of interactive site. So it is possible that she has two "copies" of her site, one with Good Decent Godfearing code, the other with the Supersecret Evil Goldi-Lox Killer code. So that if you go to libertypost.org, you get the Good Decent Godfearing version and if you go to libertypost.net, you get the Supersecrect Evil Goldi-Lox Killer version. Now, this whole notion of a good Goldi-server and an evil Goldi-server is somewhat daft. First, Goldi would have to be a lot more sophisticated than she is about servers and writing code. She doesn't do any of that stuff and hasn't even upgraded her older Pinguinite code (written by Neil and available for free) to the newer Pinguinite code that christine uses here which has more features, is more secure, etc. Beyond that, the idea of Goldi-two-servers is rather odd because the first principle of Being Evil is: Don't let anyone figure out that you really are evil. Now, I can go to any of the three known methods of reaching LibertyPost via libertypost.org or libertypost.net or 69.55.11.240 (among who knows how many other evil versions of LP there are), and the pages are the same, the most recent articles are the same, the most recent comments are the same. And again, I can sign in to any of them and they behave identically. Now, since everyone but you and a bunch of old cat ladies posting on Usenet or chatting on IRC already knows the above info, then what sense would it make to have the Goldi-two-servers thing? None whatsoever. What would be more interesting would be to have a single server with the two names pointed at it but monitor the access to it by the IP addresses of those you have banned from it. This way, you can be Evil to banned or disliked users but never look Evil to the Chosen People of LibertyPost (those who Goldi hasn't banned, this being 4um, I thought that was fun way to express it). And people on Goldi's ban list are always going to be assumed to be nursing a grudge anyway so no one will believe their accusations. So you might choose to be paranoid and think that Goldi is only running her Supersecret Evil Goldi-Code if you visit her server but she does show no signs of such code or admin sophistication and sneakiness. It really is just her chat board, I think. But you can believe or make up anything you like if you enjoy that more. But I see no reason to believe that Goldi or Christine are quite so powerful or obsessed over their chat boards as that would require. Now, me posting this much text to you, that might in and of itself seem Quite Evil to some folks despite my obvious benevolent intentions. But then, perhaps paranoid nihilism really is the safest policy and you have to assume that every website and every server and every person is out to get you and that they are also only figments of your own imagination. Of course, maybe Goldi is Christine and 4um/LP are just two of her servers. Maybe Goldi/Christine is an employee of FBI/NSA/ADL/SPLC and these sites are just honeypots, beginning to end. Let your imagination run wild. Or maybe Goldi and Christine are just two ladies who run chat forums, as they appear to be.
As much as I appreciate your thoughtful response, I have never implied that Goldi was responsible for having two websites if there were two websites. You started being rude to me by asking 'What is your problem anyway?' at post #43. Until then, I have never been anything but polite to you. I had a question and although you answered it well (finally), you have been rather mean and (I'll use your word) snarky. Than you for your very informative and sarcastic (or whatever you want to call that shit) post. Now go fuck yourself, you stupid, ass-kissing cunt. (Edited by wudidiz for grammatical errors.)
You first started off with liberty-post.org and liberty-post.net (which are entirely different domain names from libertypost.org and libertypost.net since domain names can include only lower-case letters, numerals, hyphens). Then you switched to complaining about libertypost.net, demanding attention and complaining that no one was posting about them. Despite the fact that the rest of us already knew what was up with those domains. A perfect example of why you tend to get banned from a lot of websites. What a martyr.
No I didn't. Bullshitting moron. You're really turning out to be quite the piece of work, aren't you. Didn't I tell you to go fuck yourself already?
You're really turning out to be quite the piece of work, aren't you. Didn't I tell you to go fuck yourself already? Oh. Yeah, I guess I did do that. I googled libertpost.net and one of the things that came up was this: http://whois.domaintools.com/liberty-post.net Still doesn't give Mizz smart-ass reason to jump down my throat because I'm ignorant about how websites work. What an asshole. I've been posting on this site for a couple of years now and I don't recall ever being treated as shittily as this 'TooConservative' poster has here.
I submit as evidence your last seven consecutive posts, all essentially content-free ad hominems. LOL. That you're posting like this so early in the morning in North America says something about you as well.
I submit
as evidence your last seven consecutive posts, all essentially content-free ad
hominems. LOL. Evidence for what? You seem to have a
problem understanding cause and effect. You see, you started it. I
wasn't complaining or demanding attention. I asked a simple, honest,
innocent question. I wanted to know what was up with the two domain names,
libertypost.net and libertypost.org. You had posted something about
downloading websites. You ASSUMED that I thought Gold-Lox was responsible
for.... I won't even get into it. YOU ASSUMED WRONG. You could have just
been nice and informed me with what you knew or ignored me. But you
didn't. You tried to project your own delusion, fear and doubt onto me as
if you knew what I was thinking and the reasoning for my question. You
appear to me to be one of those educated, knowledgable people that are
actually quite stupid. Anyone with enough intelligence could read this
thread and see what actually happened and how you attacked me without
provocation. You thought you would make a fool of me and you thought
wrong. You're the fool. No amount of clever wording or trickery or
projection or hypocrisy will change that. Now, go back to LP and try
attacking someone there and when they attack you back, go rat to Goldi about
how they attacked you. Asshole.
You should shut up. You are making an ass of yourself as usual. Conservative has been to nice to you. Also who are you to try and run people away from Christines site?
#47. To: wudidiz, christine, Pinguinite (#46) Forgive me Maam for my ignorance or paranoia or stupidity or whatever my problem is, but isn't the fact that there are two different urls mean that there are two different websites? Regardless of how similar they look? How snarky you are, wudi. Nevertheless, I'll answer you, noting the irony of you calling me rude. All website servers and all computers (and devices like addressable routers) occupy a dotted-IP address on the internet, just as your own home network has its own range of addresses. So your home network might use the common reserved range of addresses 192.168.0.1 - 192.168.255.255. In other words, a range of 65536 addresses (roughly) are reserved from any server or other LAN using them. So you might use an IP address of 192.168.0.1 for your DSL modem, the router you connect to it might use 192.168.1.1 and then all your home network computers could use addresses in the range 192.168.1.2 - 192.168.1.254. So a place like libertypost.org has a server with a dotted-IP address too. The dotted-IP address is what you access, the name is just a convenient way to access the actually address. You can point one domain name to a webserver. Or you can point a million names to a webserver. Or you can point different domain names to different subdirectories of the same server. Or whatever. Domain names don't care what you point them at. And the domain registrars are happy when you buy multiple names so they make more money. So, using common utilities, you can look up the actual dotted-IP address of LibertyPost. libertypost.net. 236 IN A 69.55.11.240 libertypost.org. 236 IN A 69.55.11.240 So you can see for yourself that the two names are the same. You can also just use the dotted-IP address since that is what you always end up using anyway. The name only translates to the number. So try clicking it: 69.55.11.240. Once again, you arrive at LibertyPost. Unless Goldi really is the supersecret hacker of all time. You can verify these domain names are at these numbers as I've told you by using your Command Prompt on Windows XP (or its equivalent on Vista or Win7). It's found in the Programs->Accessories folder. Once you open it, you can type nslookup to find your own primary, probably a DSL modem or router in your house. You can then enter libertypost.net or libertypost.org or whatever to check for yourself what the IP address really is. Type 'exit' to leave nslookup and then type 'exit' again to dismiss the command prompt and return to Windows. You can also google for 'nslookup' and use the online versions of these but you can't really trust those nslookup sites either because it might be Goldi running them all on her secret Evil Servers. Obviously, you can trust no one. It's really sad. Of course, Goldi's site, like 4um, runs from a single mySQL database using a bunch of Perl scripts and PHP pages, the most common kind of interactive site. So it is possible that she has two "copies" of her site, one with Good Decent Godfearing code, the other with the Supersecret Evil Goldi-Lox Killer code. So that if you go to libertypost.org, you get the Good Decent Godfearing version and if you go to libertypost.net, you get the Supersecrect Evil Goldi-Lox Killer version. Now, this whole notion of a good Goldi-server and an evil Goldi-server is somewhat daft. First, Goldi would have to be a lot more sophisticated than she is about servers and writing code. She doesn't do any of that stuff and hasn't even upgraded her older Pinguinite code (written by Neil and available for free) to the newer Pinguinite code that christine uses here which has more features, is more secure, etc. Beyond that, the idea of Goldi-two-servers is rather odd because the first principle of Being Evil is: Don't let anyone figure out that you really are evil. Now, I can go to any of the three known methods of reaching LibertyPost via libertypost.org or libertypost.net or 69.55.11.240 (among who knows how many other evil versions of LP there are), and the pages are the same, the most recent articles are the same, the most recent comments are the same. And again, I can sign in to any of them and they behave identically. Now, since everyone but you and a bunch of old cat ladies posting on Usenet or chatting on IRC already knows the above info, then what sense would it make to have the Goldi-two-servers thing? None whatsoever. What would be more interesting would be to have a single server with the two names pointed at it but monitor the access to it by the IP addresses of those you have banned from it. This way, you can be Evil to banned or disliked users but never look Evil to the Chosen People of LibertyPost (those who Goldi hasn't banned, this being 4um, I thought that was fun way to express it). And people on Goldi's ban list are always going to be assumed to be nursing a grudge anyway so no one will believe their accusations. So you might choose to be paranoid and think that Goldi is only running her Supersecret Evil Goldi-Code if you visit her server but she does show no signs of such code or admin sophistication and sneakiness. It really is just her chat board, I think. But you can believe or make up anything you like if you enjoy that more. But I see no reason to believe that Goldi or Christine are quite so powerful or obsessed over their chat boards as that would require. Now, me posting this much text to you, that might in and of itself seem Quite Evil to some folks despite my obvious benevolent intentions. But then, perhaps paranoid nihilism really is the safest policy and you have to assume that every website and every server and every person is out to get you and that they are also only figments of your own imagination. Of course, maybe Goldi is Christine and 4um/LP are just two of her servers. Maybe Goldi/Christine is an employee of FBI/NSA/ADL/SPLC and these sites are just honeypots, beginning to end. Let your imagination run wild. Or maybe Goldi and Christine are just two ladies who run chat forums, as they appear to be. TooConservative posted on 2009-11-22 8:06:06 ET Reply Trace Private Reply
Oh boy he said snarky. That is so so bad. Look at what you said fool.
Some of these Truthers are quite needy and paranoid. Kind of like a crack baby that screams for attention and then screams when anyone so much as looks them in the eye.
Speaking from personal experience? And why the cheap shot against those who know the official 9/11 legend is bullshit?
And why the cheap shot against those who know the official 9/11 legend is bullshit? Good luck getting a straight answer out of the piece of work calling itself TooConservative.
He (or she) appears to be ignoring me for some reason or another...
Maybe waiting to come up with a sarcastic, shnotty, stupid answer? Maybe to tell you how preposterous and paranoid you are?
Either of you ever going to make a meaningful informed post on the topic? However fulfilling you find making silly ad hominems on me to be, you may recall this thread is about web spiders, how to detect them, what they are, how they operate, etc.? Other than wudi's delusional notions about various liberty____.___ domain names, do you guys have anything to actually contribute to the topic?
You just don't know when to admit you're wrong do you? Got a link for that?
Like I would care quite so much about who boots you off a forum. But since you asked so nicely, I spent ten seconds looking at LP, just one site, for your posts. I found this LP thread. That was only one week ago, and here you are at 4um, trying to pretend that no one has ever banned you. Banning offense. You must be doing that to 1. force me to ban you 2 stir up a fight because of it ---------------- You called someone a pedophile. THAT isn't even a debatable ban. It's instant and without recourse. Have a nice life...someplace else. Trying to be a little disingenuous for an audience, wudi? I notice that somehow this thread has also become all about you. What a surprise. But, no, you aren't "needy", not at all.
ROTFLOL!!!!! Wudoofus, the pansy-ass, canuckistani displaying his abject stupidity once again!
Shut up, yukon.
LOL Amen!
#116. To: farmfriend, wudidiz (#113)
I think you two scared Yukon off... ;-)
Top • Page Up • Full Thread • Page Down • Bottom/LatestInternet geeks here! Who can determine how many web spiders/crawlers are on 4um??
[Apologies. Neil's 4um code won't let me post this spider code accurately. It's screwing my Sed command and turns the text red.]
#!/bin/sh
mkdir "$DIRNAME"
fi
/ s//http://$HOST1/p" lynxhtmlsource1 > lynxhtmlsource2
for URL in $(cat lynxhtmlsource2); do
ARTICLE="${URL:49:6}"
# echo "#$ARTICLE URL: $URL"
if [ ! -f "$DIRNAME/$ARTICLE" ]; then
$LYNX -source "$URL" > "4um/$ARTICLE"
echo "fetched $ARTICLE..."
let FETCHCOUNT+=1
fi
done
echo "$HOST: $FETCHCOUNT fetched"
else
echo "no new articles on $HOST"
fi
#3. To: TooConservative (#1)
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
#4. To: christine (#3)
#5. To: TooConservative (#4)
I could slowly download your entire database and reconstruct each poster's remarks.
#6. To: christine (#5)
can that be done on any forum?
#22. To: TooConservative, christine, Pinguinite, all (#6)
(Edited)
whois.domaintools.com/liberty-post.net
#23. To: All (#22)
(Edited)
#42. To: All (#23)
libertypost.net/
#43. To: wudidiz (#42)
#46. To: TooConservative (#43)
#47. To: wudidiz, christine, Pinguinite (#46)
Forgive me Maam for my ignorance or paranoia or stupidity or whatever my problem is, but isn't the fact that there are two different urls mean that there are two different websites? Regardless of how similar they look?
libertypost.net. 236 IN A 69.55.11.240
libertypost.org. 236 IN A 69.55.11.240
#50. To: TooConservative (#47)
(Edited)
#51. To: wudidiz (#50)
You started being rude to me by asking 'What is your problem anyway?' at post #43.
Now go fuck yourself, you stupid, ass-kissing cunt.
(Edited by wudidiz for grammatical errors.)
#52. To: TooConservative (#51)
You first started off with liberty-post.org and liberty-post.net (which are entirely different domain names from libertypost.org and libertypost.net since domain names can include only lower-case letters, numerals, hyphens).
#57. To: All (#52)
No I didn't. Bullshitting moron.
#58. To: All (#57)
#59. To: wudidiz (#58)
You call that complaining and demanding attention?
#60. To: TooConservative (#59)
You call that complaining and demanding attention?
#62. To: wudidiz (#60)
#63. To: A K A Stone (#62)
Okay moron, you call this being nice?
#65. To: wudidiz (#63)
#71. To: A K A Stone (#65)
Oh boy he said snarky. That is so so bad. Look at what you said fool.
#85. To: TooConservative (#71)
Kind of like a crack baby that screams for attention and then screams when anyone so much as looks them in the eye.
#86. To: FormerLurker (#85)
Speaking from personal experience?
#87. To: wudidiz, TooConservative (#86)
Good luck getting a straight answer out of the piece of work calling itself TooConservative.
#88. To: FormerLurker (#87)
(Edited)
#89. To: wudidiz, FormerLurker (#88)
#92. To: TooConservative, FormerLurker (#89)
delusional notions
#94. To: wudidiz, FormerLurker (#92)
Got a link for that?
783. To: wudidiz (#678)
#99. To: TooConservative, wudidiz (#94)
(Edited)
twoofer
#100. To: yukon (#99)
#113. To: wudidiz (#100)
Shut up, yukon.
Replies to Comment # 113.
End Trace Mode for Comment # 113.