Title: Internet geeks here! Who can determine how many web spiders/crawlers are on 4um?? Source:
none URL Source:http://none Published:Nov 20, 2009 Author:X-15 Post Date:2009-11-20 03:04:08 by X-15 Keywords:None Views:6324 Comments:170
"Web spiders/crawlers: programs that search websites looking for specific words or patterns to compile into a database."
A popular gun website I visit had 20 running, if 4um has less then I assume it has a lower profile in the eyes of FedGov.
Internet geeks here! Who can determine how many web spiders/crawlers are on 4um??
You'd need access to christine's server logs to get a good idea. However, there are many kinds of spiders, some quite difficult to detect.
Here is a quickie spider that I wrote. It runs on the Mac, OS X 10.5 Leopard. However, it is a standard Bash script and should work easily on Linux or Unix systems, probably in a Cygwin setup on Windows too.
The script uses Lynx, a venerable text-only browser, to fetch my Comments page to a file called htmlsource1. It then uses the stream editor Sed to parse this captured HTML file by scanning the right column for news stories, capturing the thread names and URLs at 4um to a file called htmlsource2.
It then uses Lynx to capture each thread to a separate file by thread number in a subdirectory called '4um'.
You could build a database or use text search tools like grep to mine the stored threads for info.
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.
No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.