[Home]  [Headlines]  [Latest Articles]  [Latest Comments]  [Post]  [Sign-in]  [Mail]  [Setup]  [Help] 

Status: Not Logged In; Sign In

“The government wants to make sure that that does not come out. A huge part of our political system is predicated on blackmail,”

You Know What Happens Next

Cash Jordan: Half-Built Tower Abandoned… as ICE Deports Entire ‘Migrant Workforce’

Heavy rainfall causes flash flooding Tuesday night, some cars stuck in high water on Chicago's West

Biden Doctor PLEADS THE FIFTH, Refuses To Testify To Congress, Biden Pardons ARE VOID

Joe Rogan says FBI director Kash Patel played him for a fool and maga for fools with the Jeff Epstein files

Elon's AI System "Grok" Went Rogue And Has Been SHUT DOWN in an Emergency!

Earthquake Swarms at One of the MOST DANGEROUS Volcanoes in the USA

Ben Shapiro Declares Epstein Case CLOSED: ‘Facts on the Ground Have Changed’

Iran receives 40 Chinese J10-C Fighter Jets

China’s Railgun Is Now Battle-Ready, Thanks to Nuclear Power

Chinese Hypersonic Advancements! Deadly new missile could decimate entire US fleet in 20 minutes

Iran Confirms Massive Chinese HQ 9 B Missile Deal

Why Is Europe Hitting 114°F And Still Rising?

The INCREDIBLE Impacts of Methylene Blue

The LARGEST Eruptions since the Merapi Disaster in 2010 at Lewotobi Laki Laki in Indonesia

Feds ARREST 11 Leftists For AMBUSH On ICE, 2 Cops Shot, Organized Terror Cell Targeted ICE In Texas

What is quantum computing?

12 Important Questions We Should Be Asking About The Cover Up The Truth About Jeffrey Epstein

TSA quietly scraps security check that every passenger dreads

Iran Receives Emergency Airlift of Chinese Air Defence Systems as Israel Considers New Attacks

Russia reportedly used its new, inexpensive Chernika kamikaze drone in the Ukraine

Iran's President Says the US Pledged Israel Wouldn't Attack During Previous Nuclear Negotiations

Will Japan's Rice Price Shock Lead To Government Collapse And Spark A Global Bond Crisis

Beware The 'Omniwar': Catherine Austin Fitts Fears 'Weaponization Of Everything'

Roger Stone: AG Pam Bondi Must Answer For 14 Terabytes Claim Of Child Torture Videos!

'Hit Us, Please' - America's Left Issues A 'Broken Arrow' Signal To Europe

Cash Jordan Trump Deports ‘Thousands of Migrants’ to Africa… on Purpose

Gunman Ambushes Border Patrol Agents In Texas Amid Anti-ICE Rhetoric From Democrats

Texas Flood


Science/Tech
See other Science/Tech Articles

Title: Internet geeks here! Who can determine how many web spiders/crawlers are on 4um??
Source: none
URL Source: http://none
Published: Nov 20, 2009
Author: X-15
Post Date: 2009-11-20 03:04:08 by X-15
Keywords: None
Views: 5671
Comments: 170

"Web spiders/crawlers: programs that search websites looking for specific words or patterns to compile into a database."

A popular gun website I visit had 20 running, if 4um has less then I assume it has a lower profile in the eyes of FedGov.

Post Comment   Private Reply   Ignore Thread  


TopPage UpFull ThreadPage DownBottom/Latest

Begin Trace Mode for Comment # 39.

#1. To: X-15, Pinguinite, christine (#0)

Internet geeks here! Who can determine how many web spiders/crawlers are on 4um??

You'd need access to christine's server logs to get a good idea. However, there are many kinds of spiders, some quite difficult to detect.

Here is a quickie spider that I wrote. It runs on the Mac, OS X 10.5 Leopard. However, it is a standard Bash script and should work easily on Linux or Unix systems, probably in a Cygwin setup on Windows too.

The script uses Lynx, a venerable text-only browser, to fetch my Comments page to a file called htmlsource1. It then uses the stream editor Sed to parse this captured HTML file by scanning the right column for news stories, capturing the thread names and URLs at 4um to a file called htmlsource2.

It then uses Lynx to capture each thread to a separate file by thread number in a subdirectory called '4um'.

You could build a database or use text search tools like grep to mine the stored threads for info.

No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.

I'm presenting this so that 4um folks can get some idea of how these agencies and busybodies operate. Geeks know this stuff but the average person has no idea how easy it is. People should know how easy it is to database their every remark since we no longer live in a free country.

#!/bin/sh

# # lynx directory LYNX=/usr/bin/lynx

# IP host HOST='freedom4um.com' DIRNAME='4um'

if [ ! -d "$DIRNAME" ]; then
mkdir "$DIRNAME"
fi

$LYNX -source "http://$HOST/cgi-bin/latestcomments.cgi?SNSearch=1&EM=on&Fm=&To=TooConservative" > lynxhtmlsource1

sed -n "/

/,/


/ s//http://$HOST1/p" lynxhtmlsource1 > lynxhtmlsource2

FETCHCOUNT=0
for URL in $(cat lynxhtmlsource2); do
ARTICLE="${URL:49:6}"
# echo "#$ARTICLE URL: $URL"
if [ ! -f "$DIRNAME/$ARTICLE" ]; then
$LYNX -source "$URL" > "4um/$ARTICLE"
echo "fetched $ARTICLE..."
let FETCHCOUNT+=1
fi
done

rm -f lynxhtmlsource1 lynxhtmlsource2

if [ $FETCHCOUNT -ne 0 ]; then
echo "$HOST: $FETCHCOUNT fetched"
else
echo "no new articles on $HOST"
fi

exit

[Apologies. Neil's 4um code won't let me post this spider code accurately. It's screwing my Sed command and turns the text red.]

TooConservative  posted on  2009-11-20   6:15:42 ET  Reply   Untrace   Trace   Private Reply  


#30. To: TooConservative (#1)

No doubt, various federal agencies and people like ADL or SPLC use scripts like this to capture many forums and use grep and other search tools to scan each thread captured for relevant keywords to flag them for review by human beings.

No doubt whatsoever.

Pinguinite  posted on  2009-11-21   0:13:30 ET  Reply   Untrace   Trace   Private Reply  


#32. To: Pinguinite, christine, X-15 (#30)

Just for fun, I ran my script again for the first time in months. Since I didn't have any recent threads in my cache of 4um threads, it processed the entire news article sidebar. You can see the time stamp so you can check your logs and find this access.

Sat Nov 21 05:29:23 CST 2009
fetched 108669... (this one is your sticky thread)
fetched 110723...
fetched 110722...
fetched 110721...
fetched 110720...
fetched 110719...
fetched 110718...
fetched 110717...

...

fetched 110650...
fetched 110649...
freedom4um.com: 68 fetched
It only took 30 seconds to capture 68 full threads to my hard drive. I note that about ten threads are missing, did you remove some perhaps? Or maybe there's some little bug in my script; it wasn't like this code was highly important to me so I didn't go nuts over it.

Then I ran the script again a few minutes later. Since no new articles had been posted, it found nothing new to cache.


Sat Nov 21 05:34:29 CST 2009
no new articles on freedom4um.com

If I used a cron job to schedule this script to run regularly, say every hour or even every 12 hours, it would capture every thread posted at 4um. You'd have to revisit the thread, perhaps sniffing the headers to detect whether the thread had changed, in order to capture all the comments because those can come in days, months or even years later. You'd capture 99% of these chat forums' content if you just wait one month to capture a thread, parse it, and store it.

As I said, this crappy little Bash script isn't really even a spider, just a webserver capture script. If you wrote in a higher level language and sniff headers and such, it would be quite easy to become extremely sophisticated about this. I could churn out a real spider in very short order as could Neil or tons of script kiddies.

As for the posters here at 4um, one should assume that FBI and other agencies may employ agentes provocateur in the classic style they used against the Klan and others. Assuming that you had, say, a half-dozen posters here that post the most vitriolic content on the site, they could greatly raise the profile of 4um with SPLC/ADL/FBI quite easily. After all, what would they do if they couldn't point to the dire threat of rampant political incorrectness online? And how else could they assemble the "evidence" that there are vicious racists out there which justifies the usual begging letters sent out to donors by SPLC/ADL or the begging to Congress for staff and vast new computer systems by FBI/NSA/etc.? Hey, if that well isn't producing enough, you just have to prime the pump a little, baby!

So even if you're running a legit free speech forum (and I have no reason to believe anything else though I consider the possibility), that doesn't mean that your forum might not be used as a honeypot for the race hustlers like ADL/SPLC or by various letters of the alphabet like F, B, or I.

As I've demonstrated, it's quite easy to do.

TooConservative  posted on  2009-11-21   7:03:49 ET  Reply   Untrace   Trace   Private Reply  


#38. To: TooConservative, Pinguinite, christine, X-15 (#32)

As for the posters here at 4um, one should assume that FBI and other agencies may employ agentes provocateur in the classic style they used against the Klan and others.

I always assume that to be the case.

If some invites, directly or indirectly, comments which could be incriminating, incitement to violence, or libelous I assume, provisionally pending further data, that I am dealing with a provocateur or shill of some kind. I also assume the government and their filthy vermin traitorous minions know exactly who I am and where I live. So, they can go fuck themselves (pardon the French). Since I do not make personal threats against even politicians and figures whom I despise as long as they have to maintain a pretense that they are obeying the Constitution I'm safe. Once they try to dispense with it I head for the hills. ;-)

Original_Intent  posted on  2009-11-21   16:33:37 ET  Reply   Untrace   Trace   Private Reply  


#39. To: Original_Intent (#38)

If some invites, directly or indirectly, comments which could be incriminating, incitement to violence, or libelous I assume, provisionally pending further data, that I am dealing with a provocateur or shill of some kind.

It's rather amazing that in every recent case where someone is caught plotting "terrorism", the accused was found trying to buy weapons or bombs of whatever sort from an FBI agent. Seems the modern terrorists never have their own sources of such things, and they instead have to rely on sources given them by these FBI provocateurs.

I don't recall any so-called terrorist plots being busted that didn't involve an FBI agent being proactively involved in the mix somehow.

Pinguinite  posted on  2009-11-21   18:31:26 ET  Reply   Untrace   Trace   Private Reply  


Replies to Comment # 39.

#40. To: Pinguinite. all (#39)

I don't recall any so-called terrorist plots being busted that didn't involve an FBI agent being proactively involved in the mix somehow.

That's why they call it Home Grown Terrorism.

Lod  posted on  2009-11-21 18:40:09 ET  Reply   Untrace   Trace   Private Reply  


#41. To: Pinguinite (#39)

I don't recall any so-called terrorist plots being busted that didn't involve an FBI agent being proactively involved in the mix somehow.

Racking my memory I can't think of any exceptions. The elites need to keep the "Sarrorism" card in play to keep the suckers supporting their fear, and not opposing their control. So, the Feral Bureau of Instigation is doing its part by creating "terrorism" on demand. This is part of the ongoing array of PsyOps to keep the ignorant who believe "Pills" Limbaugh, Vannity, Michael Savage-Weiner, BOORtz, Faux Nooze, and the other controlled Networks, buying the need to shovel the dollars off to the Banksters and MIC along with their freedom and their future.

Original_Intent  posted on  2009-11-21 20:07:35 ET  Reply   Untrace   Trace   Private Reply  


End Trace Mode for Comment # 39.

TopPage UpFull ThreadPage DownBottom/Latest


[Home]  [Headlines]  [Latest Articles]  [Latest Comments]  [Post]  [Sign-in]  [Mail]  [Setup]  [Help]