Skip to content

Instantly share code, notes, and snippets.

@kuwa72

kuwa72/scrape.pl

Created February 20, 2013 07:02
Show Gist options
  • Save kuwa72/4993540 to your computer and use it in GitHub Desktop.
Save kuwa72/4993540 to your computer and use it in GitHub Desktop.
Download ZIP
CGI, Scrape site-titles in LAN nodes with threads. **Caution! Its SEGV on win32 perl.** LAN内で動いているWebサイトからTitleを抜いてリンク集にするCGIです。 マルチスレッドのおかげで高速に動作しますが、Win32のPerlは異常終了するようです。
Raw
scrape.pl
#!"C:\xampp\perl\bin\perl.exe"
print "Content-type: text/html\n\n";
use strict;
use warnings;
use threads;
use threads::shared;
use Thread::Semaphore;
use HTTP::Tiny;
use List::Util 'shuffle';
print '<html>';
print '<head>';
print '<meta name="author" content="KUWASHIMA Yuichiro">';
print '<link href="/xampp/xampp.css" rel="stylesheet" type="text/css">';
print '<title>List</title>';
print '</head>';
print "<body>&nbsp;<p><h1>ShopList</h1>";
my @threads;
my $semaphore = Thread::Semaphore->new(1);
my $baseurl = 'http://192.168.11.';
sub my_thread {
my $i = shift;
my $ua = shift;
my $semaphore = shift;
my $url = "http://192.168.11.$i/";
my $res = $ua->get($url);
#die "Can't download $url\n" unless $res->{success};
return unless $res->{success};
my $content = $res->{content};
my $title;
while ($content =~ m{<title>([^<]+)</title>}gxmsi) {
$title = $1;
}
$semaphore->down;
print "<li><a href=\"$url\">$title</a>\n";
$semaphore->up;
return;
}
my @ips = shuffle((0 .. 255));
print "<ul>";
while ($#ips > 0) {
foreach (1 .. 64) {
if ($#ips > 0) {
my $i = pop(@ips);
my $ua = HTTP::Tiny->new(
agent => 'sample ua',
timeout => 10,
);
push(@threads, threads->new(\&my_thread, $i, $ua, $semaphore));
}
}
foreach (@threads) {
$_->join;
}
@threads = ();
}
print "</url>";
print "</body></html>";
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment