Comments on: Of Spiders and Scrapers: Decomposing Web Pages 101 http://www.linux-mag.com/id/7448/ Open Source, Open Standards Fri, 10 May 2013 08:56:11 +0000 hourly 1 http://wordpress.org/?v=3.1 By: tdbtdb http://www.linux-mag.com/id/7448/#comment-6785 tdbtdb Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7448/#comment-6785 <p>wget example does not work for me. The first grep seems a bit strange,<br /> <code><br /> grep \'<br /> `</code></p> <p>What would that do that would be good? </p> wget example does not work for me. The first grep seems a bit strange,

grep \'
`

What would that do that would be good?

]]> By: vijayekm http://www.linux-mag.com/id/7448/#comment-6786 vijayekm Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7448/#comment-6786 <p>Mechanize is good for most of the cases.</p> <p>However for interacting with pages with javascript I mostly depend on IE and watir. </p> <p>Not sure if there are any better choices out there. </p> Mechanize is good for most of the cases.

However for interacting with pages with javascript I mostly depend on IE and watir.

Not sure if there are any better choices out there.

]]> By: perljunkie http://www.linux-mag.com/id/7448/#comment-6787 perljunkie Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7448/#comment-6787 <p>When I\'m scraping, it\'s very seldom that I find myself needing something other than some text on a page. For that purpose, I\'ve almost always found the dumped ASCII output of Lynx very convenient and less of a hassle than XPathing and reading node structures. I have a small Perl module called WebPage.pm that handles all this and can recursively spider down through pages (using the references list in the Lynx output) if desired. Most of the time, it works great.</p> <p>-pj </p> When I\’m scraping, it\’s very seldom that I find myself needing something other than some text on a page. For that purpose, I\’ve almost always found the dumped ASCII output of Lynx very convenient and less of a hassle than XPathing and reading node structures. I have a small Perl module called WebPage.pm that handles all this and can recursively spider down through pages (using the references list in the Lynx output) if desired. Most of the time, it works great.

-pj

]]>