[笔记] PhantomJS爬虫小记

PhantomJS这个headless浏览器还是很好用的，配合着v8与qt化的webkit引擎，JavaScript可以很好得去处理得到的页面数据。

去年在整合各个平台搭建搜索引擎的时候，写了一个爬虫脚本，去掉一些认证和特殊功能，一个简单的js就放到github上了：

https://github.com/dna2github/dna2oldmemory/blob/master/dna2poem/phantomjs/spider.js

使用的时候直接：

phantomjs spider.js <url>

这个小爬虫包含了：

－ 资源加载监听

－ 英语分词器和词频统计

－ DOM里<a>的href抓取

这样，当phantomjs加载一个页面的时候，就可以知道访问的所有api，图像，css，js等资源url是什么，加载了多长时间。

分词器和词频统计是为了后面给文章进行大概的分类，提高搜索结果的质量。

对于<a>的href抓取，这里没有判断href是不是javascript调用；抓取后有两个用途，一个是建立page rank模型，另一个是可以继续爬取下一个网页。

例如：

phantomjs spider.js https://en.wikipedia.org/wiki/Special:Random

- Sun Mar 27 2016 11:40:50 GMT+0800 (CST) - Sun Mar 27 2016 11:40:51 GMT+0800 (CST)   <GET: 302> /wiki/Special:Random

- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:51 GMT+0800 (CST)   <GET: 200> /wiki/Adele_Wong

- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=ext.cite.styles%7Cext.gadget.DRN-wizard%2CReferenceTooltips%2Ccharinsert%2Cfeatured-articles-links%2CrefToolbar%2Cswitcher%2Cteahouse%7Cext.tmh.thumbnail.styles%7Cext.uls.nojs%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.raggett%2CsectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Cwikibase.client.init&only=styles&skin=vector

- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=site&only=styles&skin=vector

- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector

- Sun Mar 27 2016 11:40:52 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST)   <GET: 200> /w/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=tbOYKAbm

url:  https://en.wikipedia.org/wiki/Adele_Wong

= frequencies ==========================================

total: 546

in 18

wong 16

film 13

she 8

singapore 8

at 7

of 7

festival 6

s 6

this 6

page 5

for 5

also 5

which 5

lead 5

as 5

links 5

chinese 5

wikipedia 5

2009 5

= word stream ==========================================

 adele wong from wikipedia free encyclopedia this chinese name family name wong this article may require cleanup to meet wikipedia s quality standards specific problem some of links not working anymore more citation needed as well please help improve this article if you july 2013 adele wong wong in 2008 music video knowing 懂 chinese name 王 惠 珊 pinyin wáng huìsh ā n mandarin ancestry shanghai china born 8 march 1983 singapore occupation actress singer writer screenwriter years active 2006-present adele wong born 8 march 1983 singaporean actress singer writer screenwriter contents 1 biography 2 filmography 3 references 4 external links biography edit born in singapore wong traces her ancestry to shanghai china she graduated from national university of singapore nus with honors degree in sociology at nus wong active in dance theatre upon graduation wong began her professional career in acting hosting writing she played lead role in damned 1 which part of omnibus feature film which premiered at osaka asian film festival in 2009 wong also played lead role in garden girls short film which premiered at singapore film festival in 2009 wong also active in television in singapore where she has been seen in several programs she recently starred as female lead annie in jack neo s first english-language drama series for okto titled happily ever after she also seen in 2009 mediacorp chinese new year television film kuchinta family reunion wong currently filming chinese-language television drama he ain t pesky he s my brother for mediacorp s channel 8 where she plays lead female character opposite christopher lee premiering in july 2009 wong s feature film debut as female lead role in days which screened at tokyo international film festival singapore international film festival short listed for new talent award at shanghai international film festival wong also sang theme song for film knowing 懂 2 which written produced by jim lim 林 倛 玉 besides acting wong writes both fiction non-fiction she hosts column on sinema s website 3 she also recently awarded grant for short film by singapore film commission 4 hanging gardens starring wong go into production in august 2009 filmography edit short films damned garden girls feature films days music videos 懂 knowing serials perfect deception as evelyn references edit damned in osaka asian film festival 懂 knowing music video sinema column sfc external links edit adele wong at internet movie database days official movie site watch my ass official movie site watch my ass myspace site categories 1983 birthsnational university of singapore alumnisingaporean actressesliving peoplesingaporean people of chinese descent navigation menu not logged intalkcontributionscreate accountlog inarticletalkreadeditview history main page contents featured content current events random article donate to wikipedia wikipedia store interaction help about wikipedia community portal recent changes contact page tools what links here related changes upload file special pages permanent link page information wikidata item cite this page print export create book download as pdf printable version languages add links this page last modified on 26 november 2013 at 21 33 text available under creative commons attribution-sharealike license additional terms may apply by using this site you agree to terms of use privacy policy wikipedia® registered trademark of wikimedia foundation inc non-profit organization privacy policyabout wikipediadisclaimerscontact wikipediadeveloperscookie statementmobile view

= links ================================================

https://en.wikipedia.org/wiki/Adele_Wong#mw-head

https://en.wikipedia.org/wiki/Adele_Wong#p-search

https://en.wikipedia.org/wiki/Chinese_name

https://en.wikipedia.org/wiki/Chinese_surname

https://en.wikipedia.org/wiki/Wang_(surname)

https://en.wikipedia.org/wiki/Wikipedia:Cleanup

https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit

https://en.wikipedia.org/wiki/File:Adele_Wong.jpg

https://en.wikipedia.org/wiki/Standard_Mandarin

https://en.wikipedia.org/wiki/Ancestral_home_(Chinese)

https://en.wikipedia.org/wiki/Shanghai

https://en.wikipedia.org/wiki/China

https://en.wikipedia.org/wiki/Singapore

https://en.wikipedia.org/wiki/Adele_Wong#Biography

https://en.wikipedia.org/wiki/Adele_Wong#Filmography

https://en.wikipedia.org/wiki/Adele_Wong#References

https://en.wikipedia.org/wiki/Adele_Wong#External_links

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit&section=1

https://en.wikipedia.org/wiki/National_University_of_Singapore

https://en.wikipedia.org/wiki/Adele_Wong#cite_note-1

https://en.wikipedia.org/wiki/Singapore_Film_Festival

https://en.wikipedia.org/wiki/Jack_Neo

https://en.wikipedia.org/wiki/Okto

https://en.wikipedia.org/wiki/Happily_Ever_After_(TV_series)

https://en.wikipedia.org/wiki/MediaCorp

https://en.wikipedia.org/wiki/MediaCorp

https://en.wikipedia.org/wiki/Christopher_Lee

https://en.wikipedia.org/w/index.php?title=The_Days_(Singaporean_Film)&action=edit&redlink=1

https://en.wikipedia.org/wiki/Tokyo_International_Film_Festival

https://en.wikipedia.org/wiki/Singapore_International_Film_Festival

https://en.wikipedia.org/wiki/Shanghai_International_Film_Festival

https://en.wikipedia.org/wiki/Adele_Wong#cite_note-2

https://en.wikipedia.org/w/index.php?title=Jim_Lim&action=edit&redlink=1

https://en.wikipedia.org/wiki/Adele_Wong#cite_note-3

https://en.wikipedia.org/wiki/Adele_Wong#cite_note-4

https://en.wikipedia.org/wiki/Hanging_Gardens

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit&section=2

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit&section=3

https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-1

http://www.oaff.jp/lineup/lineup02.html#meet_02

https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-2

https://www.youtube.com/watch?v=grf-FZqkPao&feature=relate

https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-3

http://www.oaff.jp/lineup/lineup02.html#meet_02

https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-4

http://www.sfc.org.sg/main.html

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit&section=4

http://www.imdb.com/name/nm3044125/

https://en.wikipedia.org/wiki/Internet_Movie_Database

http://www.thedaysmovie.com/

http://watchmyass.wordpress.com/

https://www.myspace.com/watchmyassmovie

https://en.wikipedia.org/wiki/Wikipedia:Persondata

https://en.wikipedia.org/wiki/Singapore

https://en.wikipedia.org/w/index.php?title=Adele_Wong&oldid=583442184

https://en.wikipedia.org/wiki/Help:Category

https://en.wikipedia.org/wiki/Category:1983_births

https://en.wikipedia.org/wiki/Category:National_University_of_Singapore_alumni

https://en.wikipedia.org/wiki/Category:Singaporean_actresses

https://en.wikipedia.org/wiki/Category:Living_people

https://en.wikipedia.org/wiki/Category:Singaporean_people_of_Chinese_descent

https://en.wikipedia.org/wiki/Category:Articles_needing_cleanup_from_July_2013

https://en.wikipedia.org/wiki/Category:All_articles_needing_cleanup

https://en.wikipedia.org/wiki/Category:Cleanup_tagged_articles_with_a_reason_field_from_July_2013

https://en.wikipedia.org/wiki/Category:Wikipedia_pages_needing_cleanup_from_July_2013

https://en.wikipedia.org/wiki/Category:Use_dmy_dates_from_November_2010

https://en.wikipedia.org/wiki/Special:MyTalk

https://en.wikipedia.org/wiki/Special:MyContributions

https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Adele+Wong&type=signup

https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Adele+Wong

https://en.wikipedia.org/wiki/Adele_Wong

https://en.wikipedia.org/wiki/Talk:Adele_Wong

https://en.wikipedia.org/wiki/Adele_Wong#

https://en.wikipedia.org/wiki/Adele_Wong

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=history

https://en.wikipedia.org/wiki/Adele_Wong#

https://en.wikipedia.org/wiki/Main_Page

https://en.wikipedia.org/wiki/Main_Page

https://en.wikipedia.org/wiki/Portal:Contents

https://en.wikipedia.org/wiki/Portal:Featured_content

https://en.wikipedia.org/wiki/Portal:Current_events

https://en.wikipedia.org/wiki/Special:Random

https://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_en.wikipedia.org&uselang=en

https://shop.wikimedia.org/

https://en.wikipedia.org/wiki/Help:Contents

https://en.wikipedia.org/wiki/Wikipedia:About

https://en.wikipedia.org/wiki/Wikipedia:Community_portal

https://en.wikipedia.org/wiki/Special:RecentChanges

https://en.wikipedia.org/wiki/Wikipedia:Contact_us

https://en.wikipedia.org/wiki/Special:WhatLinksHere/Adele_Wong

https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Adele_Wong

https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard

https://en.wikipedia.org/wiki/Special:SpecialPages

https://en.wikipedia.org/w/index.php?title=Adele_Wong&oldid=583442184

https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=info

https://www.wikidata.org/wiki/Q4681907

https://en.wikipedia.org/w/index.php?title=Special:CiteThisPage&page=Adele_Wong&id=583442184

https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=book_creator&referer=Adele+Wong

https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=render_article&arttitle=Adele+Wong&returnto=Adele+Wong&oldid=583442184&writer=rdf2latex

https://en.wikipedia.org/w/index.php?title=Adele_Wong&printable=yes

https://en.wikipedia.org/wiki/Adele_Wong#

https://www.wikidata.org/wiki/Q4681907#sitelinks-wikipedia

https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License

https://creativecommons.org/licenses/by-sa/3.0/

https://wikimediafoundation.org/wiki/Terms_of_Use

https://wikimediafoundation.org/wiki/Privacy_policy

https://www.wikimediafoundation.org/

https://wikimediafoundation.org/wiki/Privacy_policy

https://en.wikipedia.org/wiki/Wikipedia:About

https://en.wikipedia.org/wiki/Wikipedia:General_disclaimer

https://en.wikipedia.org/wiki/Wikipedia:Contact_us

https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute

https://wikimediafoundation.org/wiki/Cookie_statement

https://en.m.wikipedia.org/w/index.php?title=Adele_Wong&mobileaction=toggle_view_mobile

https://wikimediafoundation.org/

https://www.mediawiki.org/

最后

以上就是碧蓝楼房为你收集整理的[笔记] PhantomJS爬虫小记的全部内容，希望文章能够帮你解决[笔记] PhantomJS爬虫小记所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。