文本提取及处理

文本提取及处理 

 标题提取：

wljdeMacBook-Pro:Desktop wlj$ scrapy shell "http://wz.sun0769.com/html/question/201806/375913.shtml"
2018-06-21 23:44:36 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-06-21 23:44:36 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 3.6.5 (default, Apr 25 2018, 14:23:58) - [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Darwin-17.5.0-x86_64-i386-64bit
2018-06-21 23:44:36 [scrapy.crawler] INFO: Overridden settings: {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
2018-06-21 23:44:36 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage']
2018-06-21 23:44:36 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-06-21 23:44:36 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-06-21 23:44:36 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-06-21 23:44:36 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-06-21 23:44:36 [scrapy.core.engine] INFO: Spider opened
2018-06-21 23:44:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://wz.sun0769.com/html/question/201806/375913.shtml> (referer: None)
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x102d30c88>
[s]   item       {}
[s]   request    <GET http://wz.sun0769.com/html/question/201806/375913.shtml>
[s]   response   <200 http://wz.sun0769.com/html/question/201806/375913.shtml>
[s]   settings   <scrapy.settings.Settings object at 0x103b276d8>
[s]   spider     <DefaultSpider 'default' at 0x103ddfc18>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects 
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
>>> response.url
'http://wz.sun0769.com/html/question/201806/375913.shtml'
>>> from scrapy.linkextractors import LinkExtractor
>>> response.xpath('//div[@class="pagecenter p3"]//strong/text()').extract()[0]
' 提问：森林湖路乱停乱放xa0xa0编号:189251xa0xa0'
>>> 

 正文提取：

>>> response.xpath('//div[@class="c1 text14_2"]/text()').extract()[0]
'xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。'
>>> response.xpath('//div[@class="c1 text14_2"]/text()').extract()
['xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。', 'xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。', 'xa0xa0xa0xa0    请多派警力加强执法。']
>>> 

 python3中处理：

wljdeMacBook-Pro:~ wlj$ python3
Python 3.6.5 (default, Apr 25 2018, 14:23:58) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> strs = ' 提问：森林湖路乱停乱放xa0xa0编号:189251xa0xa0'
>>> strs.split(' ')[-1].split(":")[-1]
'189251xa0xa0'
>>> strs.split(' ')[-1].split(":")[-1].strip()
'189251'
>>> lists = ['xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的 停，还衍生出噪音问题。', 'xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。', 'xa0xa0xa0xa0    请多派警力加强执法。']
>>> lists
['xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。', 'xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。', 'xa0xa0xa0xa0    请多派警力加强执法。']
>>> string = "".join(lists)
>>> string
'xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。xa0xa0xa0xa0    请多派警力加强执法。'
>>> string = "*****".join(lists)
>>> string
'xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。*****xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。*****xa0xa0xa0xa0    请多派警力加强执法。'
>>> string = "".join(lists)
>>> string
'xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。xa0xa0xa0xa0    请多派警力加强执法。'
>>> string.replace(" ","")
'xa0xa0xa0xa0南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。xa0xa0xa0xa0而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。xa0xa0xa0xa0请多派警力加强执法。'
>>> string.strip()
'南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。xa0xa0xa0xa0    请多派警力加强执法。'
>>> string.lstrip()
'南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。xa0xa0xa0xa0    请多派警力加强执法。'
>>> string.rstrip()
'xa0xa0xa0xa0    南城森林湖楼盘的森林湖路，车辆乱停乱放越趋严重。现在四车道的马路，一到晚上经常占用2~3条车道，严重影响通行、造成行车安全隐患。严重的乱停，还衍生出噪音问题。xa0xa0xa0xa0    而且这条路不知为何经常有大货车出入，每当有大货车出入，必造成拥堵。xa0xa0xa0xa0    请多派警力加强执法。'
>>>