用爬虫来爬取廖老师的python教程的url

59 阅读 0 评论 39 点赞

我是靠谱客的博主不安可乐，最近开发中收集的这篇文章主要介绍用爬虫来爬取廖老师的python教程的url，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

输入汉字，能够查询到知识点的url，就像下面这样

请输入你要查询的知识点：函数
h s
调用函数:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014316784721058975e02b46cc45cb836bb0827607738d000
定义函数:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431679203477b5b364aeba8c4e05a9bd4ec1b32911e2000
函数的参数:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431752945034eb82ac80a3e64b9bb4929b16eeed1eb9000
递归函数:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431756044276a15558a759ec43de8e30eb0ed169fb11000

这里要用到pinyin这个库，文档在这对于这个库不需要了解很多，只要会用就行。如何使程序的运行结果像上面那样，可以先观察一下网站源码

 <li id="001431608990315a01b575e2ab041168ff0df194698afac000" style="margin-left:1em;">
<a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431608990315a01b575e2ab041168ff0df194698afac000">Python简介</a>
</li>
<li id="0014316090478912dab2a3a9e8f4ed49d28854b292f85bb000" style="margin-left:1em;">
<a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014316090478912dab2a3a9e8f4ed49d28854b292f85bb000">安装Python</a>
</li>
<li id="00143161198846783e33de56d4041058c3dfc7e44ee1203000" style="margin-left:2em;">
<a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/00143161198846783e33de56d4041058c3dfc7e44ee1203000">Python解释器</a>
</li>
<li id="001431611988455689d4c116b2c4ed6aec000776c00ed52000" style="margin-left:1em;">
<a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431611988455689d4c116b2c4ed6aec000776c00ed52000">第一个Python程序</a>
</li>
<li id="0014316399410395f704750ee9440228135925a6ca1dad8000" style="margin-left:2em;">
<a href="/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014316399410395f704750ee9440228135925a6ca1dad8000">使用文本编辑器</a>
</li>

可以观察到每一个小的章节<li>标签的style属性和章节下面的每一小节是不一样的，因此这就是突破口。每一个小的章节的li标签为<li id="..." style="margin-left:1em;">，小的章节下面的小节li标签为<li id="..." style="margin-left:2em;">。获取小的章节之后要用兄弟节点来获取每一小节。if detial_tag == 'n':这段代码的解释在这。

for chapter_tag in original_tag.find_all("li", style="margin-left:1em;"):
'''
首先要获取每一个小的章节的章节名字，并转换为拼音存储
'''
L[sign].append(pinyin.get_initial(chapter_tag.find("a").get_text()))
'''
在每一个小的章节后面紧跟的兄弟标签就是小的章节下面的每一小节
当提取的li标签的style属性！="margin-left:2em，即进入下一章节（即进入下一轮循环）
'''
for detial_tag in chapter_tag.next_siblings:
if detial_tag == 'n':
pass
else:
style = detial_tag.get("style")
if style == 'margin-left:1em;':
break
else:
L[sign].append('n' + detial_tag.find("a").get_text() + ':
' + 'https://www.liaoxuefeng.com' + detial_tag.find("a").get("href"))
sign += 1

要用列表来存储获取的数据

L = []
for all_tag in original_tag.find_all("li", style="margin-left:1em;"):
count += 1
L.append([])

通过输入汉字来获取url的函数

def judgle(pinyin):
for i in range(sign):
if pinyin == L[i][0]:
for n in range(len(L[i])):
print(L[i][n])

下面就是所有的代码了

import urllib.request
import pinyin
from bs4 import BeautifulSoup
def judgle(pinyin):
for i in range(sign):
if pinyin == L[i][0]:
for n in range(len(L[i])):
print(L[i][n])
url = 'https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000'
response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')
url_soup = BeautifulSoup(html, 'html.parser')
original_tag = url_soup.find_all("ul", class_="uk-nav-side")[1]
count = 0
L = []
for all_tag in original_tag.find_all("li", style="margin-left:1em;"):
count += 1
L.append([])
sign = 0
for chapter_tag in original_tag.find_all("li", style="margin-left:1em;"):
L[sign].append(pinyin.get_initial(chapter_tag.find("a").get_text()))
for detial_tag in chapter_tag.next_siblings:
if detial_tag == 'n':
pass
else:
style = detial_tag.get("style")
if style == 'margin-left:1em;':
break
else:
L[sign].append('n' + detial_tag.find("a").get_text() + ':
' + 'https://www.liaoxuefeng.com' + detial_tag.find("a").get("href"))
sign += 1
chinese = input('请输入你要查询的知识点：')
pinyin = pinyin.get_initial(chinese)
judgle(pinyin)

运行结果

请输入你要查询的知识点：实战
s z
Day 1 - 搭建开发环境:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432170937506ecfb2f6adf8e4757939732f3e32b781c000
Day 2 - 编写Web App骨架:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/00143217133614028a244ea855b40a586b551c616d3b2c9000
Day 3 - 编写ORM:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014323389656575142d0bcfeec434e9639a80d3684a7da000
Day 4 - 编写Model:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432338991719a4c5c42ef08e4f44ad0f293ad728a27b000
Day 5 - 编写Web框架:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339008728d0ddbe19ee594980be3f0644a9371894000
Day 6 - 编写配置文件:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339034336cbf72acd43354d72831461e3871d9f2e000
Day 7 - 编写MVC:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339095180ce91c53cdab841bfa9c342a297b886fe000
Day 8 - 构建前端:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339124159f00f6ab876c44349a3fd8eb26d0c291e000
Day 9 - 编写API:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014323391480651a75b5fda4cb4c789208191682fc2c70000
Day 10 - 用户注册和登录:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339169382f45b9bd7b45d47ceb3e2b42846e0e991000
Day 11 - 编写日志创建页:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/00143233918656129f4ad3ac29e4f728dc72b5d2368215a000
Day 12 - 编写日志列表页:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339210950e063b4795d574036bc5dcf0c2449bc52000
Day 13 - 提升开发效率:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339228196a8eb6fb8832b48b5aa0d740346536ead000
Day 14 - 完成Web App:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339247097eea476bf61f8496092cc1b663eae1848000
Day 15 - 部署Web App:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/0014323392805925d5b69ddad514511bf0391fe2a0df2b0000
Day 16 - 编写移动App:
https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432339330096121ae7e38be44570b7fbd0d8faae26f6000