Created
June 16, 2018 06:16
-
-
Save Tomtao626/2d21e5b145b76773658b7baf6966b976 to your computer and use it in GitHub Desktop.
豆瓣爬虫
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
肖申克的救赎 https://movie.douban.com/subject/1292052/ | |
霸王别姬 https://movie.douban.com/subject/1291546/ | |
这个杀手不太冷 https://movie.douban.com/subject/1295644/ | |
阿甘正传 https://movie.douban.com/subject/1292720/ | |
美丽人生 https://movie.douban.com/subject/1292063/ | |
千与千寻 https://movie.douban.com/subject/1291561/ | |
泰坦尼克号 https://movie.douban.com/subject/1292722/ | |
辛德勒的名单 https://movie.douban.com/subject/1295124/ | |
盗梦空间 https://movie.douban.com/subject/3541415/ | |
机器人总动员 https://movie.douban.com/subject/2131459/ | |
三傻大闹宝莱坞 https://movie.douban.com/subject/3793023/ | |
海上钢琴师 https://movie.douban.com/subject/1292001/ | |
忠犬八公的故事 https://movie.douban.com/subject/3011091/ | |
放牛班的春天 https://movie.douban.com/subject/1291549/ | |
大话西游之大圣娶亲 https://movie.douban.com/subject/1292213/ | |
楚门的世界 https://movie.douban.com/subject/1292064/ | |
龙猫 https://movie.douban.com/subject/1291560/ | |
教父 https://movie.douban.com/subject/1291841/ | |
星际穿越 https://movie.douban.com/subject/1889243/ | |
熔炉 https://movie.douban.com/subject/5912992/ | |
乱世佳人 https://movie.douban.com/subject/1300267/ | |
触不可及 https://movie.douban.com/subject/6786002/ | |
无间道 https://movie.douban.com/subject/1307914/ | |
当幸福来敲门 https://movie.douban.com/subject/1849031/ | |
天堂电影院 https://movie.douban.com/subject/1291828/ | |
怦然心动 https://movie.douban.com/subject/3319755/ | |
十二怒汉 https://movie.douban.com/subject/1293182/ | |
搏击俱乐部 https://movie.douban.com/subject/1292000/ | |
少年派的奇幻漂流 https://movie.douban.com/subject/1929463/ | |
蝙蝠侠:黑暗骑士 https://movie.douban.com/subject/1851857/ | |
鬼子来了 https://movie.douban.com/subject/1291858/ | |
活着 https://movie.douban.com/subject/1292365/ | |
指环王3:王者无敌 https://movie.douban.com/subject/1291552/ | |
天空之城 https://movie.douban.com/subject/1291583/ | |
疯狂动物城 https://movie.douban.com/subject/25662329/ | |
大话西游之月光宝盒 https://movie.douban.com/subject/1299398/ | |
罗马假日 https://movie.douban.com/subject/1293839/ | |
飞屋环游记 https://movie.douban.com/subject/2129039/ | |
窃听风暴 https://movie.douban.com/subject/1900841/ | |
控方证人 https://movie.douban.com/subject/1296141/ | |
两杆大烟枪 https://movie.douban.com/subject/1293350/ | |
飞越疯人院 https://movie.douban.com/subject/1292224/ | |
闻香识女人 https://movie.douban.com/subject/1298624/ | |
哈尔的移动城堡 https://movie.douban.com/subject/1308807/ | |
海豚湾 https://movie.douban.com/subject/3442220/ | |
辩护人 https://movie.douban.com/subject/21937445/ | |
V字仇杀队 https://movie.douban.com/subject/1309046/ | |
死亡诗社 https://movie.douban.com/subject/1291548/ | |
教父2 https://movie.douban.com/subject/1299131/ | |
美丽心灵 https://movie.douban.com/subject/1306029/ | |
指环王2:双塔奇兵 https://movie.douban.com/subject/1291572/ | |
指环王1:魔戒再现 https://movie.douban.com/subject/1291571/ | |
摔跤吧!爸爸 https://movie.douban.com/subject/26387939/ | |
饮食男女 https://movie.douban.com/subject/1291818/ | |
情书 https://movie.douban.com/subject/1292220/ | |
美国往事 https://movie.douban.com/subject/1292262/ | |
狮子王 https://movie.douban.com/subject/1301753/ | |
钢琴家 https://movie.douban.com/subject/1296736/ | |
素媛 https://movie.douban.com/subject/21937452/ | |
天使爱美丽 https://movie.douban.com/subject/1292215/ | |
七宗罪 https://movie.douban.com/subject/1292223/ | |
小鞋子 https://movie.douban.com/subject/1303021/ | |
被嫌弃的松子的一生 https://movie.douban.com/subject/1787291/ | |
致命魔术 https://movie.douban.com/subject/1780330/ | |
看不见的客人 https://movie.douban.com/subject/26580232/ | |
音乐之声 https://movie.douban.com/subject/1294408/ | |
勇敢的心 https://movie.douban.com/subject/1294639/ | |
本杰明·巴顿奇事 https://movie.douban.com/subject/1485260/ | |
剪刀手爱德华 https://movie.douban.com/subject/1292370/ | |
低俗小说 https://movie.douban.com/subject/1291832/ | |
西西里的美丽传说 https://movie.douban.com/subject/1292402/ | |
黑客帝国 https://movie.douban.com/subject/1291843/ | |
拯救大兵瑞恩 https://movie.douban.com/subject/1292849/ | |
沉默的羔羊 https://movie.douban.com/subject/1293544/ | |
入殓师 https://movie.douban.com/subject/2149806/ | |
蝴蝶效应 https://movie.douban.com/subject/1292343/ | |
让子弹飞 https://movie.douban.com/subject/3742360/ | |
春光乍泄 https://movie.douban.com/subject/1292679/ | |
玛丽和马克思 https://movie.douban.com/subject/3072124/ | |
大闹天宫 https://movie.douban.com/subject/1418019/ | |
心灵捕手 https://movie.douban.com/subject/1292656/ | |
末代皇帝 https://movie.douban.com/subject/1293172/ | |
阳光灿烂的日子 https://movie.douban.com/subject/1291875/ | |
幽灵公主 https://movie.douban.com/subject/1297359/ | |
第六感 https://movie.douban.com/subject/1297630/ | |
重庆森林 https://movie.douban.com/subject/1291999/ | |
布达佩斯大饭店 https://movie.douban.com/subject/11525673/ | |
禁闭岛 https://movie.douban.com/subject/2334904/ | |
哈利·波特与魔法石 https://movie.douban.com/subject/1295038/ | |
狩猎 https://movie.douban.com/subject/6985810/ | |
大鱼 https://movie.douban.com/subject/1291545/ | |
致命ID https://movie.douban.com/subject/1297192/ | |
猫鼠游戏 https://movie.douban.com/subject/1305487/ | |
射雕英雄传之东成西就 https://movie.douban.com/subject/1316510/ | |
甜蜜蜜 https://movie.douban.com/subject/1305164/ | |
断背山 https://movie.douban.com/subject/1418834/ | |
一一 https://movie.douban.com/subject/1292434/ | |
告白 https://movie.douban.com/subject/4268598/ | |
阳光姐妹淘 https://movie.douban.com/subject/4917726/ | |
加勒比海盗 https://movie.douban.com/subject/1298070/ | |
上帝之城 https://movie.douban.com/subject/1292208/ | |
摩登时代 https://movie.douban.com/subject/1294371/ | |
穿条纹睡衣的男孩 https://movie.douban.com/subject/3008247/ | |
阿凡达 https://movie.douban.com/subject/1652587/ | |
爱在黎明破晓前 https://movie.douban.com/subject/1296339/ | |
消失的爱人 https://movie.douban.com/subject/21318488/ | |
风之谷 https://movie.douban.com/subject/1291585/ | |
爱在日落黄昏时 https://movie.douban.com/subject/1291990/ | |
侧耳倾听 https://movie.douban.com/subject/1297052/ | |
超脱 https://movie.douban.com/subject/5322596/ | |
红辣椒 https://movie.douban.com/subject/1865703/ | |
倩女幽魂 https://movie.douban.com/subject/1297447/ | |
恐怖直播 https://movie.douban.com/subject/21360417/ | |
驯龙高手 https://movie.douban.com/subject/2353023/ | |
萤火虫之墓 https://movie.douban.com/subject/1293318/ | |
幸福终点站 https://movie.douban.com/subject/1292274/ | |
小森林 夏秋篇 https://movie.douban.com/subject/25814705/ | |
菊次郎的夏天 https://movie.douban.com/subject/1293359/ | |
喜剧之王 https://movie.douban.com/subject/1302425/ | |
借东西的小人阿莉埃蒂 https://movie.douban.com/subject/4202302/ | |
七武士 https://movie.douban.com/subject/1295399/ | |
岁月神偷 https://movie.douban.com/subject/3792799/ | |
神偷奶爸 https://movie.douban.com/subject/3287562/ | |
杀人回忆 https://movie.douban.com/subject/1300299/ | |
真爱至上 https://movie.douban.com/subject/1292401/ | |
电锯惊魂 https://movie.douban.com/subject/1417598/ | |
贫民窟的百万富翁 https://movie.douban.com/subject/2209573/ | |
海洋 https://movie.douban.com/subject/3443389/ | |
喜宴 https://movie.douban.com/subject/1303037/ | |
谍影重重3 https://movie.douban.com/subject/1578507/ | |
东邪西毒 https://movie.douban.com/subject/1292328/ | |
记忆碎片 https://movie.douban.com/subject/1304447/ | |
雨人 https://movie.douban.com/subject/1291870/ | |
怪兽电力公司 https://movie.douban.com/subject/1291579/ | |
黑天鹅 https://movie.douban.com/subject/1978709/ | |
英雄本色 https://movie.douban.com/subject/1297574/ | |
疯狂原始人 https://movie.douban.com/subject/1907966/ | |
卢旺达饭店 https://movie.douban.com/subject/1291822/ | |
燃情岁月 https://movie.douban.com/subject/1295865/ | |
虎口脱险 https://movie.douban.com/subject/1296909/ | |
小森林 冬春篇 https://movie.douban.com/subject/25814707/ | |
7号房的礼物 https://movie.douban.com/subject/10777687/ | |
哈利·波特与死亡圣器(下) https://movie.douban.com/subject/3011235/ | |
傲慢与偏见 https://movie.douban.com/subject/1418200/ | |
海边的曼彻斯特 https://movie.douban.com/subject/25980443/ | |
恋恋笔记本 https://movie.douban.com/subject/1309163/ | |
荒蛮故事 https://movie.douban.com/subject/24750126/ | |
萤火之森 https://movie.douban.com/subject/5989818/ | |
完美的世界 https://movie.douban.com/subject/1300992/ | |
教父3 https://movie.douban.com/subject/1294240/ | |
纵横四海 https://movie.douban.com/subject/1295409/ | |
心迷宫 https://movie.douban.com/subject/25917973/ | |
魂断蓝桥 https://movie.douban.com/subject/1293964/ | |
猜火车 https://movie.douban.com/subject/1292528/ | |
花样年华 https://movie.douban.com/subject/1291557/ | |
玩具总动员3 https://movie.douban.com/subject/1858711/ | |
穿越时空的少女 https://movie.douban.com/subject/1937946/ | |
二十二 https://movie.douban.com/subject/26430107/ | |
雨中曲 https://movie.douban.com/subject/1293460/ | |
唐伯虎点秋香 https://movie.douban.com/subject/1306249/ | |
时空恋旅人 https://movie.douban.com/subject/10577869/ | |
超能陆战队 https://movie.douban.com/subject/11026735/ | |
蝙蝠侠:黑暗骑士崛起 https://movie.douban.com/subject/3395373/ | |
我是山姆 https://movie.douban.com/subject/1306861/ | |
人工智能 https://movie.douban.com/subject/1302827/ | |
请以你的名字呼唤我 https://movie.douban.com/subject/26799731/ | |
浪潮 https://movie.douban.com/subject/2297265/ | |
冰川时代 https://movie.douban.com/subject/1291578/ | |
朗读者 https://movie.douban.com/subject/2213597/ | |
香水 https://movie.douban.com/subject/1760622/ | |
罗生门 https://movie.douban.com/subject/1291879/ | |
血战钢锯岭 https://movie.douban.com/subject/26325320/ | |
爆裂鼓手 https://movie.douban.com/subject/25773932/ | |
未麻的部屋 https://movie.douban.com/subject/1395091/ | |
一次别离 https://movie.douban.com/subject/5964718/ | |
追随 https://movie.douban.com/subject/1397546/ | |
撞车 https://movie.douban.com/subject/1388216/ | |
可可西里 https://movie.douban.com/subject/1308857/ | |
战争之王 https://movie.douban.com/subject/1419936/ | |
恐怖游轮 https://movie.douban.com/subject/3011051/ | |
地球上的星星 https://movie.douban.com/subject/2363506/ | |
达拉斯买家俱乐部 https://movie.douban.com/subject/1793929/ | |
被解救的姜戈 https://movie.douban.com/subject/6307447/ | |
梦之安魂曲 https://movie.douban.com/subject/1292270/ | |
小萝莉的猴神大叔 https://movie.douban.com/subject/26393561/ | |
牯岭街少年杀人事件 https://movie.douban.com/subject/1292329/ | |
谍影重重 https://movie.douban.com/subject/1304102/ | |
魔女宅急便 https://movie.douban.com/subject/1307811/ | |
房间 https://movie.douban.com/subject/25724855/ | |
谍影重重2 https://movie.douban.com/subject/1308767/ | |
头脑特工队 https://movie.douban.com/subject/10533913/ | |
惊魂记 https://movie.douban.com/subject/1293181/ | |
忠犬八公物语 https://movie.douban.com/subject/1959195/ | |
阿飞正传 https://movie.douban.com/subject/1305690/ | |
再次出发之纽约遇见你 https://movie.douban.com/subject/6874403/ | |
青蛇 https://movie.douban.com/subject/1303394/ | |
碧海蓝天 https://movie.douban.com/subject/1300960/ | |
东京物语 https://movie.douban.com/subject/1291568/ | |
哪吒闹海 https://movie.douban.com/subject/1307315/ | |
秒速5厘米 https://movie.douban.com/subject/2043546/ | |
模仿游戏 https://movie.douban.com/subject/10463953/ | |
末路狂花 https://movie.douban.com/subject/1291992/ | |
源代码 https://movie.douban.com/subject/3075287/ | |
终结者2:审判日 https://movie.douban.com/subject/1291844/ | |
海盗电台 https://movie.douban.com/subject/3007773/ | |
一个叫欧维的男人决定去死 https://movie.douban.com/subject/26628357/ | |
绿里奇迹 https://movie.douban.com/subject/1300374/ | |
黑客帝国3:矩阵革命 https://movie.douban.com/subject/1302467/ | |
你的名字。 https://movie.douban.com/subject/26683290/ | |
新龙门客栈 https://movie.douban.com/subject/1292287/ | |
勇闯夺命岛 https://movie.douban.com/subject/1292728/ | |
这个男人来自地球 https://movie.douban.com/subject/2300586/ | |
卡萨布兰卡 https://movie.douban.com/subject/1296753/ | |
城市之光 https://movie.douban.com/subject/1293908/ | |
变脸 https://movie.douban.com/subject/1292659/ | |
无人知晓 https://movie.douban.com/subject/1292337/ | |
无耻混蛋 https://movie.douban.com/subject/1438652/ | |
初恋这件小事 https://movie.douban.com/subject/4739952/ | |
发条橙 https://movie.douban.com/subject/1292233/ | |
E.T. 外星人 https://movie.douban.com/subject/1294638/ | |
荒野生存 https://movie.douban.com/subject/1905462/ | |
黄金三镖客 https://movie.douban.com/subject/1401118/ | |
迁徙的鸟 https://movie.douban.com/subject/1292281/ | |
美国丽人 https://movie.douban.com/subject/1292062/ | |
爱在午夜降临前 https://movie.douban.com/subject/10808442/ | |
英国病人 https://movie.douban.com/subject/1291853/ | |
燕尾蝶 https://movie.douban.com/subject/1307793/ | |
疯狂的石头 https://movie.douban.com/subject/1862151/ | |
非常嫌疑犯 https://movie.douban.com/subject/1292214/ | |
无敌破坏王 https://movie.douban.com/subject/6534248/ | |
步履不停 https://movie.douban.com/subject/2222996/ | |
勇士 https://movie.douban.com/subject/3217169/ | |
叫我第一名 https://movie.douban.com/subject/4798888/ | |
穆赫兰道 https://movie.douban.com/subject/1292217/ | |
国王的演讲 https://movie.douban.com/subject/4023638/ | |
血钻 https://movie.douban.com/subject/1428175/ | |
彗星来的那一夜 https://movie.douban.com/subject/25807345/ | |
上帝也疯狂 https://movie.douban.com/subject/1297478/ | |
枪火 https://movie.douban.com/subject/1300741/ | |
聚焦 https://movie.douban.com/subject/25954475/ | |
遗愿清单 https://movie.douban.com/subject/1867345/ | |
大卫·戈尔的一生 https://movie.douban.com/subject/1305725/ | |
我爱你 https://movie.douban.com/subject/5908478/ | |
蓝色大门 https://movie.douban.com/subject/1308575/ | |
荒岛余生 https://movie.douban.com/subject/1298653/ | |
千钧一发 https://movie.douban.com/subject/1300117/ | |
黑鹰坠落 https://movie.douban.com/subject/1291824/ | |
爱·回家 https://movie.douban.com/subject/1299361/ | |
海街日记 https://movie.douban.com/subject/25895901/ | |
麦兜故事 https://movie.douban.com/subject/1302476/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import requests | |
import json | |
for i in range(5): | |
page_start = str(i*20) | |
url = 'https://movie.douban.com/explore#!type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=20&page_start=0' | |
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"} | |
response = requests.get(url = url,headers = headers,verify = False) | |
content = response.content.decode() | |
content_list = json.loads(content)['subjects'] | |
for item in content_list: | |
title = item['title'] | |
rate = item['rate'] | |
link = item['url'] | |
print(title,rate,link) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/python | |
# -*- coding: utf-8 -*- # | |
import requests,sys,re,openpyxl,MySQLdb,time | |
from bs4 import BeautifulSoup | |
reload(sys) | |
sys.setdefaultencoding('utf-8') | |
print '正在从豆瓣电影Top250抓取数据......' | |
# --------------------------创建列表用于存放数据-----------------------------# | |
nameList=[] | |
linkList=[] | |
#---------------------------------爬取模块------------------------------------# | |
def topMovie(): | |
for page in range(10): | |
url='https://movie.douban.com/top250?start='+str(page*25) | |
print '正在爬取第---'+str(page+1)+'---页......' | |
html=requests.get(url) | |
html.raise_for_status() | |
try: | |
soup=BeautifulSoup(html.text,'html.parser') | |
soup=str(soup) # 利用正则表达式需要将网页文本转换成字符串 | |
name=re.compile(r'<span class="title">(.*)</span>') | |
links=re.compile(r'<a class="" href="(.*)">') | |
movieNames=re.findall(name,soup) | |
movieLinks=re.findall(links,soup) | |
for name in movieNames: | |
if name.find('/')==-1: # 剔除英文名(英文名特征是含有'/') | |
nameList.append(name) | |
for link in movieLinks: | |
linkList.append(link) | |
except Exception as e: | |
print e | |
print '爬取完毕!' | |
return nameList,linkList | |
# ---------------------------------储存为文本文件-----------------------------------# | |
def save_to_txt(): | |
print 'txt文件存储中......' | |
try: | |
f=open('data.txt','w') | |
for i in range(250): | |
f.write(nameList[i]) | |
f.write('\t'*3) | |
f.write(linkList[i]) | |
f.write('\n') | |
f.close() | |
except Exception as e: | |
print e | |
print 'txt文件存储结束!' | |
# ---------------------------------储存为excel文件-----------------------------------# | |
def save_to_Excel(): | |
print 'Excel文件存储中......' | |
try: | |
wb=openpyxl.Workbook() | |
sheet=wb.get_active_sheet() | |
sheet.title='Movie Top 250' | |
for i in range(1,251): | |
one='a'+str(i) | |
two='b'+str(i) | |
sheet[one]=nameList[i-1] | |
sheet[two]=linkList[i-1] | |
wb.save(ur'豆瓣电影Top250.xlsx') # 保证文件名为中文 | |
except Exception as e: | |
print e | |
print 'Excel文件存储结束!' | |
# ---------------------------------储存到文数据库-----------------------------------# | |
def save_to_MySQL(): | |
print 'MySQL数据库存储中......' | |
try: | |
conn = MySQLdb.connect(host="127.0.0.1", user="root", passwd="123456", db="test", charset="utf8") | |
cursor = conn.cursor() | |
print "数据库连接成功" | |
cursor.execute('Drop table if EXISTS MovieTop250') # 如果表存在就删除 | |
time.sleep(3) | |
cursor.execute('''create table if not EXISTS MovieTop250( | |
movieName VARCHAR (200), | |
link VARCHAR (200))''') | |
for i in range(250): | |
sql='insert into MovieTop250(movieName,link) VALUES (%s,%s)' | |
param=(nameList[i],linkList[i]) | |
cursor.execute(sql,param) | |
conn.commit() | |
cursor.close() | |
conn.close() | |
except Exception as e: | |
print e | |
print 'MySQL数据库存储结束!' | |
# -------------------------------------主模块--------------------------------------# | |
if __name__=="__main__": | |
try: | |
topMovie() | |
save_to_txt() | |
save_to_Excel() | |
save_to_MySQL() | |
except Exception as e: | |
print e |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1豆瓣Top250电影爬虫并存入mysql
2豆瓣评论