python并发编程实战（四）：使用多线程，python爬虫被加速10倍（python 爬虫 多线程）

python并发编程实战（四）：使用多线程，python爬虫被加速10倍（python 爬虫 多线程）

python创建多线程的方法

单线程、多线程爬取博客园速度对比

tmp/blog_spider.py

import requestsurls = [ f" for page in range(1, 50+1)]def craw(url): r = requests.get(url) print(url, len(r.text))craw(urls[0])

01.multi_thread_craw.py

import blog_spiderimport threading, timedef single_thread(): print("single_thread begin") for url in blog_spider.urls: blog_spider.craw(url) print("single_thread end")def multi_thread(): print("single_thread begin") threads = [] for url in blog_spider.urls: threads.append( threading.Thread(target=blog_spider.craw, args=(url, )) ) for thread in threads: thread.start() for thread in threads: thread.join() print("single_thread end")if __name__ == '__main__': start = time.time() single_thread() end = time.time() print("single_thread cost: ", end - start, "seconds") start = time.time() multi_thread() end = time.time() print("multi_thread cost: ", end - start, "seconds")

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。