生产者消费者模式的多线程小爬虫, 写完后进程居然都没开始??最后才发现pages队列太小。
def main():
base_url = '/////////'
pages = Queue(300)
detail_pages = Queue(100000)
imgs = Queue(100000)
for x in range(0, 540):
url = base_url.format(x)
pages.put(url)
for x in range(5):
Producer(pages, detail_pages, imgs).start()
for x in range(20):
Producer_2(pages, detail_pages, imgs).start()
for x in range(20):
Consumer(pages, detail_pages, imgs).start()