手把手教你30行代码爬取《某某某报》-编程知识

一、项目简介

大家好，这个爬虫项目是自己开发用来阅览报纸的，大概已经用了5年了，很稳定。看到社区爬虫征集令，就来献丑了。

1.思路介绍

1.爬《某某某报》官网，获取指定日期报纸pdf
2.合并爬取的每片pdf,并进行合并
3.输出终版报纸pdf

2.功能介绍

本项实现根据爬虫原理获取数据，最终输出《某某某报》电子版全过程
代码简短，适合爬虫初学者练手

二、项目环境构建

1.python环境

起码python3吧，python2很少有人用。

2.依赖包

依赖os、httplib2、PyPDF2等包

2.1 PyPDF2等包

PyPDF2是一个纯Python PDF库，能够读取、写入、分割、合并、裁剪和转换PDF文件的页面，还可以为PDF文件添加自定义数据、查看选项和密码，以及从PDF中检索文本和元数据。

需要注意的是，PyPDF2不能操作PDF获取文字信息，且PyPDF2=3.0.X将是PyPDF2的最后一个版本，开发将继续使用pypdf==3.1.012

2.2 httplib2包

httplib2是一个使用Python编写的，支持非常全面的HTTP特性的库1。

httplib2需要Python2.3或更高版本的运行环境，0.5.0版本及其以后包含了对Python3的支持。httplib2支持HTTP/1.1，具有持久连接、连接池、分块传输编码、内容编码（gzip和deflate）、身份验证（基本、摘要和OAuth）、缓存、重试、重定向、错误处理、进度通知、SOCKS代理、TLS/SSL等特性1

!pip install httplib2 pypdf2

Collecting httplib2Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)|████████████████████████████████| 96 kB 16 kB/s            
[?25hCollecting pypdf2Downloading pypdf2-3.0.1-py3-none-any.whl (232 kB)|████████████████████████████████| 232 kB 11 kB/s             
[?25hCollecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)|████████████████████████████████| 98 kB 13 kB/s             
[?25hCollecting typing_extensions>=3.10.0.0Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Collecting dataclassesDownloading dataclasses-0.8-py3-none-any.whl (19 kB)
Installing collected packages: typing-extensions, pyparsing, dataclasses, pypdf2, httplib2Attempting uninstall: typing-extensionsFound existing installation: typing-extensions 3.7.4Uninstalling typing-extensions-3.7.4:Successfully uninstalled typing-extensions-3.7.4Attempting uninstall: pyparsingFound existing installation: pyparsing 2.1.10Uninstalling pyparsing-2.1.10:Successfully uninstalled pyparsing-2.1.10
[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
auto-sklearn 0.5.2 requires scikit-learn<0.20,>=0.19, but you have scikit-learn 0.21.1 which is incompatible.[0m
Successfully installed dataclasses-0.8 httplib2-0.22.0 pyparsing-3.0.7 pypdf2-3.0.1 typing-extensions-4.1.1

三、实操环节

1.页面分析

分析目标：paper.people.com.cn
版面url：paper.people.com.cn/rmrb/html/2…
版面pdf下载地址：paper.people.com.cn/rmrb/images…

截图_选择区域_20240314231407.png

2.导入所需包

import os
import httplib2
import urllib.request
import PyPDF2
import time

3.下载单页pdf

def downFile1(mytime):year = mytime.tm_yearmonth = mytime.tm_monday = mytime.tm_mdayyear = str(year)month = str('%02d' % month)day = str('%02d' % day)files = []for i in range(1, 9, 1):connection = httplib2.HTTPConnectionWithTimeout("paper.people.com.cn")path = 'rmrb/images/' + year + '-' + month + '/' + day + '/' + '01' + '/rmrb' + year + month + day + str('%02d' % i) + '.pdf'connection.request("HEAD", path)response = connection.getresponse()if response.status == 400:name = 'rmrb' + year + month + day + str('%02d' % i) + '.pdf'url = r'http://paper.people.com.cn/rmrb/images/' + year + '-' + month + '/' + day + '/' + str('%02d' % i) + '/' + nameurllib.request.urlretrieve(url, name)files.append(name)targetfile = 'rmrb' + year + month + day + '.pdf'merge_pdfs(files, targetfile)for file in files:os.remove(file)

4.合并pdf

def merge_pdfs(paths, output):pdf_writer = PyPDF2.PdfWriter()for path in paths:pdf_reader = PyPDF2.PdfReader(path, strict=False)for page_num in range(len(pdf_reader.pages)):pdf_writer.add_page(pdf_reader.pages[page_num])with open(output, 'wb') as out:pdf_writer.write(out)