selenium模拟浏览器查询导出参考文献-编程知识

selenium模拟浏览器查询导出参考文献

通过使用Selenium和BeautifulSoup，在CNKI网站上，以"知识图谱"为关键词，通过自动化工具在搜索页面提取相关文章信息。点击清楚并全选进行文献导出，随后从导出页面和管理导出的页面提取参考文献。

浏览器及WebDriver下载
https://googlechromelabs.github.io/chrome-for-testing/#stable

放到/usr/local/bin （mac的配置）

mv chromedriver /usr/local/bin
cd /usr/local/bin
xattr -d com.apple.quarantine chromedriver

提取文献代码

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoupdef driver_open(driver, key_word):url = "https://www.cnki.net/"driver.get(url)time.sleep(2)driver.find_element(By.CSS_SELECTOR ,'#txt_SearchText').send_keys(key_word)time.sleep(2)# 点击搜索按钮driver.find_element(By.CSS_SELECTOR ,'body > div.wrapper.section1 > div.searchmain > div > div.input-box > input.search-btn').click()time.sleep(5)driver.find_element(By.CSS_SELECTOR,'#briefBox > div:nth-child(1) > div > div.toolbar-col > div.checkcount > a').click()time.sleep(2)driver.find_element(By.CSS_SELECTOR,'#selectCheckAll1').click()driver.find_element(By.CSS_SELECTOR,'#batchOpsBox > li:nth-child(2) > a').click()driver.find_element(By.CSS_SELECTOR,'#batchOpsBox > li:nth-child(2) > ul > li.export > a').click()driver.find_element(By.CSS_SELECTOR,'#batchOpsBox > li:nth-child(2) > ul > li.export > ul > li:nth-child(1) > a').click()time.sleep(2)url1="https://kns.cnki.net/dm8/manage/export.html?language=CHS&uniplatform=NZKPT"driver.get(url1)time.sleep(5)driver.find_element(By.CSS_SELECTOR,'#result > ul').click()content = driver.page_source.encode('utf-8')soup = BeautifulSoup(content, 'lxml')ul = soup.find_all('ul')ul = BeautifulSoup(str(ul[5]), 'lxml')lis = ul.find_all('li')text=""for li in lis:text = text + li.get_text().strip().replace(" ","")+"\n"return text
if __name__ == '__main__':chrome_options= webdriver.ChromeOptions()chrome_options.add_argument('--headless')chrome_options.add_argument('--disable‐gpu')path="/usr/local/bin"chrome_options.binary_location = pathdriver = webdriver.Chrome(options=chrome_options)text = driver_open(driver, '知识图谱') print(text)