使用selenium爬取简书用户的最新评论、标题、评论时间

偏执的太偏执、 2022-12-29 09:15 156阅读 0赞

> **任务要求**：  
> 网址为https://www.jianshu.com/u/9104ebf5e177，爬取内容为简书用户的最新评论中的评论题目、评论内容及评论时间，爬取5页，用selenium爬取，将这些评论存入Excel文件中，文件后缀为.xls。将ipynb文件和后缀为.xls的文件压缩打包。

![在这里插入图片描述][resize_m_lfit_w_962_pic_center]

--------------------

> 通过博客对selenium的简单介绍，现在开始实战啦，没有看过的，可以先看看

*  [使用selenium定位获取标签对象并提取数据][selenium]
 *  [利用selenium爬取数据总结][selenium 1]

直接上代码

# -*- coding: utf-8 -*-
    """
    Created on Mon Dec 21 14:03:06 2020
    
    @author: kun
    """
    
    from selenium import webdriver
    from time import sleep
    from random import uniform
    import pandas as pd
    
    url = "https://www.jianshu.com/u/9104ebf5e177"
    browser = webdriver.Chrome()
    browser.maximize_window()
    browser.implicitly_wait(3)
    browser.get(url)
    
    title,comment,time1 =[],[],[]
    browser.find_element_by_xpath("/html/body/div[2]/div/div[1]/ul/li[3]/a").click()
    #browser.find_element_by_link_text("最新评论").click()
    sleep(uniform(2,3))
    for i in range(1,20):
        browser.execute_script("window.scrollTo(0,document.body.scrollHeight)") 
        time.sleep(uniform(1,2))
    def get_info():
        titles = browser.find_elements_by_css_selector("a[class='title']")
        comments = browser.find_elements_by_css_selector("p[class='abstract']")
        times = browser.find_elements_by_css_selector(" div > div > span.time")   
        try:
            for i in titles:
                title.append(i.text)
            for i in comments:
                comment.append(i.text)
            for i in times:
                time1.append(i.text)
            data = {
        "title":title,
                    "comment":comment,
                    "time":time1}
        except:
            pass
        finally:
            df = pd.DataFrame(data)
            df.to_excel("jianshu.xlsx",index=False,na_rep="null")
    if __name__ == "__main__": 
        get_info()
        sleep(uniform(1,2))

结果如下：  
| ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70]

[resize_m_lfit_w_962_pic_center]: /images/20221120/29ac045b8e3f4b4b9d660c2d3085cbc1.png
[selenium]: https://blog.csdn.net/qq_45176548/article/details/111404630
[selenium 1]: https://blog.csdn.net/qq_45176548/article/details/111160138
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70]: /images/20221120/f24dba8a724340c2a055d9b568793742.png