0304网页爬虫-使用lxml+xpath爬取丁香园回复内容

柔光的暖阳◎ 2022-03-12 01:53 155阅读 0赞

1、XPath  
XPath的选择功能十分强大，它提供了非常简洁明了的路径选择表达式。另外，它还提供了超过100个内建函数，用于字符串、数值、时间的匹配以及节点、序列的处理等。几乎所有我们想要定位的节点，都可以用XPath来选择。  
2、XPath常用规则  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0d1aWFiYmV5_size_16_color_FFFFFF_t_70]  
3、实战

from lxml import etree
    import requests
    
    def get_html(url, headers):
        response = requests.get(url, headers = headers)
        try:
            if response.status_code == 200:
                return response.text
        except:
            pass
    
    def  get_parse(html):
        tree = etree.HTML(html)
        user = tree.xpath('//*/table/tbody/tr/td/div/a/text()')
        reply = tree.xpath('//*/table/tbody/tr/td/div/div/table/tbody/tr/td/text()')
    
        for users, replys in zip(user, reply):
            print('用户名:'+users, '回复内容：'+ replys.strip())
    
    
    def main():
        url = 'http://www.dxy.cn/bbs/thread/626626'
        headers = {
        
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'
                   }
        html = get_html(url, headers)
        get_parse(html)
    
    if __name__ == '__main__':
        main()

输出结果：  
![在这里插入图片描述][20190304204301981.png]  
参考资料：  
[https://cuiqingcai.com/5545.html][https_cuiqingcai.com_5545.html]  
[https://blog.csdn.net/gaifuxi9518/article/details/80859116][https_blog.csdn.net_gaifuxi9518_article_details_80859116]  
[https://blog.csdn.net/asdfqwersdv3we/article/details/88124401][https_blog.csdn.net_asdfqwersdv3we_article_details_88124401]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0d1aWFiYmV5_size_16_color_FFFFFF_t_70]: /images/20220312/78df37918cf84a2cabd009921ac47c19.png
[20190304204301981.png]: /images/20220312/0c953543526d49edb3c6dca2ee5b2748.png
[https_cuiqingcai.com_5545.html]: https://cuiqingcai.com/5545.html
[https_blog.csdn.net_gaifuxi9518_article_details_80859116]: https://blog.csdn.net/gaifuxi9518/article/details/80859116
[https_blog.csdn.net_asdfqwersdv3we_article_details_88124401]: https://blog.csdn.net/asdfqwersdv3we/article/details/88124401