实战项目一、安居客(北京) 二手房抓取房源信息

本是古典何须时尚 2022-03-07 11:42 241阅读 0赞

一、首先明确爬取的数据为安居客（北京）的二手房源的数据信息，主要有房源链接地址，房源价格，房源单价，房源规模，房源大小，房源建造年份，房源地址。

[https://beijing.anjuke.com/sale/p1/\#filtersort ][https_beijing.anjuke.com_sale_p1_filtersort]

二、分析网页和确定储存方式

首先需要做的是确定翻页规则

第一页 发现地址为

![20190316215838983.png][]

第二页 发现地址为

![20190316215731839.png][]

第三页 发现地址为

![20190316215808958.png][]

细心的同学可能已经看出来了，变化的只有最后面图中高亮部分，也就是说，我们第一页就是p1，第二页就是p2，第三页就是p3了

其次我们需要明确解析数据所使用的方法，通过抓包我们可以发现响应是html格式的，所以小编这里采用的是lxml解析，其中每个数据的xpath解析式为

.xpath('//div[@class="house-title"]/a/@href')  # 房源链接
    .xpath('//div[@class="house-title"]/a/@title')  # 房源名称
    .xpath('//span[@class="price-det"]/strong/text()')  # 房源价格
    .xpath('//span[@class="unit-price"]/text()')  # 房源单价
    .xpath('//div[@class="house-details"]/div[2]/span[1]/text()')  # 房源规模 
    .xpath('//div[@class="house-details"]/div[2]/span[2]/text()')  # 房源大小
    .xpath('//div[@class="house-details"]/div[2]/span[4]/text()')  # 建筑年份
    .xpath('//div[@class="house-details"]/div[2]/span[5]/text()')  # 联系人
    .xpath('//div[@class="housedetails"]/div[3]/span[1]/@title'))  # 房源地址

最后，小编思考了一下数据存储的方式，这里小编采用了一个字典存取这个房源的所有信息，当然最好使用房源链接作为key，因为这个是不可能重复的，而且当我们翻页的时候,还能保证不会抓取到重复数据，其中value就作为房源的一些基本信息(类型也是字典，key为每个信息的名称，value为具体的值),最后每个房源的信息，存到一个列表中

所以基本确定最后的储存格式为

单个房源信息（字典）=\{‘房屋名称’：‘名称xx’，‘房屋价格’：‘价格xx’，‘房屋单价’：单价xx‘’，‘房屋规模’：‘规模xx’，‘房屋大小’：‘大小xxx’.................\}

单个房源信息（字典）=\{‘房源链接’:房源信息''\}

所有房源信息（列表）=\[\{房屋1\},\{房屋2\},\{房屋3\}\]

三、代码部分

小编这里还格外写了一个方法，将抓取的数据，存到excel中了，这里是使用了pandas模块,这里有个比较容易进坑的点

以你为通过xpath提取出来的 类型是<class 'lxml.etree.\_ElementUnicodeResult'>，他不是字符串，所以最后我们写入excel的时候，需要转换成str才能写入。

# author:xjp
    # email:1163739580@qq.com
    # filename:Anjuke.py
    # project:python_code_warehouse
    # datetime:2019/3/15 21:19
    # software: PyCharm
    # python_version:3.6
    # message:北京安居客获取 房源链接(唯一，作为key),房源的名称，价格，几房几厅 ，大小，建造年份，联系人，地址，网页地址为https://beijing.anjuke.com/sale/p1/#filtersort
    import requests
    from lxml import etree
    import time
    import pandas as pd
    import os
    from requests.packages.urllib3.exceptions import InsecureRequestWarning
    
    requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
    
    
    class Anjuke(object):
    
        def __init__(self):
            self.houses_info = {}  # 所有房源信息 房源地址作为key 房源信息作为value
            self.start_page = 1  # 开始页码
            self.end_page = 5  # 结束页码
            self.page = 1  # 当前页码
            self.base_url = "https://beijing.anjuke.com"
            self.r = requests.session()
            self.header = {
                'User-Agent': 'Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/72.0.3626.96Safari/537.36', }
    
        def get_houses_info(self):
            while True:
                get_house_info_url = self.base_url + "/sale/p{}/#filtersort".format(self.page)
                print("正在爬取第%d页 地址为%s " % (self.page, get_house_info_url))
                house_html = self.r.get(url=get_house_info_url, verify=False, headers=self.header).text
                house_url = etree.HTML(house_html).xpath('//div[@class="house-title"]/a/@href')  # 房源链接
                house_name = etree.HTML(house_html).xpath('//div[@class="house-title"]/a/@title')  # 房源名称
                house_price = etree.HTML(house_html).xpath('//span[@class="price-det"]/strong/text()')  # 房源价格
                house_unit_price = etree.HTML(house_html).xpath('//span[@class="unit-price"]/text()')  # 房源单价
                house_scale = etree.HTML(house_html).xpath(
                    '//div[@class="house-details"]/div[2]/span[1]/text()')  # 房源规模 及几室几厅
                house_size = etree.HTML(house_html).xpath('//div[@class="house-details"]/div[2]/span[2]/text()')  # 房源大小
                house_year_built = etree.HTML(house_html).xpath(
                    '//div[@class="house-details"]/div[2]/span[4]/text()')  # 建筑年份
                house_contact_people = etree.HTML(house_html).xpath(
                    '//div[@class="house-details"]/div[2]/span[5]/text()')  # 联系人
                house_address = self.remove_blank_space(
                    *etree.HTML(house_html).xpath('//div[@class="house-details"]/div[3]/span[1]/@title'))  # 房源地址
                # 将信息存入
                house_info = {}
                for i in range(len(house_url)):
                    house_info["house_name"] = str(house_name[i])
                    house_info["house_price"] = str(house_price[i])
                    house_info["house_unit_price"] = str(house_unit_price[i])
                    house_info["house_scale"] = str(house_scale[i])
                    house_info["house_size"] = str(house_size[i])
                    house_info["house_year_built"] = str(house_year_built[i])
                    house_info["house_contact_people"] = str(house_contact_people[i])
                    house_info["house_address"] = str(house_address[i])
                    self.houses_info[str(house_url[i])] = house_info  # 自动去重
                    house_info = {}  # !!!每次需要清空，否则会导致会被覆盖
                if self.page == self.end_page:
                    break
                self.page += 1
                time.sleep(2)
    
        @staticmethod
        def remove_blank_space(*ak):
            """将序列里面的每个值，去除空格
    
            :param ak: 接受的序列
            :return:序列
            """
            sequence = []
            for a in ak:
                sequence.append("".join(str(a).split()))
            return sequence
    
        @staticmethod
        def generating_excel_data(**houses_info):
            """将房源信息生成excel
    
            """
            houses_info_excel = {}  # key为excel的列名
            house_name_list = []
            house_price_list = []
            house_unit_price_list = []
            house_scale_list = []
            house_size_list = []
            house_year_built_list = []
            house_contact_people_list = []
            house_address_list = []
            house_url_list = []
            for house_url in houses_info:
                house_name_list.append(housess_info[house_url]["house_name"])
                house_price_list.append(houses_info[house_url]["house_price"])
                house_unit_price_list.append(houses_info[house_url]["house_unit_price"])
                house_scale_list.append(houses_info[house_url]["house_scale"])
                house_size_list.append(houses_info[house_url]["house_size"])
                house_year_built_list.append(houses_info[house_url]["house_year_built"])
                house_contact_people_list.append(houses_info[house_url]["house_contact_people"])
                house_address_list.append(houses_info[house_url]["house_address"])
                house_url_list.append(house_url)
            houses_info_excel["房源名称"] = house_name_list
            houses_info_excel["价格(万元)"] = house_price_list
            houses_info_excel["均价(元)"] = house_unit_price_list
            houses_info_excel["房屋规模"] = house_scale_list
            houses_info_excel["房屋大小"] = house_size_list
            houses_info_excel["房屋建立时间"] = house_year_built_list
            houses_info_excel["房东"] = house_contact_people_list
            houses_info_excel["房屋地址"] = house_address_list
            houses_info_excel["房屋链接"] = house_url_list
            file_path = r'北京安居客二手房屋信息.xlsx'  # 文件需要保存路径
            writer = pd.ExcelWriter(file_path)
            df = pd.DataFrame(houses_info_excel)
            df.to_excel(writer, columns=houses_info_excel.keys(), index=False, encoding='utf-8',
                        sheet_name='北京二手房屋信息')
            writer.save()
            print("合计：%s条房屋信息写入excel完成 生成路径为%s" % (len(houses_info), os.getcwd() + r"/" + file_path))
    
    
    if __name__ == "__main__":
        xiejiangpeng = Anjuke()
        xiejiangpeng.get_houses_info()
        housess_info = xiejiangpeng.houses_info
        xiejiangpeng.generating_excel_data(**housess_info)

四、运行截图以及结果图展示

源网页展示

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70][]

程序运行展示

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 1][]

生成的excel展示

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 2][]

[https_beijing.anjuke.com_sale_p1_filtersort]: https://beijing.anjuke.com/sale/p1/#filtersort%C2%A0
[20190316215838983.png]: /images/20220307/1a69c75ce22e404996c6736c71854b74.png
[20190316215731839.png]: /images/20220307/503a31ac5c6f46cd925e43ae8bba4564.png
[20190316215808958.png]: /images/20220307/e2179dd3341f422f9a02d09128d6f2e7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70]: /images/20220307/09d250d07888474595c21e91c3ad1799.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 1]: /images/20220307/17007e6e617843338f9223783a131caf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1Rlc3Rlcl94anA_size_16_color_FFFFFF_t_70 2]: /images/20220307/4961c0daa1f141f88abd98929970c6c6.png