python 爬虫抓取19楼租房信息

以你之姓@ 2022-07-24 11:16 241阅读 0赞

## 查看19lou.com的Cookie ##

chrome中打开19lou.com，按F12可以打开开发者工具查看  
![这里写图片描述][20160521164932189]  
***不获取Cookie会导致爬取网站时重定向而抓不到内容***

## 定义headers ##

headers = {
       'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1',
           'Referer':'http://www.19lou.com',
           'Cookie':"your cookie"}

## 请求页面 ##

我们请求的url为：[http://www.19lou.com/thread/category/structure/search/result?mf\_1831\_1=4&mf\_1831\_2=3&mf\_55=2&mf\_55\_field=18&mf\_low\_65=2001&mf\_high\_65=2500&mf\_68=0&mf\_62=0&mf\_Q=%E5%92%8C%E7%9D%A6%E6%96%B0%E6%9D%91&fid=1637&m=10001&page=1][http_www.19lou.com_thread_category_structure_search_result_mf_1831_1_4_mf_1831_2_3_mf_55_2_mf_55_field_18_mf_low_65_2001_mf_high_65_2500_mf_68_0_mf_62_0_mf_Q_E5_92_8C_E7_9D_A6_E6_96_B0_E6_9D_91_fid_1637_m_10001_page_1]  
***page=1为第一页***

try:
        request = urllib2.Request(url,headers=headers)
        response = urllib2.urlopen(request)
    except:
        print 'server connect failed'
        raw_input('Press enter key to exit')
        exit()

## BeautifulSoup解析网页 ##

### 整个网页结构 ###

![这里写图片描述][20160521170905526]

### 首先用开发者工具找出需要获取到的元素位置 ###

![这里写图片描述][20160521170741618]

### 用BeautifulSoup解析 ###

html = response.read()
    soup = BeautifulSoup(html,"html.parser",fromEncoding="gb18030")
    for child in soup.table.find_all('a'):
        url = child.get('href')
        pass

***循环 `<a>` 标签获取所有子页面的url，然后请求详情页获取租房信息和图片***

## 详情页解析 ##

for child in soup.table.find_all('a'):
        url = child.get('href')
        try:
            request = urllib2.Request(url,headers=headers)
            response = urllib2.urlopen(request)
        except:
            print 'server conect failed'
            raw_input('Press enter key to exit')
            exit()

### 通过开发者工具查看详情页结构，抓取相关信息 ###

html = response.read()
    soup = BeautifulSoup(html,"html.parser",fromEncoding="gb18030")
    ul = soup.find('ul',{
       'id':'slide-data'})
    tr = soup.find('table',{
       'class':'view-table link0'}).find_all('tr')
    hx = tr[2].td.get_text()  #获取 户型 信息
    #循环获取房屋图片地址
    for row in ul.find_all('li'):
        image = row.find('a',{
       'class':'J_login nail'}).img.get('src')
        image_big = row.find('p').img.get('src')

***以上为部分讲解，下面是代码链接***  
[抓取到的信息][Link 1]  
[github完整代码][github]

[20160521164932189]: /images/20220722/c3450a9009a54ee29685248dffa15ee7.png
[http_www.19lou.com_thread_category_structure_search_result_mf_1831_1_4_mf_1831_2_3_mf_55_2_mf_55_field_18_mf_low_65_2001_mf_high_65_2500_mf_68_0_mf_62_0_mf_Q_E5_92_8C_E7_9D_A6_E6_96_B0_E6_9D_91_fid_1637_m_10001_page_1]: http://www.19lou.com/thread/category/structure/search/result?mf_1831_1=4&mf_1831_2=3&mf_55=2&mf_55_field=18&mf_low_65=2001&mf_high_65=2500&mf_68=0&mf_62=0&mf_Q=%E5%92%8C%E7%9D%A6%E6%96%B0%E6%9D%91&fid=1637&m=10001&page=1
[20160521170905526]: /images/20220722/d5d9f1bcc97341fa8747fa473d289ff3.png
[20160521170741618]: /images/20220722/a20d17da859442149f3e80182375dd2c.png
[Link 1]: http://hayson.top/zufang.html
[github]: https://github.com/JasonHugh/py_crawler/blob/master/zufang.py