aiohttp遇到非法字符的处理(UnicodeDecodeError: 'utf-8' codec can't decode bytes in position......)

分手后的思念是犯贱 2022-06-10 14:15 321阅读 0赞

这个问题困扰了我将近一天时间，如果使用text()函数会一直报“UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 24461-24462: invalid continuation byte”的错误，如果使用read()函数以二进制输出在后面解析的时候中文是乱码，网上查了很多资料，主要也是自己的疏忽自己看了源码，一直纠结在编码问题忽略了另一个带默认值的参数

下面是解决方案：

import aiohttp
    import asyncio
    
    headers = {
            "Upgrade-Insecure-Requests": "1",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
            "Accept-Encoding": "gzip, deflate, sdch, br",
            "Accept-Language": "zh-CN,zh;q=0.8",
        }
    
    async def ss():
        async with aiohttp.ClientSession() as session:
            async with session.get('http://www.iteye.com/blogs/tag/java',headers=headers) as resp:
                print(resp.status)
                # ignore，则会忽略非法字符，默认是strict，代表遇到非法字符时抛出异常
                d = (await resp.text("utf-8","ignore"))
                # d = await resp.read()
                # d = await resp.text()
                cc(d)
    
    def cc(v):
        print(v)
        soup = BeautifulSoup(v, "lxml")
        contents = soup.select("div.content")
        for conten in contents:
            articleAuthor = conten.select("div.blog_info > a")
            if articleAuthor:
                print(articleAuthor)
                articleAuthor = articleAuthor[0]
            else:
                articleAuthor = ""
            print(articleAuthor)
    
    loop = asyncio.get_event_loop()
    tasks = [ss() ]
    loop.run_until_complete(asyncio.gather(*tasks))

这样结果中文就正常显示("utf-8"其实不是必须的因为默认就是utf-8)：

![Center][]

如果是await resp.text()就直接报错：

![Center 1][]

如果是await resp.read()，在解析时中文乱码：

![Center 2][]

text的源码：

@asyncio.coroutine
    def text(self, encoding=None, errors='strict'):
        """Read response payload and decode."""
        if self._content is None:
            yield from self.read()
    
        if encoding is None:
            encoding = self._get_encoding()
    
        return self._content.decode(encoding, errors=errors)

默认的参数就是strict，代表遇到非法字符时抛出异常；   
如果设置为ignore，则会忽略非法字符；

[Center]: /images/20220610/7fea7f32564a4725afce5427b3b46b75.png
[Center 1]: /images/20220610/f6e49fa046f54a6e81f128ce866034a2.png
[Center 2]: /images/20220610/3dd5314c69a943c796c984a8178be577.png

aiohttp遇到非法字符的处理(UnicodeDecodeError: 'utf-8' codec can't decode bytes in position......)

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关解决UnicodeDecodeError: ‘utf-8‘ codec can‘t decode byte 0xbc in position 2: invalid start byte

相关解决UnicodeDecodeError utf-8 codec cant decode byte 0xd0 in position 3150: invalid con

相关 UnicodeDecodeError: ‘utf-8‘ codec can‘t decode byte 0xed in position 5682: invalid continuation byte

相关 UnicodeDecodeError: ‘utf-8‘ codec can‘t decode byte 0xd6 in position 19: invalid continuation byte

相关 UnicodeDecodeError: ‘utf-8‘ codec can‘t decode byte 0xc7 in position 18: invalid continuation byte

相关 UnicodeDecodeError: 'utf8' codec can't decode byte 0xb1 in position 0: invalid start byte

相关处理UnicodeDecodeError: ‘XXX' codec can't decode bytes in position...的问题

相关 aiohttp遇到非法字符的处理(UnicodeDecodeError: 'utf-8' codec can't decode bytes in position......)

相关 UnicodeDecodeError: 'utf-8' codec can't decode byte

相关 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte