spring boot 集成gzip和zip数据压缩传输(适用大数据信息传输)

水深无声 2023-10-01 08:06 16阅读 0赞

## 1、背景 ##

在查询数据库信息的时候，由于数据库信息返回数据条数较多，数据从服务器端传至客户端耗费大量时间，导致查询数据变慢。

## 2、方案思路 ##

1）、从查询sql上入手，进行sql优化；

2）、从业务层面优化，复杂接口拆分成多个接口，避免大量数据堆积返回（视业务需求而定）；

3）、对返回的大数据信息进行数据压缩。（本文要点）

![watermark_type_d3F5LXplbmhlaQ_shadow_50_text_Q1NETiBA6ZqP6aOO5Li26aOY_size_20_color_FFFFFF_t_70_g_se_x_16][]

## 3、压缩数据方案 ##

1）、gzip压缩

2）、zip压缩

## 4、具体实现 ##

### **（1）、gzip压缩方案** ###

GzipUtils工具类

package com.自己的包.util;
    
    import org.springframework.stereotype.Component;
    
    import java.io.ByteArrayInputStream;
    import java.io.ByteArrayOutputStream;
    import java.io.IOException;
    import java.nio.charset.StandardCharsets;
    import java.util.zip.GZIPInputStream;
    import java.util.zip.GZIPOutputStream;
    
    /**
     * @program: tool_java
     * @description:
     * @author: sfp
     * @create: 2021-11-30 14:33
     **/
    @Component
    public class GzipUtils {
        /**
         * 压缩
         *
         * @param data 数据流
         * @return 压缩数据流
         * @throws IOException 异常
         */
        public byte[] compress(byte[] data) throws IOException {
            if (data == null || data.length == 0) {
                return null;
            }
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            GZIPOutputStream gzip = new GZIPOutputStream(out);
            gzip.write(data);
            gzip.close();
            return out.toByteArray();
        }
    
        /**
         * 压缩
         *
         * @param str 需要压缩数据信息
         * @return 压缩数据流
         * @throws IOException 异常
         */
        public byte[] compress(String str) throws IOException {
            if (str == null || str.length() == 0) {
                return null;
            }
            return compress(str.getBytes(StandardCharsets.UTF_8));
        }
    
        /**
         * 解压
         *
         * @param data 欲解压数据流
         * @return 原数据流
         * @throws IOException 异常
         */
        public byte[] uncompress(byte[] data) throws IOException {
            if (data == null || data.length == 0) {
                return data;
            }
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            ByteArrayInputStream in = new ByteArrayInputStream(data);
            GZIPInputStream gunzip = new GZIPInputStream(in);
            byte[] buffer = new byte[1024];
            int n;
            while ((n = gunzip.read(buffer)) >= 0) {
                out.write(buffer, 0, n);
            }
            gunzip.close();
            in.close();
            return out.toByteArray();
        }
    
        /**
         * 解压
         *
         * @param str 欲解压数据字符串
         * @return 原数据
         * @throws IOException 异常
         */
        public String uncompress(String str) throws IOException {
            if (str == null || str.length() == 0) {
                return str;
            }
            byte[] data = uncompress(str.getBytes(StandardCharsets.ISO_8859_1));
            return new String(data);
        }
    }

**数据压缩**

@Autowired
        private GzipUtils gzipUtils;
    
        @RequestMapping(value = "testGzip", method = RequestMethod.POST)
        public JSONBeansResponse testGzip(@RequestBody Map<String, String> map) throws IOException {
            if (null != map) {
                String sqlStr = map.get("paramStr");
                // 调用数据库获取数据
                Map<String, Object> resMap = testMapper.findInfo(sqlStr);
                String dataStr = JSONObject.toJSONString(resMap);
    
                // 开始压缩数据
                byte[] compress1 = gzipUtils.compress(dataStr);
                String FileBuf = Base64.getEncoder().encodeToString(compress1);
                return new JSONBeansResponse<>(FileBuf);
            }
            return new JSONBeansResponse<>(new ArrayList<>(0));
        }

**数据解压**

@RequestMapping(value = "testUnGzip", method = RequestMethod.POST)
        public JSONBeansResponse testUnGzip(@RequestBody Map<String, String> map) throws IOException {
            if (null != map) {
                String dataStream = map.get("dataStream ");
                byte[] decode = Base64.getDecoder().decode(dataStream);
                byte[] compress1 = gzipUtils.uncompress(decode);
                String dataStr = new String(compress1);
                Map<String, Object> res = JSONObject.parseObject(dataStr, Map.class);
                return new JSONBeansResponse<>(res);
            }
            return new JSONBeansResponse<>(new ArrayList<>(0));
        }

**遇到问题**

解压时候报错：**java.util.zip.ZipException: Not in GZIP format**

解决方案：**在转换为字符串时，一定要使用ISO-8859-1这样的单字节编码**

![watermark_type_d3F5LXplbmhlaQ_shadow_50_text_Q1NETiBA6ZqP6aOO5Li26aOY_size_20_color_FFFFFF_t_70_g_se_x_16 1][]

### **（2）、zip压缩方案** ###

ZipUtils工具类

package com.自己的包.util;
    
    import org.springframework.stereotype.Component;
    
    import java.io.ByteArrayInputStream;
    import java.io.ByteArrayOutputStream;
    import java.io.IOException;
    import java.nio.charset.StandardCharsets;
    import java.util.Base64;
    import java.util.zip.ZipEntry;
    import java.util.zip.ZipInputStream;
    import java.util.zip.ZipOutputStream;
    
    /**
     * @program: tool_java
     * @description: zip压缩工具
     * @author: sfp
     * @create: 2021-12-01 14:11
     **/
    @Component
    public class ZipUtils {
    /** 压缩
         * @param data  原数据流
         * @return 压缩后的数据流
         * @throws IOException 异常
         */
        public byte[] compress(byte[] data) throws IOException {
            if (data == null || data.length == 0) {
                return null;
            }
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            ZipOutputStream gzip = new ZipOutputStream(out);
            gzip.putNextEntry(new ZipEntry("json"));
            gzip.write(data);
            gzip.close();
            return out.toByteArray();
        }
    
        /** 压缩
         * @param str  原数据字符串
         * @return 压缩后的数据流
         * @throws IOException 异常
         */
        public byte[] compress(String str) throws IOException {
            if (str == null || str.length() == 0) {
                return null;
            }
            return compress(str.getBytes(StandardCharsets.UTF_8));
        }
    
        /** 解压缩
         * @param data  压缩后的数据流
         * @return 原数据的数据流
         * @throws IOException 异常
         */
        public byte[] uncompress(byte[] data) throws IOException {
            if (data == null || data.length == 0) {
                return data;
            }
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            ByteArrayInputStream in = new ByteArrayInputStream(data);
            ZipInputStream gunzip = new ZipInputStream(in);
            ZipEntry nextEntry = gunzip.getNextEntry();
            while (nextEntry != null) {
                final String fileName = nextEntry.getName();
                if (nextEntry.isDirectory()) {
                    nextEntry = gunzip.getNextEntry();
                } else if (fileName.equals("json")) {
                    byte[] buffer = new byte[1024];
                    int n;
                    while ((n = gunzip.read(buffer)) >= 0) {
                        out.write(buffer, 0, n);
                    }
                    gunzip.close();
                    in.close();
                    return out.toByteArray();
                }
            }
    
            return out.toByteArray();
        }
    
        /** 解压
         * @param str  压缩后的base64流
         * @return 原数据字符串
         * @throws IOException 异常
         */
        public String uncompress(String str) throws IOException {
            if (str == null || str.length() == 0) {
                return str;
            }
            byte[] data = uncompress(Base64.getDecoder().decode(str));
            return new String(data);
        }
    
    }

**zip使用**

@Autowired
        private ZipUtils zipUtils;
    
        @RequestMapping(value = "testzip", method = RequestMethod.POST)
        public JSONBeansResponse testzip(@RequestBody Map<String, String> map) throws IOException {
            String sqlStr = map.get("paramStr");
            List<Map<String, Object>> resMap = testMapper.findInfo(sqlStr);;
            String dataStr = JSONObject.toJSONString(resMap);
            
            // 开始压缩数据
            byte[] compress1 = zipUtils.compress(dataStr);
            String FileBuf = Base64.getEncoder().encodeToString(compress1);
            
            // 开始解压数据
            String s = zipUtils.uncompress(FileBuf);
            List<Map> arrayLists = JSONObject.parseArray(s, Map.class);
            return new JSONBeansResponse<>(arrayLists);
        }

## 5、总结 ##

在大数据量的传输中，压缩数据后进行传输可以一定程度的解决速度问题。

zip和gzip的压缩率测试过几次大概在5-6倍大小左右。

## 6、收工图 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1MjczMDA5_size_16_color_FFFFFF_t_70][]

[watermark_type_d3F5LXplbmhlaQ_shadow_50_text_Q1NETiBA6ZqP6aOO5Li26aOY_size_20_color_FFFFFF_t_70_g_se_x_16]: https://img-blog.csdnimg.cn/0997f0b5a8694e1a933c8769c3cc7557.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA6ZqP6aOO5Li26aOY,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_d3F5LXplbmhlaQ_shadow_50_text_Q1NETiBA6ZqP6aOO5Li26aOY_size_20_color_FFFFFF_t_70_g_se_x_16 1]: https://img-blog.csdnimg.cn/ac4d9a53fcef46f69314638878d9d766.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA6ZqP6aOO5Li26aOY,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1MjczMDA5_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20210709171031927.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1MjczMDA5,size_16,color_FFFFFF,t_70