java-nio-缓冲区ByteBuffer、Netty的ByteBuf和MappedByteBuffer的使用

叁歲伎倆 2022-03-06 09:06 450阅读 0赞

# java-nio-缓冲区ByteBuffer、Netty的ByteBuf和MappedByteBuffer的使用 #

### 文章目录 ###

*  java-nio-缓冲区ByteBuffer、Netty的ByteBuf和MappedByteBuffer的使用
 *   *   *  1.ByteBuffer
         *  2.MappedByteBuffer
         *  3 Netty的ByteBuf
         *  4.测试

**环境**

> idea2018，jdk1.8

记录一下缓冲区ByteBuffer、Netty的ByteBuf和MappedByteBuffer的使用笔记，参考文献：

ByteBuffer:《Java NIO》

Netty的ByteBuf:[https://www.cnblogs.com/duanxz/p/3724448.html][https_www.cnblogs.com_duanxz_p_3724448.html]

MappedByteBuffer:[https://www.cnblogs.com/ironPhoenix/p/4204472.html][https_www.cnblogs.com_ironPhoenix_p_4204472.html]

### 1.ByteBuffer ###

> 1）ByteBuffer
> 
> mark的作用：
> 
> 为某一读过的位置做标记，便于某些时候回退到该位置。
> 
> 2）ByteBuffer directBuffer=ByteBuffer.allocateDirect(10);
> 
> 直接内存的读写操作比普通Buffer快，但它的创建、销毁比普Buffer慢。因此直接内存使用于需要大内存空间且频繁访问的场合，不适用于频繁申请释放内存的场合

ByteBuffer使用示例：

public static void main(String[] args) throws Exception {
    		System.out.println("----------Test allocate--------");
    		//堆内内存是由JVM进程内存
    		ByteBuffer buffer = ByteBuffer.allocate(10240000);
    		System.out.println("buffer = " + buffer);
    
    		// 这部分直接用的系统内存，所以对JVM的内存没有影响
    		ByteBuffer directBuffer = ByteBuffer.allocateDirect(10240000);
    		System.out.println("directBuffer = " + directBuffer);
    
    		System.out.println("----------Test wrap--------");
    		byte[] bytes = new byte[32];
    		buffer = ByteBuffer.wrap(bytes);
    		System.out.println(buffer);
    		buffer = ByteBuffer.wrap(bytes, 10, 22);
    		System.out.println(buffer);
    	}

/**
     * @Description: ByteBuffer的操作:
     * 1)NIO ByteBuffer只有一个标识位置的指针position，读写的时候需要手工调用flip()和rewind()等方法
     * 2)直接内存的读写操作比普通Buffer快，但它的创建、销毁比普通Buffer慢。
     * 因此直接内存使用于需要大内存空间且频繁访问的场合，不适用于频繁申请释放内存的场合。
     */
    public class TestByteBufferMethod {
        public static ByteBuffer getByteBuffer(String str) {
            return ByteBuffer.wrap(str.getBytes());
        }
    
        public static String getString(ByteBuffer buffer) {
            buffer.flip();
            Charset charset = null;
            CharsetDecoder decoder = null;
            CharBuffer charBuffer = null;
            try {
                charset = Charset.forName("UTF-8");
                decoder = charset.newDecoder();
                // charBuffer = decoder.decode(buffer);//用这个的话，只能输出来一次结果，第二次显示为空
                charBuffer = decoder.decode(buffer.asReadOnlyBuffer());
                return charBuffer.toString();
            } catch (Exception ex) {
                ex.printStackTrace();
            }
            return "";
        }
    
        public static void main(String[] args) throws Exception {
            System.out.println("--------Test reset----------");
            ByteBuffer buffer = ByteBuffer.allocate(32);
            //一般在把数据写入Buffer前调用
            buffer.clear();
            //设置position位置
            buffer.position(5);
            //调用mark()会将mark设为当前的position的值，以后调用reset()会将position属性设置为mark的值
            buffer.mark();
            buffer.position(10);
            System.out.println("before reset:" + buffer);
            buffer.reset();
            System.out.println("after reset:" + buffer);
    
            System.out.println("--------Test rewind--------");
            buffer.clear();
            buffer.position(10);
            buffer.limit(15);
            System.out.println("before rewind:" + buffer);
            //把position设为0，limit不变，取消mark,一般在把数据重写入Buffer前调用
            buffer.rewind();
            System.out.println("before rewind:" + buffer);
    
            System.out.println("--------Test compact--------");
            buffer.clear();
            buffer.put("abcd".getBytes());
            System.out.println("before compact:" + buffer);
            System.out.println(new String(buffer.array()));
            buffer.flip();
            System.out.println("after flip:" + buffer);
            System.out.println((char) buffer.get());
            System.out.println((char) buffer.get());
            System.out.println("after 2 gets:" + buffer);
            //compact()方法将所有未读的数据拷贝到Buffer起始处。然后将position设到最后一个未读元素正后面
            buffer.compact();
            System.out.println("after compact:" + buffer);
            System.out.println("\t" + new String(buffer.array()));
    
            System.out.println("------Test get-------------");
            buffer = ByteBuffer.allocate(32);
            String s = "123456";
            buffer.put(s.getBytes());
            System.out.println("before flip()" + buffer);
            // 转换为读取模式,flip才能从position（置为0）开始读取到limit的数据
            buffer.flip();
            System.out.println("after flip():" + buffer);
            //相对读，从position位置读取一个byte，并将position+1，为下次读写作准备
            System.out.println((char) buffer.get());
            System.out.println("after get():" + buffer);
            //绝对读，读取byteBuffer底层的bytes中下标为index的byte，不改变position
            System.out.println((char) buffer.get(2));
            System.out.println("after get(index):" + buffer);
            byte[] dst = new byte[5];
            //从position位置开始相对读，读length个byte，并写入dst下标从offset到offset+length的区域
            buffer.get(dst, 3, 1);
            System.out.println("after get(dst, 3, 2):" + buffer);
            for (int i = 0; i < dst.length; i++) {
                System.out.println("dst[i]:" + dst[i]);
            }
            System.out.println("buffer now is:" + buffer);
    
            System.out.println("--------Test put-------");
            ByteBuffer bb = ByteBuffer.allocate(32);
            System.out.println("before put(byte):" + bb);
            //相对写，向position的位置写入一个byte，并将postion+1，为下次读写作准备
            bb.put((byte) 'z');
            System.out.println("after put(byte):" + bb);
            //绝对写，向byteBuffer底层的bytes中下标为index的位置插入byte b，不改变position
            bb.put(5, (byte) 'c');
            System.out.println("after put(5,(byte) 'c'):" + bb);
            // 这里的buffer是 abcdef[pos=3 lim=6 cap=32]
            bb.put(buffer);
            System.out.println("after put(buffer):" + bb);
            byte[] bytes = new byte[4];
            ByteBuffer buffer2 = ByteBuffer.wrap(bytes);
            bb.put(bytes, 0, bytes.length);
            System.out.println("after put(bytes):" + bb);
            /
            ByteBuffer directBuffer=ByteBuffer.allocateDirect(10);
        }
    }

### 2.MappedByteBuffer ###

> MappedByteBuffer文件内存映射，常用于操作读写大文件。

public class TestMappedByteBuffer {
        public static void main(String[] args)throws Exception {
            int length = 0x00006;//一个Byte占1B
            try (FileChannel channel = FileChannel.open(Paths.get("D:\\test\\2\\TestMappedByteBuffer.txt"),
                    StandardOpenOption.READ, StandardOpenOption.WRITE);) {
                MappedByteBuffer mapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, length);
                for(int i=0;i<length;i++) {
                    mapBuffer.put((byte)'a');
                }
                System.out.println("mapBuffer:"+mapBuffer.toString());
                mapBuffer.flip();
                for(int i = 0;i<3;i++) {
                    //像数组一样访问，但是不改变position位置
                    System.out.println(mapBuffer.toString()+":"+mapBuffer.get(i));
                }
                while (mapBuffer.hasRemaining()){
                    System.out.println(mapBuffer.toString()+":"+mapBuffer.get());
                }
    
        }
    }
    }

### 3 Netty的ByteBuf ###

> Netty的 Zero-coyp 完全是在用户态(Java 层面)的,在操作数据时,
> 
>  *  不需要将数据 buffer 从一个内存区域拷贝到另一个内存区域. 少了一次内存的拷贝
>     
>     Netty 的 Zero-copy 体现在如下几个个方面:  
>     \*
>     
>      *  Netty 提供了 CompositeByteBuf 类, 它可以将多个 ByteBuf 合并为一个逻辑上的 ByteBuf, 避免了各个 ByteBuf 之间的拷贝.  
>         \*
>      *  通过 wrap 操作, 我们可以将 byte\[\] 数组、ByteBuf、ByteBuffer等包装成一个 Netty ByteBuf 对象, 进而避免了拷贝操作.  
>         \*
>      *  ByteBuf 支持 slice 操作, 因此可以将 ByteBuf 分解为多个共享同一个存储区域的 ByteBuf, 避免了内存的拷贝.  
>         \*
>      *  通过 FileRegion 包装的FileChannel.tranferTo 实现文件传输, 可以直接将文件缓冲区的数据发送到目标 Channel, 避免了传统通过循环 write 方式导致的内存拷贝问题.

public class TestByteBuf {
    
        public static String convertByteBufToString(ByteBuf buf) {
            String str;
            if (buf.hasArray()) {
                // 处理堆缓冲区
                str = new String(buf.array(), buf.arrayOffset() + buf.readerIndex(), buf.readableBytes());
            } else { // 处理直接缓冲区以及复合缓冲区
                byte[] bytes = new byte[buf.readableBytes()];
                buf.getBytes(buf.readerIndex(), bytes);
                str = new String(bytes, 0, buf.readableBytes());
            }
            return str;
        }
    
        public static ByteBuf convertStringToByteBuf(String str) {
    
            return Unpooled.wrappedBuffer("abcdef".getBytes());
        }
    
        public static void main(String[] args) throws Exception {
            //默认是创建heap buffer
            System.out.print("\n-------test heapBuf----\n");
            ByteBuf byteBuf1 = Unpooled.buffer(5);
            //写数据
            byteBuf1.writeBytes("abcdefgh".getBytes("utf-8"));
            byteBuf1.setByte(0,(byte)'1');
            //读数据
            for (int i = 0; i < byteBuf1.capacity(); i++) {
                System.out.print("\n-------test heapBuf----buf.getByte:" + (byteBuf1.getByte(i)));
            }
            System.out.printf("\n-------test heapBuf's capcity:" +byteBuf1.capacity());
            System.out.printf("\n-------test heapBuf's ----convertByteBufToString(byteBuf1):" + convertByteBufToString(byteBuf1));
            /**
             * 通过 CompositeByteBuf 实现零拷贝,添加ByteBuf到CompositeByteBuf
             * CompositeByteBuf 类, 它可以将多个 ByteBuf 合并为一个逻辑上的 ByteBuf, 避免了各个 ByteBuf 之间的拷贝
             * CompositeByteBuf和添加的ByteBuf共用存储空间
             */
            System.out.print("\n-------test CompositeByteBuf----\n");
            //组合缓冲区
            CompositeByteBuf compBuf = Unpooled.compositeBuffer();
            //堆缓冲区
            ByteBuf heapBuf = Unpooled.buffer(4);
            //直接缓冲区
            ByteBuf directBuf = Unpooled.directBuffer(4);
            heapBuf.writeBytes("a".getBytes());
            directBuf.writeBytes("ef".getBytes());
            compBuf.addComponents(true, heapBuf, directBuf);
            compBuf.writeBytes("cdgh".getBytes());
            compBuf.setByte(0,(byte)'g');
            System.out.printf("\n-------test CompositeByteBuf:" + heapBuf);
            System.out.printf("\n-------test CompositeByteBuf:" + directBuf);
            System.out.printf("\n-------test CompositeByteBuf:" + compBuf);
            System.out.printf("\n-------test CompositeByteBuf heapBuf:" + convertByteBufToString(heapBuf));
            System.out.printf("\n-------test CompositeByteBuf directBuf:" + convertByteBufToString(directBuf));
            System.out.printf("\n-------test CompositeByteBuf compBuf:" + convertByteBufToString(compBuf));
            //查看内存地址
            AddressPrint addressPrint=new AddressPrint();
            System.out.printf("\n-------test CompositeByteBuf heapBuf address:%s;directBuf address:%s;compBuf address:%s;",addressPrint.addressOf(heapBuf.getByte(0)),addressPrint.addressOf(directBuf.getByte(0)),addressPrint.addressOf(compBuf.getByte(0)));
    
            /**
             *  额外的拷贝将byte 数组拷贝到 ByteBuf
             */
            /*ByteBuf byteBuf = Unpooled.buffer();
            byteBuf.writeBytes(bytes2);*/
            /**
             * 通过 wrap 操作, 可以将 byte[] 数组、ByteBuf、ByteBuffer等包装成一个 Netty ByteBuf 对象, 进而避免了拷贝操作
             * ByteBuf 对象是和 bytes 数组共用了同一个存储空间, 对 bytes 的修改也会反映到 ByteBuf 对象中
             */
            ByteBuf byteBuf = Unpooled.wrappedBuffer("abcdef".getBytes());
            /**
             * 通过 slice 操作,slice 操作可以将一个 ByteBuf 切片 为多个共享一个存储区域的 ByteBuf 对象
             */
            ByteBuf byteBuf3 = Unpooled.buffer(16);
            byteBuf3.writeBytes("abcdef".getBytes());
            ByteBuf header = byteBuf3.slice(0, 5);
            ByteBuf body = byteBuf3.slice(5, 10);
            System.out.printf("\n-------test wrap,slice----byteBuf3:" + byteBuf3);
            System.out.printf("\n-------test wrap,slice----header:" + header);
            System.out.printf("\n-------test wrap,slice----body:" + body);
            System.out.printf("\n-------test wrap,slice----byteBuf3 address:%s;header address:%s;body address:%s;",addressPrint.addressOf(byteBuf3.getByte(0)),addressPrint.addressOf(header.getByte(0)),addressPrint.addressOf(body.getByte(0)));
    
            /**
             *
             */
            ByteBuf buf1 = Unpooled.copiedBuffer("Netty in Action test!", CharsetUtil.UTF_8);
            //复制ByteBuf 0-14位置的内容
            ByteBuf copy = buf1.copy(0, 14);
            //同样修改内容
            buf1.setByte(0, (byte) 'J');
            //复制缓冲和原始缓冲并没有共享数据
            System.out.print("\nbuf1.getByte(0)" + (char) buf1.getByte(0));
            System.out.print("\ncopy.getByte(0)" + (char) copy.getByte(0));
            netty池化的内存分配：PooledByteBufAllocator采用了jemalloc内存分配算法///
            ByteBuf poolBuf = PooledByteBufAllocator.DEFAULT.buffer(5);
            poolBuf.writeBytes("abc".getBytes());
    
        }
        public static class AddressPrint {
            private  Unsafe unsafe;
    
            {
                try
                {
                    Field field = Unsafe.class.getDeclaredField("theUnsafe");
                    field.setAccessible(true);
                    unsafe = (Unsafe)field.get(null);
                }
                catch (Exception e)
                {
                    e.printStackTrace();
                }
            }
    
            public  String addressOf(Object o)
                    throws Exception
            {
                Object[] array = new Object[] {o};
    
                long baseOffset = unsafe.arrayBaseOffset(Object[].class);
                int addressSize = unsafe.addressSize();
                long objectAddress;
                switch (addressSize)
                {
                    case 4:
                        objectAddress = unsafe.getInt(array, baseOffset);
                        break;
                    case 8:
                        objectAddress = unsafe.getLong(array, baseOffset);
                        break;
                    default:
                        throw new Error("unsupported address size: " + addressSize);
                }
    
                return(Long.toHexString(objectAddress));
            }
    
    
            public  void printBytes(long objectAddress, int num)
            {
                for (long i = 0; i < num; i++)
                {
                    int cur = unsafe.getByte(objectAddress + i);
                    System.out.print((char)cur);
                }
                System.out.println();
            }
        }
    }

### 4.测试 ###

1）CompositeByteBuf组合缓存区的首字节内存地址和添加的ByteBuf的首字节内存地址关系：

//组合缓冲区
            CompositeByteBuf compBuf = Unpooled.compositeBuffer();
            //堆缓冲区
            ByteBuf heapBuf = Unpooled.buffer(4);
            //直接缓冲区
            ByteBuf directBuf = Unpooled.directBuffer(4);
            heapBuf.writeBytes("a".getBytes());
            directBuf.writeBytes("ef".getBytes());
            compBuf.addComponents(true, heapBuf, directBuf);
            compBuf.writeBytes("cdgh".getBytes());

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3loaHloaHloaHloaA_size_16_color_FFFFFF_t_70]

2）slice后各ByteBuf的首字节内存地址关系：

ByteBuf byteBuf3 = Unpooled.buffer(16);
            byteBuf3.writeBytes("abcdef".getBytes());
            ByteBuf header = byteBuf3.slice(0, 5);
            ByteBuf body = byteBuf3.slice(5, 10);

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3loaHloaHloaHloaA_size_16_color_FFFFFF_t_70 1]

[https_www.cnblogs.com_duanxz_p_3724448.html]: https://www.cnblogs.com/duanxz/p/3724448.html
[https_www.cnblogs.com_ironPhoenix_p_4204472.html]: https://www.cnblogs.com/ironPhoenix/p/4204472.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3loaHloaHloaHloaA_size_16_color_FFFFFF_t_70]: /images/20220303/460653c9e60b492f9280bb468b16682c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3loaHloaHloaHloaA_size_16_color_FFFFFF_t_70 1]: /images/20220303/3009c47075f64e02a414d878d6628c40.png