C语言fp32转为fp16的代码,FP32转FP16能否加速libtorch调用

心已赠人 2022-10-14 15:45 441阅读 0赞

FP32转FP16能否加速libtorch调用

pytorchlibtorchFP16

\#\#\#1. PYTORCH 采用FP16后的速度提升问题

pytorch可以使用half()函数将模型由FP32迅速简洁的转换成FP16.但FP16速度是否提升还依赖于GPU。以下面的代码为例，

import time

import torch

from torch.autograd import Variable

import torchvision.models as models

import torch.backends.cudnn as cudnn

cudnn.benchmark = True

net = models.resnet18().cuda()

inp = torch.randn(64, 3, 224, 224).cuda()

for i in range(5):

net.zero\_grad()

out = net.forward(Variable(inp, requires\_grad=True))

loss = out.sum()

loss.backward()

torch.cuda.synchronize()

start=time.time()

for i in range(100):

net.zero\_grad()

out = net.forward(Variable(inp, requires\_grad=True))

loss = out.sum()

loss.backward()

torch.cuda.synchronize()

end=time.time()

print("FP32 Iterations per second: ", 100/(end-start))

net = models.resnet18().cuda().half()

inp = torch.randn(64, 3, 224, 224).cuda().half()

torch.cuda.synchronize()

start=time.time()

for i in range(100):

net.zero\_grad()

out = net.forward(Variable(inp, requires\_grad=True))

loss = out.sum()

loss.backward()

torch.cuda.synchronize()

end=time.time()

print("FP16 Iterations per second: ", 100/(end-start))

在1080Ti上的性能对比：

FP32 Iterations per second: 10.37743206218922

FP16 Iterations per second: 9.855269155760238

FP32 Memory：2497M

FP16 Memory：1611M

可以发现FP16显著的降低了显存，但是速度没有提升，反而有些许下降。

然后观察在 V100 上的性能对比：

FP32 Iterations per second: 16.325794715481173

FP16 Iterations per second: 24.853492643300903

FP32 Memory: 3202M

FP16 Memory: 2272M

\#\#\#2. Libtorch采用FP16后的速度提升问题

我们在V100上测试FP16是否能提升libtorch的推理速度。

\#\#\#\#2.1 下载libtorch

wget https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.6.0%2Bcu101.zip

unzip libtorch-cxx11-abi-shared-with-deps-1.6.0+cu101.zip

在pytorch官网找到对应版本的libtorch，libtorch一般会向下支持，我这里的libtorch版本1.6.0， pytorch安装的是1.1.0

import torch

import torchvision.models as models

net = models.resnet18().cuda()

net.eval()

inp = torch.randn(64, 3, 224, 224).cuda()

traced\_script\_module = torch.jit.trace(net, inp)

traced\_script\_module.save("RESNET18\_trace.pt")

print("trace has been saved!")

\#\#\#\#2.3 libtorch 调用trace

\#include

using namespace std;

int main()

at::globalContext().setBenchmarkCuDNN(true);

std::string model\_file = "/home/zwzhou/Code/test\_libtorch/RESNET18\_trace.pt";

torch::Tensor inputs = torch::rand(\{64, 3, 224, 224\}).to(at::kCUDA);

torch::jit::script::Module net = torch::jit::load(model\_file); // load model

net.to(at::kCUDA);

auto outputs = net.forward(\{inputs\});

cudaDeviceSynchronize();

auto before = std::chrono::system\_clock::now();

for (int i=0; i<100; ++i)

outputs = net.forward(\{inputs\});

cudaDeviceSynchronize();

auto after = std::chrono::system\_clock::now();

std::chrono::duration all\_time = after - before;

std::cout<

net.to(torch::kHalf);

cudaDeviceSynchronize();

before = std::chrono::system\_clock::now();

for (int i=0; i<100; ++i)

outputs = net.forward(\{inputs.to(torch::kHalf)\});

cudaDeviceSynchronize();

after = std::chrono::system\_clock::now();

std::chrono::duration all\_time2 = after - before;

std::cout<

\#\#\#\#2.4 编写CMakeLists.txt

cmake\_minimum\_required(VERSION 3.0 FATAL\_ERROR)

project(FP\_TEST)

set(CMAKE\_PREFIX\_PATH "/home/zwzhou/packages/libtorch/share/cmake/Torch")

set(DCMAKE\_PREFIX\_PATH /home/zwzhou/packages/libtorch)

find\_package(Torch REQUIRED)

add\_executable(mtest ./libtorch\_test.cpp)

target\_link\_libraries(mtest $\{TORCH\_LIBRARIES\})

set\_property(TARGET mtest PROPERTY CXX\_STANDARD 14)

\#\#\#\#2.5 测评时间

cd build

cmake ..

make

./mtest

\#\#\#\#2.6 输出时间

FP32 iteration per second: 60.6978

FP16 iteration per second: 91.5507

可以发现，libtorch版本比pytorch版本速度提升比较明显；另外，可以看出在V100上FP16同样能够提升libtorch的推理速度。

C语言fp32转为fp16的代码,FP32转FP16能否加速libtorch调用

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关 fp16训练（混合精度训练）

相关 javascript fp demo

相关 FP side-effects

相关 FP-Growth算法Python实现（FP增长算法python实现）

相关 C语言fp32转为fp16的代码,FP32转FP16能否加速libtorch调用

相关 MXNet半精度（FP16）

相关 Apriori算法改进--FP-Tree(FP-Growth)算法

相关 FP-growth算法

相关 27. 计算FPS

相关 FPS检测