TensorRT英伟达官方示例解析(一)

系列文章目录

TensorRT英伟达官方示例解析(一)
TensorRT英伟达官方示例解析(二)
TensorRT英伟达官方示例解析(三)


文章目录

  • 系列文章目录
  • 前言
  • 一、参考资料
  • 二、配置系统环境
  • 三、00-MNISTData
  • 四、01-SimpleDemo
    • 4.1 Makefile
    • 4.2 main.cpp
    • 4.3 main.py
  • 总结


前言


一、参考资料

官方示例:https://github.com/NVIDIA/trt-samples-for-hackathon-cn/tree/master
官方视频:https://www.bilibili.com/video/BV1jj411Z7wG/?spm_id_from=333.337.search-card.all.click&vd_source=ce674108fa2e19e5322d710724193487
MINST数据集下载地址:http://yann.lecun.com/exdb/mnist/

二、配置系统环境

先安装所需的依赖包,在cookbook下面

pip install -r requirements.txt

Linux–Tensorrt环境配置: https://blog.csdn.net/m0_70420861/article/details/135700049
Window–TensorRT环境配置: https://blog.csdn.net/m0_70420861/article/details/135658922?spm=1001.2014.3001.5502

1. 添加python环境变量–方法一

查看python环境变量:

echo $PYTHONPATH

如果没有找到 TensorRT 的安装路径,请将其添加到环境变量中。例如,在 Linux 或 macOS 中,您可以在命令行中执行以下命令:

export PYTHONPATH=$PYTHONPATH:/root/TensorRT-8.6.1.6

记得替换成自己TensorRT的安装路径
在这里插入图片描述

2. 添加python环境变量–方法二

要将环境变量永久添加到 ~/.bashrc 或 ~/.bash_profile 文件中可以使用以下方法(推荐)

vi ~/.bashrc

编辑.bashrc文件,在文件末尾添加export PYTHONPATH=$PYTHONPATH:/root/TensorRT-8.6.1.6

在这里插入图片描述

在终端中执行以下命令使更改生效:

source ~/.bashrc

三、00-MNISTData

loadMnistData.py

#
# Copyright (c) 2021-2023, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
## http://yann.lecun.com/exdb/mnist/, https://storage.googleapis.com/cvdf-datasets/mnist/import gzipimport cv2
import numpy as npclass MnistData():def __init__(self, dataPath, isOneHot=False, randomSeed=97):with open(dataPath + "train-images-idx3-ubyte.gz", "rb") as f:self.trainImage = self.extractImage(f)with open(dataPath + "train-labels-idx1-ubyte.gz", "rb") as f:self.trainLabel = self.extractLabel(f)with open(dataPath + "t10k-images-idx3-ubyte.gz", "rb") as f:self.testImage = self.extractImage(f)with open(dataPath + "t10k-labels-idx1-ubyte.gz", "rb") as f:self.testLabel = self.extractLabel(f, isOneHot=isOneHot)self.isOneHot = isOneHotif self.isOneHot:self.trainLabel = self.convertToOneHot(self.trainLabel)self.testLabel = self.convertToOneHot(self.testLabel)else:self.trainLabel = self.trainLabel.astype(np.float32)self.testLabel = self.testLabel.astype(np.float32)np.random.seed(randomSeed)def getBatch(self, batchSize, isTrain):if isTrain:index = np.random.choice(len(self.trainImage), batchSize, True)return self.trainImage[index], self.trainLabel[index]else:index = np.random.choice(len(self.testImage), batchSize, True)return self.testImage[index], self.testLabel[index]def read4Byte(self, byteStream):dt = np.dtype(np.uint32).newbyteorder(">")return np.frombuffer(byteStream.read(4), dtype=dt)[0]def extractImage(self, f):print("Extracting", f.name)with gzip.GzipFile(fileobj=f) as byteStream:if self.read4Byte(byteStream) != 2051:raise ValueError("Failed reading file!")nImage = self.read4Byte(byteStream)rows = self.read4Byte(byteStream)cols = self.read4Byte(byteStream)buf = byteStream.read(rows * cols * nImage)return np.frombuffer(buf, dtype=np.uint8).astype(np.float32).reshape(nImage, rows, cols, 1) / 255def extractLabel(self, f, isOneHot=False, nClass=10):print("Extracting", f.name)with gzip.GzipFile(fileobj=f) as byteStream:if self.read4Byte(byteStream) != 2049:raise ValueError("Failed reading file!")nLabel = self.read4Byte(byteStream)buf = byteStream.read(nLabel)return np.frombuffer(buf, dtype=np.uint8)def convertToOneHot(self, labelIndex, nClass=10):nLabel = labelIndex.shape[0]res = np.zeros((nLabel, nClass), dtype=np.float32)offset = np.arange(nLabel) * nClassres.flat[offset + labelIndex] = 1return resdef saveImage(self, count, outputPath, isTrain):if self.isOneHot:returnimage, label = ([self.testImage, self.testLabel], [self.trainImage, self.trainLabel])[isTrain]for i in range(min(count, 10000)):cv2.imwrite(outputPath + str(i).zfill(5) + "-" + str(label[i]) + ".jpg", (image[i] * 255).astype(np.uint8))

用于处理MNIST数据集的类MnistData。加载MNIST数据集的图像和标签,并提供获取批量数据的方法。在初始化时,会读取训练集和测试集的图像和标签文件,并将它们存储在trainImage、trainLabel、testImage和testLabel属性中。如果指定了isOneHot为True,则标签将被转换为one-hot编码形式。

  • getBatch方法用于获取指定大小的批量数据。如果isTrain为True,则从训练集中随机选择指定数量的样本;否则从测试集中选择。

  • extractImage和extractLabel方法用于解析MNIST数据集文件。它们读取文件的魔数和其他元数据,然后解析图像和标签数据。

    (对于图像数据文件(train-images-idx3-ubyte.gz和t10k-images-idx3-ubyte.gz),文件的前4个字节是魔数,用于验证文件类型。接下来的4个字节表示图像数量(nImage),接着的4个字节表示每张图像的行数(rows),再接着的4个字节表示每张图像的列数(cols)。之后的字节表示实际的图像数据。)

  • convertToOneHot方法用于将标签索引转换为one-hot编码形式。

  • saveImage方法用于将图像保存到指定路径。这个方法只在isOneHot为False时才有效,因为one-hot编码的标签无法直接保存为图像文件名。

extractMnistData.py

import sysimport loadMnistDatanTrain =int(sys.argv[1]) if len(sys.argv) > 1 and sys.argv[1].isdigit() else 3000
nTest = int(sys.argv[2]) if len(sys.argv) > 2 and sys.argv[2].isdigit() else 500mnist = loadMnistData.MnistData("./", isOneHot=False)
mnist.saveImage(nTrain, "./train/", True)  # 60000 images in total
mnist.saveImage(nTest, "./test/", False)  # 10000 images in total

导入了loadMnistData模块并定义了两个变量nTrain和nTest用于指定训练和测试样本的数量。接下来,它使用loadMnistData.MnistData类加载MNIST数据集,并将数据保存为图像文件。saveImage方法用于将指定数量的图像保存到指定目录中,总共有60000张训练图像和10000张测试图像。

首先下载MINST数据集,放到该项目根目录下
MINST数据集下载地址:http://yann.lecun.com/exdb/mnist/

python extractMnistData.py

在这里插入图片描述

在这里插入图片描述

四、01-SimpleDemo

进入SimpleDemo,文件夹里有四个文件,逐个解释
在这里插入图片描述

4.1 Makefile

Makefile文件,用于管理C++和Python代码的编译和构建。

include ../../include/Makefile.incSOURCE_CPP  = $(shell find . -name '*.cpp' 2>/dev/null)
SOURCE_PY   = $(shell find . -name '*.py' 2>/dev/null)
OBJ         = $(shell find . -name *.o 2>/dev/null)
DEP         = $(OBJ:.o=.d)
TARGET_EXE  = $(SOURCE_CPP:.cpp=.exe)-include $(DEP)all: $(TARGET_EXE)%.exe: %.o$(NVCC) $(CCFLAG) $(LDFLAG) -o $@ $+%.o: %.cpp$(NVCC) $(CCFLAG) $(INCLUDE) -M -MT $@ -o $(@:.o=.d) $<$(NVCC) $(CCFLAG) $(INCLUDE) -Xcompiler -fPIC -o $@ -c $<.PHONY: test
test:make cleanmakepython3 $(SOURCE_PY)rm -rf ./*.plan./$(TARGET_EXE).PHONY: clean
clean:rm -rf ./*.d ./*.o ./*.so ./*.exe ./*.plan
  • include …/…/include/Makefile.inc:
    包含一个名为Makefile.inc的文件,该文件位于上级目录的include子目录中。Makefile.inc通常包含一些全局配置和共享的变量和指令。

然后让我们查看一下Makefile.inc,定义了一些变量,用于指定编译和链接的相关选项。

CUDA_PATH       = /usr/local/cuda
NVCC            = $(CUDA_PATH)/bin/nvcc
TRT_INC_PATH    = /usr/include/x86_64-linux-gnu
TRT_LIB_PATH    = /usr/lib/x86_64-linux-gnu
GENCODE         = -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_89,code=sm_89
DEBUG_MACRO     = -UDEBUG
WARNING_MACRO   = -w
CUFLAG          = -std=c++14 -O3 $(DEBUG_MACRO) -Xcompiler -fPIC $(GENCODE)
CCFLAG          = -std=c++14 -O3 $(DEBUG_MACRO) -Xcompiler -fPIC -use_fast_math
SOFLAG          = -shared
INCLUDE         = -I. -I$(CUDA_PATH)/include -I$(TRT_INC_PATH)
INCLUDE        += -I../../include -I../../../include
LDFLAG          = -L$(CUDA_PATH)/lib64 -lcudart -L$(TRT_LIB_PATH) -lnvinfer

其中TRT_INC_PATH和TRT_LIB_PATH分别指定了TensorRT的头文件路径和库文件路径,可以在这里进行设置。如果安装TensorRT时使用默认路径,则上述设置应该是正确的。但是如果您安装TensorRT时使用了自定义路径,则需要相应地更改这些路径。

查询TensorRT路径

sudo find / -name 'libnvinfer.so*'

在这里插入图片描述
所以应该将TRT_INC_PATH设置为/root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/include,将TRT_LIB_PATH设置为/root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib
在这里插入图片描述
如果不确定自己的cuda安装路径是什么可以用下面的查询,一般就只要改TRT_INC_PATH、TRT_LIB_PATH和CUDA_PATH

which nvcc

在这里插入图片描述

  • SOURCE_CPP = $(shell find . -name ‘*.cpp’ 2>/dev/null):

    使用shell命令查找当前目录及其子目录下的所有.cpp文件,并将它们的路径保存在变量SOURCE_CPP中。

  • SOURCE_PY = $(shell find . -name ‘*.py’ 2>/dev/null):

    使用shell命令查找当前目录及其子目录下的所有.py文件,并将它们的路径保存在变量SOURCE_PY中。

  • OBJ = $(shell find . -name *.o 2>/dev/null):

    使用shell命令查找当前目录及其子目录下的所有.o文件,并将它们的路径保存在变量OBJ中。

  • DEP = $(OBJ:.o=.d):

    将OBJ中所有.o后缀的文件替换为.d后缀,生成依赖关系文件的路径列表,并将其保存在变量DEP中。

  • TARGET_EXE = $(SOURCE_CPP:.cpp=.exe):

    将SOURCE_CPP中所有.cpp后缀的文件替换为.exe后缀,生成可执行文件的路径列表,并将其保存在变量TARGET_EXE中。

  • -include $(DEP):
    包含所有依赖关系文件,如果某个依赖关系文件不存在,则忽略该文件而不产生错误。

  • all: $(TARGET_EXE):

    all是一个伪目标,用于指定Makefile的默认目标。在本例中,all目标会构建所有的可执行文件$(TARGET_EXE)。

  • %.exe: %.o:

    定义了一个模式规则,用于根据.o文件构建可执行文件。%表示通配符,可以匹配任何字符串,例如a.exe可以匹配a.o、b.exe可以匹配b.o等。

4.2 main.cpp

这是一个使用TensorRT API创建和执行推理引擎的示例程序

  1. 加载或创建推理引擎:
    程序首先检查是否存在名为 “model.plan” 的序列化引擎文件,如果存在则加载该文件,否则创建新的引擎。如果创建了新的引擎,则将其序列化并保存到 “model.plan” 文件中。

  2. 创建执行上下文:
    使用ICudaEngine::createExecutionContext()创建一个推理上下文对象,该对象可以用于执行推理。

  3. 设置输入数据:
    为输入缓冲区分配内存,并将数据传输到设备(GPU)上。

  4. 执行推理:
    调用IExecutionContext::enqueueV3()方法执行推理。

  5. 获取输出数据:
    从设备(GPU)上复制输出数据,并打印输出结果。

/** Copyright (c) 2021-2023, NVIDIA CORPORATION. All rights reserved.** Licensed under the Apache License, Version 2.0 (the "License");* you may not use this file except in compliance with the License.* You may obtain a copy of the License at**     http://www.apache.org/licenses/LICENSE-2.0** Unless required by applicable law or agreed to in writing, software* distributed under the License is distributed on an "AS IS" BASIS,* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.* See the License for the specific language governing permissions and* limitations under the License.*/#include "cookbookHelper.cuh"using namespace nvinfer1;const std::string trtFile {"./model.plan"};
static Logger     gLogger(ILogger::Severity::kERROR);void run()
{ICudaEngine *engine = nullptr;if (access(trtFile.c_str(), F_OK) == 0){std::ifstream engineFile(trtFile, std::ios::binary);long int      fsize = 0;engineFile.seekg(0, engineFile.end);fsize = engineFile.tellg();engineFile.seekg(0, engineFile.beg);std::vector<char> engineString(fsize);engineFile.read(engineString.data(), fsize);if (engineString.size() == 0){std::cout << "Failed getting serialized engine!" << std::endl;return;}std::cout << "Succeeded getting serialized engine!" << std::endl;IRuntime *runtime {createInferRuntime(gLogger)};engine = runtime->deserializeCudaEngine(engineString.data(), fsize);if (engine == nullptr){std::cout << "Failed loading engine!" << std::endl;return;}std::cout << "Succeeded loading engine!" << std::endl;}else{IBuilder             *builder = createInferBuilder(gLogger);INetworkDefinition   *network = builder->createNetworkV2(1U << int(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH));IOptimizationProfile *profile = builder->createOptimizationProfile();IBuilderConfig       *config  = builder->createBuilderConfig();config->setMemoryPoolLimit(MemoryPoolType::kWORKSPACE, 1 << 30);ITensor *inputTensor = network->addInput("inputT0", DataType::kFLOAT, Dims32 {3, {-1, -1, -1}});profile->setDimensions(inputTensor->getName(), OptProfileSelector::kMIN, Dims32 {3, {1, 1, 1}});profile->setDimensions(inputTensor->getName(), OptProfileSelector::kOPT, Dims32 {3, {3, 4, 5}});profile->setDimensions(inputTensor->getName(), OptProfileSelector::kMAX, Dims32 {3, {6, 8, 10}});config->addOptimizationProfile(profile);IIdentityLayer *identityLayer = network->addIdentity(*inputTensor);network->markOutput(*identityLayer->getOutput(0));IHostMemory *engineString = builder->buildSerializedNetwork(*network, *config);if (engineString == nullptr || engineString->size() == 0){std::cout << "Failed building serialized engine!" << std::endl;return;}std::cout << "Succeeded building serialized engine!" << std::endl;IRuntime *runtime {createInferRuntime(gLogger)};engine = runtime->deserializeCudaEngine(engineString->data(), engineString->size());if (engine == nullptr){std::cout << "Failed building engine!" << std::endl;return;}std::cout << "Succeeded building engine!" << std::endl;std::ofstream engineFile(trtFile, std::ios::binary);if (!engineFile){std::cout << "Failed opening file to write" << std::endl;return;}engineFile.write(static_cast<char *>(engineString->data()), engineString->size());if (engineFile.fail()){std::cout << "Failed saving .plan file!" << std::endl;return;}std::cout << "Succeeded saving .plan file!" << std::endl;}long unsigned int        nIO     = engine->getNbIOTensors();long unsigned int        nInput  = 0;long unsigned int        nOutput = 0;std::vector<std::string> vTensorName(nIO);for (int i = 0; i < nIO; ++i){vTensorName[i] = std::string(engine->getIOTensorName(i));nInput += int(engine->getTensorIOMode(vTensorName[i].c_str()) == TensorIOMode::kINPUT);nOutput += int(engine->getTensorIOMode(vTensorName[i].c_str()) == TensorIOMode::kOUTPUT);}IExecutionContext *context = engine->createExecutionContext();context->setInputShape(vTensorName[0].c_str(), Dims32 {3, {3, 4, 5}});for (int i = 0; i < nIO; ++i){std::cout << std::string(i < nInput ? "Input [" : "Output[");std::cout << i << std::string("]-> ");std::cout << dataTypeToString(engine->getTensorDataType(vTensorName[i].c_str())) << std::string(" ");std::cout << shapeToString(engine->getTensorShape(vTensorName[i].c_str())) << std::string(" ");std::cout << shapeToString(context->getTensorShape(vTensorName[i].c_str())) << std::string(" ");std::cout << vTensorName[i] << std::endl;}std::vector<int> vTensorSize(nIO, 0);for (int i = 0; i < nIO; ++i){Dims32 dim  = context->getTensorShape(vTensorName[i].c_str());int    size = 1;for (int j = 0; j < dim.nbDims; ++j){size *= dim.d[j];}vTensorSize[i] = size * dataTypeToSize(engine->getTensorDataType(vTensorName[i].c_str()));}std::vector<void *>vBufferH {nIO, nullptr};std::vector<void *> vBufferD {nIO, nullptr};for (int i = 0; i < nIO; ++i){vBufferH[i] = (void *)new char[vTensorSize[i]];CHECK(cudaMalloc(&vBufferD[i], vTensorSize[i]));}float *pData = (float *)vBufferH[0];for (int i = 0; i < vTensorSize[0] / dataTypeToSize(engine->getTensorDataType(vTensorName[0].c_str())); ++i){pData[i] = float(i);}for (int i = 0; i < nInput; ++i){CHECK(cudaMemcpy(vBufferD[i], vBufferH[i], vTensorSize[i], cudaMemcpyHostToDevice));}for (int i = 0; i < nIO; ++i){context->setTensorAddress(vTensorName[i].c_str(), vBufferD[i]);}context->enqueueV3(0);for (int i = nInput; i < nIO; ++i){CHECK(cudaMemcpy(vBufferH[i], vBufferD[i], vTensorSize[i], cudaMemcpyDeviceToHost));}for (int i = 0; i < nIO; ++i){printArrayInformation((float *)vBufferH[i], context->getTensorShape(vTensorName[i].c_str()), vTensorName[i], true, true);}for (int i = 0; i < nIO; ++i){delete[] (char *)vBufferH[i];CHECK(cudaFree(vBufferD[i]));}return;
}int main()
{CHECK(cudaSetDevice(0));run();run();return 0;
}

make之后生成main.exe和main.d文件说明编译成功,再使用./main.exe直接运行。

make
./main.exe

输出如下
在这里插入图片描述

4.3 main.py

在这里插入图片描述

在这里插入图片描述

TensorRT C++ API 基础解析 https://blog.csdn.net/m0_70420861/article/details/135574423

ppt官方代码代码里面有下载地址
在这里插入图片描述
main.py
使用TensorRT进行推理的示例代码。它首先检查是否存在一个已序列化的网络文件,如果存在,则加载该文件并进行推理;如果不存在,则从头开始构建一个网络,并将其序列化保存到文件中,然后进行推理。

#
# Copyright (c) 2021-2023, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#import osimport numpy as np
import tensorrt as trt
from cuda import cudart# yapf:disabletrtFile = "./model.plan"
data = np.arange(3 * 4 * 5, dtype=np.float32).reshape(3, 4, 5)                  # input data for inferencedef run():logger = trt.Logger(trt.Logger.ERROR)                                       # create Logger, avaiable level: VERBOSE, INFO, WARNING, ERRROR, INTERNAL_ERRORif os.path.isfile(trtFile):                                                 # load serialized network and skip building process if .plan file existedwith open(trtFile, "rb") as f:engineString = f.read()if engineString == None:print("Failed getting serialized engine!")returnprint("Succeeded getting serialized engine!")else:                                                                       # build a serialized network from scratchbuilder = trt.Builder(logger)                                           # create Buildernetwork = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))  # create Networkprofile = builder.create_optimization_profile()                         # create Optimization Profile if using Dynamic Shape modeconfig = builder.create_builder_config()                                # create BuidlerConfig to set meta data of the networkconfig.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)     # set workspace for the optimization process (default value is total GPU memory)inputTensor = network.add_input("inputT0", trt.float32, [-1, -1, -1])   # set inpute tensor for the networkprofile.set_shape(inputTensor.name, [1, 1, 1], [3, 4, 5], [6, 8, 10])   # set danamic range of the input tensorconfig.add_optimization_profile(profile)                                # add the Optimization Profile into the BuilderConfigidentityLayer = network.add_identity(inputTensor)                       # here is only a identity transformation layer in our simple network, which the output is exactly equal to inputnetwork.mark_output(identityLayer.get_output(0))                        # mark the output tensor of the networkengineString = builder.build_serialized_network(network, config)        # create a serialized networkif engineString == None:print("Failed building serialized engine!")returnprint("Succeeded building serialized engine!")with open(trtFile, "wb") as f:                                          # write the serialized netwok into a .plan filef.write(engineString)print("Succeeded saving .plan file!")engine = trt.Runtime(logger).deserialize_cuda_engine(engineString)          # create inference Engine using Runtimeif engine == None:print("Failed building engine!")returnprint("Succeeded building engine!")nIO = engine.num_io_tensors                                                 # since TensorRT 8.5, the concept of Binding is replaced by I/O Tensor, all the APIs with "binding" in their name are deprecatedlTensorName = [engine.get_tensor_name(i) for i in range(nIO)]               # get a list of I/O tensor names of the engine, because all I/O tensor in Engine and Excution Context are indexed by name, not binding number like TensorRT 8.4 or beforenInput = [engine.get_tensor_mode(lTensorName[i]) for i in range(nIO)].count(trt.TensorIOMode.INPUT)  # get the count of input tensor#nOutput = [engine.get_tensor_mode(lTensorName[i]) for i in range(nIO)].count(trt.TensorIOMode.OUTPUT)  # get the count of output tensorcontext = engine.create_execution_context()                                 # create Excution Context from the engine (analogy to a GPU context, or a CPU process)context.set_input_shape(lTensorName[0], [3, 4, 5])                          # set actual size of input tensor if using Dynamic Shape modefor i in range(nIO):print("[%2d]%s->" % (i, "Input " if i < nInput else "Output"), engine.get_tensor_dtype(lTensorName[i]), engine.get_tensor_shape(lTensorName[i]), context.get_tensor_shape(lTensorName[i]), lTensorName[i])bufferH = []                                                                # prepare the memory buffer on host and devicebufferH.append(np.ascontiguousarray(data))for i in range(nInput, nIO):bufferH.append(np.empty(context.get_tensor_shape(lTensorName[i]), dtype=trt.nptype(engine.get_tensor_dtype(lTensorName[i]))))bufferD = []for i in range(nIO):bufferD.append(cudart.cudaMalloc(bufferH[i].nbytes)[1])for i in range(nInput):                                                     # copy input data from host buffer into device buffercudart.cudaMemcpy(bufferD[i], bufferH[i].ctypes.data, bufferH[i].nbytes, cudart.cudaMemcpyKind.cudaMemcpyHostToDevice)for i in range(nIO):context.set_tensor_address(lTensorName[i], int(bufferD[i]))             # set address of all input and output data in device buffercontext.execute_async_v3(0)                                                 # do inference computationfor i in range(nInput, nIO):                                                # copy output data from device buffer into host buffercudart.cudaMemcpy(bufferH[i].ctypes.data, bufferD[i], bufferH[i].nbytes, cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost)for i in range(nIO):print(lTensorName[i])print(bufferH[i])for b in bufferD:                                                           # free the GPU memory buffer after all workcudart.cudaFree(b)if __name__ == "__main__":os.system("rm -rf ./*.plan")run()                                                                       # create a serialized network of TensorRT and do inferencerun()                                                                       # load a serialized network of TensorRT and do inference
python main.py

如果报以下错误
在这里插入图片描述

pip install cuda-python -i https://mirrors.aliyun.com/pypi/simple/

输出结果
在这里插入图片描述
会生成一个model.plan文件

注意:安装 TensorRT Python 包是使用 TensorRT 的关键步骤之一,如果报tensorrt模块未找到就是没装。
可以用以下命令进行安装

pip install tensorrt

这个快一点

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple tensorrt

总结

TensorRT英伟达官方示例解析 00 和 01 后续更新

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.hqwc.cn/news/427650.html

如若内容造成侵权/违法违规/事实不符,请联系编程知识网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

了解HTTP/1.1、HTTP/1.0 和 HTTP/2.0

HTTP/1.1、HTTP/1.0 和 HTTP/2.0 是超文本传输协议&#xff08;HTTP&#xff09;的三个主要版本 先解释一下什么是超文本协议 超文本传输协议&#xff08;HyperText Transfer Protocol&#xff0c;简称 HTTP&#xff09;是互联网上应用最广泛的一种网络协议。设计 HTTP 的初衷是…

如何在Windows 10/11中设置IPv4?这里有详细步骤

在Windows上&#xff0c;路由器的动态主机配置协议&#xff08;DHCP&#xff09;服务器&#xff08;通常&#xff09;负责将动态传输控制协议/互联网协议&#xff08;TCP/IP&#xff09;配置分配给网络中的每个设备&#xff0c;包括运行Windows 11或Windows 10的计算机。 尽管…

基于Springboot的大学生心理健康管理系统(有报告)。Javaee项目,springboot项目。

演示视频&#xff1a; 基于Springboot的大学生心理健康管理系统&#xff08;有报告&#xff09;。Javaee项目&#xff0c;springboot项目。 项目介绍&#xff1a; 采用M&#xff08;model&#xff09;V&#xff08;view&#xff09;C&#xff08;controller&#xff09;三层体…

c++:类和对象(5),运算符重载

目录 运算符重载概念&#xff1a; 运算符重载 1.成员函数重载号 2.全局函数重载号 打印结果&#xff1a; <<运算符重载 递增运算符重载 简单例子 输出结果为&#xff1a; 赋值运算符重载 如何重载 输出结果为&#xff1a; 什么时候重载 关系运算符重载 简单例…

基于Java SSM框架实现医院预约挂号系统项目【项目源码】计算机毕业设计

基于java的SSM框架实现医院预约挂号系统演示 JSP技术介绍 JSP技术本身是一种脚本语言&#xff0c;但它的功能是十分强大的&#xff0c;因为它可以使用所有的JAVA类。当它与JavaBeans 类进行结合时&#xff0c;它可以使显示逻辑和内容分开&#xff0c;这就极大的方便了用户的需…

.git 文件夹结构解析

.git 文件夹结构解析 在这篇文章就让我们来看看这个 Git 仓库里的文件分别都是用来干什么的&#xff0c;以及在执行了相关的 Git 命令后这些文件会如何响应。 hooks&#xff08;钩&#xff09;&#xff1a;存放一些shell脚本info&#xff1a;存放仓库的一些信息logs&#xff…

JAVA文件操作IO流(一)

File 文件对象 文件 在计算机中&#xff0c;文件是非常重要的存储方式&#xff0c;JAVA的标准库java.io提供了File对象来操作文件和目录。要构建一个File对象&#xff0c;需要通过构建方法&#xff0c;传入该文件的本地物理路径。 例如&#xff1a; 获取文件对象的1.绝对路径…

51单片机8*8点阵屏

8*8点阵屏 8*8点阵屏是一种LED显示屏&#xff0c;它由8行和8列的LED灯组成。每个LED灯的开闭状态都可以独立控制&#xff0c;从而可以显示出数字、字母、符号、图形等信息。 8*8点阵屏的原理是通过行列扫描的方式&#xff0c;控制LED灯的亮灭&#xff0c;从而显示出所需的图案或…

USRP相关报错解决办法

文章目录 前言一、本地环境二、相关报错信息二、解决办法1、更换电脑操作系统2、升级最新版固件 前言 在进行 USRP 开发时遇到了一些报错&#xff0c;这里做个记录解决问题的方法。 一、本地环境 电脑操作系统&#xff1a;Windows11MATLAB 版本&#xff1a;MATLAB 2021aUSRP …

C#,生成图片的指定尺寸缩略图的源代码

编程的时候经常用到图像的缩略图。 本文发布一个用于生成指定尺寸的缩略图的简单方法。 1 文本格式 private void button1_Click(object sender, EventArgs e) { CreateThumbnail("demo.jpg", "demo_thumb.jpg", 128, 128); } private void CreateTh…

【Web前端开发基础】CSS3之Web字体、字体图标、平面转换、渐变

CSS3之Web字体、字体图标、平面转换、渐变 目录 CSS3之Web字体、字体图标、平面转换、渐变一、Web字体1.1 Web字体概述1.2 字体文件1.3 font-face 规则 二、字体图标2.1 字体图标2.2 字体图标的优点2.3 图标库2.4 下载字体包2.5 字体图标的使用步骤2.6 字体图标使用注意点2.7 上…

Poe聊天机器人平台吸引新用户可获得最高50美元的收入

Poe聊天机器人平台正加大对创作者的奖励力度&#xff0c;机器人创建者每吸引一个新订阅者&#xff0c;最高可获得50美元&#xff08;约等于358.465人民币&#xff09;的收入。在GPTs商店上传过自定义GPT的朋友可以将自己的机器人复刻到Poe上&#xff0c;看看能不能拿到一些零食…