xiajingrui/id_ocr
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
xiajingrui
2024-01-17 15:18:16 +08:00
commit 1bd4060e68
1514 changed files with 193581 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/fastdeploy/serving/fastdeploy_serving/README.md
+127
View File
@@ -0,0 +1,127 @@
[English](README.md) | 简体中文
# PaddleOCR服务化部署示例
PaddleOCR 服务化部署示例是利用FastDeploy Serving搭建的服务化部署示例。FastDeploy Serving是基于Triton Inference Server框架封装的适用于高并发、高吞吐量请求的服务化部署框架，是一套可用于实际生产的完备且性能卓越的服务化部署框架。如没有高并发，高吞吐场景的需求，只想快速检验模型线上部署的可行性，请参考[simple_serving](../simple_serving/)
## 1. 部署环境准备
在服务化部署前，需确认服务化镜像的软硬件环境要求和镜像拉取命令，请参考[FastDeploy服务化部署](https://github.com/PaddlePaddle/FastDeploy/blob/develop/serving/README_CN.md)
## 2. PP-OCRv3服务化部署介绍
本文介绍了使用FastDeploy搭建PP-OCRv3模型服务的方法.
服务端必须在docker内启动,而客户端不是必须在docker容器内.
**本文所在路径($PWD)下的models里包含模型的配置和代码(服务端会加载模型和代码以启动服务), 需要将其映射到docker中使用.**
PP-OCRv3由det(检测)、cls(分类)和rec(识别)三个模型组成.
服务化部署串联的示意图如下图所示,其中`pp_ocr`串联了`det_preprocess`、`det_runtime`和`det_postprocess`,`cls_pp`串联了`cls_runtime`和`cls_postprocess`,`rec_pp`串联了`rec_runtime`和`rec_postprocess`.
特别的是,在`det_postprocess`中会多次调用`cls_pp`和`rec_pp`服务,来实现对检测结果(多个框)进行分类和识别,,最后返回给用户最终的识别结果。
<p align="center">
<br>
<img src='./ppocr.png'">
<br>
<p>
## 3. 服务端的使用
### 3.1 下载模型并使用服务化Docker
```bash
# 下载仓库代码
# 下载部署示例代码
git clone https://github.com/PaddlePaddle/FastDeploy.git
cd FastDeploy/examples/vision/ocr/PP-OCR/serving/fastdeploy_serving
# 如果您希望从PaddleOCR下载示例代码，请运行
git clone https://github.com/PaddlePaddle/PaddleOCR.git
# 注意：如果当前分支找不到下面的fastdeploy测试代码，请切换到dygraph分支
git checkout dygraph
cd PaddleOCR/deploy/fastdeploy/serving/fastdeploy_serving
# 下载模型,图片和字典文件
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar
tar xvf ch_PP-OCRv3_det_infer.tar && mv ch_PP-OCRv3_det_infer 1
mv 1/inference.pdiparams 1/model.pdiparams && mv 1/inference.pdmodel 1/model.pdmodel
mv 1 models/det_runtime/ && rm -rf ch_PP-OCRv3_det_infer.tar
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar
tar xvf ch_ppocr_mobile_v2.0_cls_infer.tar && mv ch_ppocr_mobile_v2.0_cls_infer 1
mv 1/inference.pdiparams 1/model.pdiparams && mv 1/inference.pdmodel 1/model.pdmodel
mv 1 models/cls_runtime/ && rm -rf ch_ppocr_mobile_v2.0_cls_infer.tar
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar
tar xvf ch_PP-OCRv3_rec_infer.tar && mv ch_PP-OCRv3_rec_infer 1
mv 1/inference.pdiparams 1/model.pdiparams && mv 1/inference.pdmodel 1/model.pdmodel
mv 1 models/rec_runtime/ && rm -rf ch_PP-OCRv3_rec_infer.tar
mkdir models/pp_ocr/1 && mkdir models/rec_pp/1 && mkdir models/cls_pp/1
wget https://gitee.com/paddlepaddle/PaddleOCR/raw/release/2.6/ppocr/utils/ppocr_keys_v1.txt
mv ppocr_keys_v1.txt models/rec_postprocess/1/
wget https://gitee.com/paddlepaddle/PaddleOCR/raw/release/2.6/doc/imgs/12.jpg
# x.y.z为镜像版本号，需参照serving文档替换为数字
docker pull registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10
docker run -dit --net=host --name fastdeploy --shm-size="1g" -v $PWD:/ocr_serving registry.baidubce.com/paddlepaddle/fastdeploy:x.y.z-gpu-cuda11.4-trt8.4-21.10 bash
docker exec -it -u root fastdeploy bash
```
### 3.2 安装(在docker内)
```bash
ldconfig
apt-get install libgl1
```
#### 3.3 启动服务端(在docker内)
```bash
fastdeployserver --model-repository=/ocr_serving/models
```
参数:
- `model-repository`(required): 整套模型streaming_pp_tts存放的路径.
- `http-port`(optional): HTTP服务的端口号. 默认: `8000`. 本示例中未使用该端口.
- `grpc-port`(optional): GRPC服务的端口号. 默认: `8001`.
- `metrics-port`(optional): 服务端指标的端口号. 默认: `8002`. 本示例中未使用该端口.
## 4. 客户端的使用
### 4.1 安装
```bash
pip3 install tritonclient[all]
```
### 4.2 发送请求
```bash
python3 client.py
```
## 5.配置修改
当前默认配置在GPU上运行， 如果要在CPU或其他推理引擎上运行。 需要修改`models/runtime/config.pbtxt`中配置，详情请参考[配置文档](../../../../../serving/docs/zh_CN/model_configuration.md)
## 6. 其他指南
- 使用PP-OCRv2进行服务化部署, 除了自行准备PP-OCRv2模型之外, 只需手动添加一行代码即可.
在[model.py](./models/det_postprocess/1/model.py#L109)文件**109行添加以下代码**：
```
self.rec_preprocessor.cls_image_shape[1] = 32
```
- [使用 VisualDL 进行 Serving 可视化部署](https://github.com/PaddlePaddle/FastDeploy/blob/develop/serving/docs/zh_CN/vdl_management.md)
通过VisualDL的可视化界面对PP-OCRv3进行服务化部署只需要如下三步：
```text
1. 载入模型库：./vision/ocr/PP-OCRv3/serving
2. 下载模型资源文件：点击det_runtime模型，点击版本号1添加预训练模型，选择文字识别模型ch_PP-OCRv3_det进行下载。点击cls_runtime模型，点击版本号1添加预训练模型，选择文字识别模型ch_ppocr_mobile_v2.0_cls进行下载。点击rec_runtime模型，点击版本号1添加预训练模型，选择文字识别模型ch_PP-OCRv3_rec进行下载。点击rec_postprocess模型，点击版本号1添加预训练模型，选择文字识别模型ch_PP-OCRv3_rec进行下载。
3. 启动服务：点击启动服务按钮，输入启动参数。
```
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/211709324-b07bb303-ced2-4137-9df7-0d2574ba84c8.gif" width="100%"/>
</p>
## 7. 常见问题
- [如何编写客户端 HTTP/GRPC 请求](https://github.com/PaddlePaddle/FastDeploy/blob/develop/serving/docs/zh_CN/client.md)
- [如何编译服务化部署镜像](https://github.com/PaddlePaddle/FastDeploy/blob/develop/serving/docs/zh_CN/compile.md)
- [服务化部署原理及动态Batch介绍](https://github.com/PaddlePaddle/FastDeploy/blob/develop/serving/docs/zh_CN/demo.md)
- [模型仓库介绍](https://github.com/PaddlePaddle/FastDeploy/blob/develop/serving/docs/zh_CN/model_repository.md)
deploy/fastdeploy/serving/fastdeploy_serving/client.py
+109
View File
@@ -0,0 +1,109 @@
import logging
import numpy as np
import time
from typing import Optional
import cv2
import json
from tritonclient import utils as client_utils
from tritonclient.grpc import InferenceServerClient, InferInput, InferRequestedOutput, service_pb2_grpc, service_pb2
LOGGER = logging.getLogger("run_inference_on_triton")
class SyncGRPCTritonRunner:
DEFAULT_MAX_RESP_WAIT_S = 120
def __init__(
self,
server_url: str,
model_name: str,
model_version: str,
*,
verbose=False,
resp_wait_s: Optional[float]=None, ):
self._server_url = server_url
self._model_name = model_name
self._model_version = model_version
self._verbose = verbose
self._response_wait_t = self.DEFAULT_MAX_RESP_WAIT_S if resp_wait_s is None else resp_wait_s
self._client = InferenceServerClient(
self._server_url, verbose=self._verbose)
error = self._verify_triton_state(self._client)
if error:
raise RuntimeError(
f"Could not communicate to Triton Server: {error}")
LOGGER.debug(
f"Triton server {self._server_url} and model {self._model_name}:{self._model_version} "
f"are up and ready!")
model_config = self._client.get_model_config(self._model_name,
self._model_version)
model_metadata = self._client.get_model_metadata(self._model_name,
self._model_version)
LOGGER.info(f"Model config {model_config}")
LOGGER.info(f"Model metadata {model_metadata}")
self._inputs = {tm.name: tm for tm in model_metadata.inputs}
self._input_names = list(self._inputs)
self._outputs = {tm.name: tm for tm in model_metadata.outputs}
self._output_names = list(self._outputs)
self._outputs_req = [
InferRequestedOutput(name) for name in self._outputs
]
def Run(self, inputs):
"""
Args:
inputs: list, Each value corresponds to an input name of self._input_names
Returns:
results: dict, {name : numpy.array}
"""
infer_inputs = []
for idx, data in enumerate(inputs):
infer_input = InferInput(self._input_names[idx], data.shape,
"UINT8")
infer_input.set_data_from_numpy(data)
infer_inputs.append(infer_input)
results = self._client.infer(
model_name=self._model_name,
model_version=self._model_version,
inputs=infer_inputs,
outputs=self._outputs_req,
client_timeout=self._response_wait_t, )
results = {name: results.as_numpy(name) for name in self._output_names}
return results
def _verify_triton_state(self, triton_client):
if not triton_client.is_server_live():
return f"Triton server {self._server_url} is not live"
elif not triton_client.is_server_ready():
return f"Triton server {self._server_url} is not ready"
elif not triton_client.is_model_ready(self._model_name,
self._model_version):
return f"Model {self._model_name}:{self._model_version} is not ready"
return None
if __name__ == "__main__":
model_name = "pp_ocr"
model_version = "1"
url = "localhost:8001"
runner = SyncGRPCTritonRunner(url, model_name, model_version)
im = cv2.imread("12.jpg")
im = np.array([im, ])
for i in range(1):
result = runner.Run([im, ])
batch_texts = result['rec_texts']
batch_scores = result['rec_scores']
batch_bboxes = result['det_bboxes']
for i_batch in range(len(batch_texts)):
texts = batch_texts[i_batch]
scores = batch_scores[i_batch]
bboxes = batch_bboxes[i_batch]
for i_box in range(len(texts)):
print('text=', texts[i_box].decode('utf-8'), ' score=',
scores[i_box], ' bbox=', bboxes[i_box])
deploy/fastdeploy/serving/fastdeploy_serving/models/cls_postprocess/1/model.py
+105
View File
@@ -0,0 +1,105 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import numpy as np
import time
import fastdeploy as fd
# triton_python_backend_utils is available in every Triton Python model. You
# need to use this module to create inference requests and responses. It also
# contains some utility functions for extracting information from model_config
# and converting Triton input/output types to numpy types.
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
"""Your Python model must use the same class name. Every Python model
that is created must have "TritonPythonModel" as the class name.
"""
def initialize(self, args):
"""`initialize` is called only once when the model is being loaded.
Implementing `initialize` function is optional. This function allows
the model to intialize any state associated with this model.
Parameters
----------
args : dict
Both keys and values are strings. The dictionary keys and values are:
* model_config: A JSON string containing the model configuration
* model_instance_kind: A string containing model instance kind
* model_instance_device_id: A string containing model instance device ID
* model_repository: Model repository path
* model_version: Model version
* model_name: Model name
"""
# You must parse model_config. JSON string is not parsed here
self.model_config = json.loads(args['model_config'])
print("model_config:", self.model_config)
self.input_names = []
for input_config in self.model_config["input"]:
self.input_names.append(input_config["name"])
print("postprocess input names:", self.input_names)
self.output_names = []
self.output_dtype = []
for output_config in self.model_config["output"]:
self.output_names.append(output_config["name"])
dtype = pb_utils.triton_string_to_numpy(output_config["data_type"])
self.output_dtype.append(dtype)
print("postprocess output names:", self.output_names)
self.postprocessor = fd.vision.ocr.ClassifierPostprocessor()
def execute(self, requests):
"""`execute` must be implemented in every Python model. `execute`
function receives a list of pb_utils.InferenceRequest as the only
argument. This function is called when an inference is requested
for this model. Depending on the batching configuration (e.g. Dynamic
Batching) used, `requests` may contain multiple requests. Every
Python model, must create one pb_utils.InferenceResponse for every
pb_utils.InferenceRequest in `requests`. If there is an error, you can
set the error argument when creating a pb_utils.InferenceResponse.
Parameters
----------
requests : list
A list of pb_utils.InferenceRequest
Returns
-------
list
A list of pb_utils.InferenceResponse. The length of this list must
be the same as `requests`
"""
responses = []
for request in requests:
infer_outputs = pb_utils.get_input_tensor_by_name(
request, self.input_names[0])
infer_outputs = infer_outputs.as_numpy()
results = self.postprocessor.run([infer_outputs])
out_tensor_0 = pb_utils.Tensor(self.output_names[0],
np.array(results[0]))
out_tensor_1 = pb_utils.Tensor(self.output_names[1],
np.array(results[1]))
inference_response = pb_utils.InferenceResponse(
output_tensors=[out_tensor_0, out_tensor_1])
responses.append(inference_response)
return responses
def finalize(self):
"""`finalize` is called only once when the model is being unloaded.
Implementing `finalize` function is optional. This function allows
the model to perform any necessary clean ups before exit.
"""
print('Cleaning up...')
deploy/fastdeploy/serving/fastdeploy_serving/models/cls_postprocess/config.pbtxt
+30
View File
@@ -0,0 +1,30 @@
name: "cls_postprocess"
backend: "python"
max_batch_size: 128
input [
{
name: "POST_INPUT_0"
data_type: TYPE_FP32
dims: [ 2 ]
}
]
output [
{
name: "POST_OUTPUT_0"
data_type: TYPE_INT32
dims: [ 1 ]
},
{
name: "POST_OUTPUT_1"
data_type: TYPE_FP32
dims: [ 1 ]
}
]
instance_group [
{
count: 1
kind: KIND_CPU
}
]
deploy/fastdeploy/serving/fastdeploy_serving/models/cls_pp/config.pbtxt
+54
View File
@@ -0,0 +1,54 @@
name: "cls_pp"
platform: "ensemble"
max_batch_size: 128
input [
{
name: "x"
data_type: TYPE_FP32
dims: [ 3, -1, -1 ]
}
]
output [
{
name: "cls_labels"
data_type: TYPE_INT32
dims: [ 1 ]
},
{
name: "cls_scores"
data_type: TYPE_FP32
dims: [ 1 ]
}
]
ensemble_scheduling {
step [
{
model_name: "cls_runtime"
model_version: 1
input_map {
key: "x"
value: "x"
}
output_map {
key: "softmax_0.tmp_0"
value: "infer_output"
}
},
{
model_name: "cls_postprocess"
model_version: 1
input_map {
key: "POST_INPUT_0"
value: "infer_output"
}
output_map {
key: "POST_OUTPUT_0"
value: "cls_labels"
}
output_map {
key: "POST_OUTPUT_1"
value: "cls_scores"
}
}
]
}
deploy/fastdeploy/serving/fastdeploy_serving/models/cls_runtime/config.pbtxt
+52
View File
@@ -0,0 +1,52 @@
# optional, If name is specified it must match the name of the model repository directory containing the model.
name: "cls_runtime"
backend: "fastdeploy"
max_batch_size: 128
# Input configuration of the model
input [
{
# input name
name: "x"
# input type such as TYPE_FP32、TYPE_UINT8、TYPE_INT8、TYPE_INT16、TYPE_INT32、TYPE_INT64、TYPE_FP16、TYPE_STRING
data_type: TYPE_FP32
# input shape， The batch dimension is omitted and the actual shape is [batch, c, h, w]
dims: [ 3, -1, -1 ]
}
]
# The output of the model is configured in the same format as the input
output [
{
name: "softmax_0.tmp_0"
data_type: TYPE_FP32
dims: [ 2 ]
}
]
# Number of instances of the model
instance_group [
{
# The number of instances is 1
count: 1
# Use GPU, CPU inference option is:KIND_CPU
kind: KIND_GPU
# The instance is deployed on the 0th GPU card
gpus: [0]
}
]
optimization {
execution_accelerators {
# GPU推理配置， 配合KIND_GPU使用
gpu_execution_accelerator : [
{
name : "paddle"
# 设置推理并行计算线程数为4
parameters { key: "cpu_threads" value: "4" }
# 开启mkldnn加速，设置为0关闭mkldnn
parameters { key: "use_mkldnn" value: "1" }
}
]
}
}
deploy/fastdeploy/serving/fastdeploy_serving/models/det_postprocess/1/model.py
+238
View File
@@ -0,0 +1,238 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import numpy as np
import time
import math
import cv2
import fastdeploy as fd
# triton_python_backend_utils is available in every Triton Python model. You
# need to use this module to create inference requests and responses. It also
# contains some utility functions for extracting information from model_config
# and converting Triton input/output types to numpy types.
import triton_python_backend_utils as pb_utils
def get_rotate_crop_image(img, box):
'''
img_height, img_width = img.shape[0:2]
left = int(np.min(points[:, 0]))
right = int(np.max(points[:, 0]))
top = int(np.min(points[:, 1]))
bottom = int(np.max(points[:, 1]))
img_crop = img[top:bottom, left:right, :].copy()
points[:, 0] = points[:, 0] - left
points[:, 1] = points[:, 1] - top
'''
points = []
for i in range(4):
points.append([box[2 * i], box[2 * i + 1]])
points = np.array(points, dtype=np.float32)
img = img.astype(np.float32)
assert len(points) == 4, "shape of points must be 4*2"
img_crop_width = int(
max(
np.linalg.norm(points[0] - points[1]),
np.linalg.norm(points[2] - points[3])))
img_crop_height = int(
max(
np.linalg.norm(points[0] - points[3]),
np.linalg.norm(points[1] - points[2])))
pts_std = np.float32([[0, 0], [img_crop_width, 0],
[img_crop_width, img_crop_height],
[0, img_crop_height]])
M = cv2.getPerspectiveTransform(points, pts_std)
dst_img = cv2.warpPerspective(
img,
M, (img_crop_width, img_crop_height),
borderMode=cv2.BORDER_REPLICATE,
flags=cv2.INTER_CUBIC)
dst_img_height, dst_img_width = dst_img.shape[0:2]
if dst_img_height * 1.0 / dst_img_width >= 1.5:
dst_img = np.rot90(dst_img)
return dst_img
class TritonPythonModel:
"""Your Python model must use the same class name. Every Python model
that is created must have "TritonPythonModel" as the class name.
"""
def initialize(self, args):
"""`initialize` is called only once when the model is being loaded.
Implementing `initialize` function is optional. This function allows
the model to intialize any state associated with this model.
Parameters
----------
args : dict
Both keys and values are strings. The dictionary keys and values are:
* model_config: A JSON string containing the model configuration
* model_instance_kind: A string containing model instance kind
* model_instance_device_id: A string containing model instance device ID
* model_repository: Model repository path
* model_version: Model version
* model_name: Model name
"""
# You must parse model_config. JSON string is not parsed here
self.model_config = json.loads(args['model_config'])
print("model_config:", self.model_config)
self.input_names = []
for input_config in self.model_config["input"]:
self.input_names.append(input_config["name"])
print("postprocess input names:", self.input_names)
self.output_names = []
self.output_dtype = []
for output_config in self.model_config["output"]:
self.output_names.append(output_config["name"])
dtype = pb_utils.triton_string_to_numpy(output_config["data_type"])
self.output_dtype.append(dtype)
print("postprocess output names:", self.output_names)
self.postprocessor = fd.vision.ocr.DBDetectorPostprocessor()
self.cls_preprocessor = fd.vision.ocr.ClassifierPreprocessor()
self.rec_preprocessor = fd.vision.ocr.RecognizerPreprocessor()
self.cls_threshold = 0.9
def execute(self, requests):
"""`execute` must be implemented in every Python model. `execute`
function receives a list of pb_utils.InferenceRequest as the only
argument. This function is called when an inference is requested
for this model. Depending on the batching configuration (e.g. Dynamic
Batching) used, `requests` may contain multiple requests. Every
Python model, must create one pb_utils.InferenceResponse for every
pb_utils.InferenceRequest in `requests`. If there is an error, you can
set the error argument when creating a pb_utils.InferenceResponse.
Parameters
----------
requests : list
A list of pb_utils.InferenceRequest
Returns
-------
list
A list of pb_utils.InferenceResponse. The length of this list must
be the same as `requests`
"""
responses = []
for request in requests:
infer_outputs = pb_utils.get_input_tensor_by_name(
request, self.input_names[0])
im_infos = pb_utils.get_input_tensor_by_name(request,
self.input_names[1])
ori_imgs = pb_utils.get_input_tensor_by_name(request,
self.input_names[2])
infer_outputs = infer_outputs.as_numpy()
im_infos = im_infos.as_numpy()
ori_imgs = ori_imgs.as_numpy()
results = self.postprocessor.run([infer_outputs], im_infos)
batch_rec_texts = []
batch_rec_scores = []
batch_box_list = []
for i_batch in range(len(results)):
cls_labels = []
cls_scores = []
rec_texts = []
rec_scores = []
box_list = fd.vision.ocr.sort_boxes(results[i_batch])
image_list = []
if len(box_list) == 0:
image_list.append(ori_imgs[i_batch])
else:
for box in box_list:
crop_img = get_rotate_crop_image(ori_imgs[i_batch], box)
image_list.append(crop_img)
batch_box_list.append(box_list)
cls_pre_tensors = self.cls_preprocessor.run(image_list)
cls_dlpack_tensor = cls_pre_tensors[0].to_dlpack()
cls_input_tensor = pb_utils.Tensor.from_dlpack(
"x", cls_dlpack_tensor)
inference_request = pb_utils.InferenceRequest(
model_name='cls_pp',
requested_output_names=['cls_labels', 'cls_scores'],
inputs=[cls_input_tensor])
inference_response = inference_request.exec()
if inference_response.has_error():
raise pb_utils.TritonModelException(
inference_response.error().message())
else:
# Extract the output tensors from the inference response.
cls_labels = pb_utils.get_output_tensor_by_name(
inference_response, 'cls_labels')
cls_labels = cls_labels.as_numpy()
cls_scores = pb_utils.get_output_tensor_by_name(
inference_response, 'cls_scores')
cls_scores = cls_scores.as_numpy()
for index in range(len(image_list)):
if cls_labels[index] == 1 and cls_scores[
index] > self.cls_threshold:
image_list[index] = cv2.rotate(
image_list[index].astype(np.float32), 1)
image_list[index] = np.astype(np.uint8)
rec_pre_tensors = self.rec_preprocessor.run(image_list)
rec_dlpack_tensor = rec_pre_tensors[0].to_dlpack()
rec_input_tensor = pb_utils.Tensor.from_dlpack(
"x", rec_dlpack_tensor)
inference_request = pb_utils.InferenceRequest(
model_name='rec_pp',
requested_output_names=['rec_texts', 'rec_scores'],
inputs=[rec_input_tensor])
inference_response = inference_request.exec()
if inference_response.has_error():
raise pb_utils.TritonModelException(
inference_response.error().message())
else:
# Extract the output tensors from the inference response.
rec_texts = pb_utils.get_output_tensor_by_name(
inference_response, 'rec_texts')
rec_texts = rec_texts.as_numpy()
rec_scores = pb_utils.get_output_tensor_by_name(
inference_response, 'rec_scores')
rec_scores = rec_scores.as_numpy()
batch_rec_texts.append(rec_texts)
batch_rec_scores.append(rec_scores)
out_tensor_0 = pb_utils.Tensor(
self.output_names[0],
np.array(
batch_rec_texts, dtype=np.object_))
out_tensor_1 = pb_utils.Tensor(self.output_names[1],
np.array(batch_rec_scores))
out_tensor_2 = pb_utils.Tensor(self.output_names[2],
np.array(batch_box_list))
inference_response = pb_utils.InferenceResponse(
output_tensors=[out_tensor_0, out_tensor_1, out_tensor_2])
responses.append(inference_response)
return responses
def finalize(self):
"""`finalize` is called only once when the model is being unloaded.
Implementing `finalize` function is optional. This function allows
the model to perform any necessary clean ups before exit.
"""
print('Cleaning up...')
deploy/fastdeploy/serving/fastdeploy_serving/models/det_postprocess/config.pbtxt
+45
View File
@@ -0,0 +1,45 @@
name: "det_postprocess"
backend: "python"
max_batch_size: 128
input [
{
name: "POST_INPUT_0"
data_type: TYPE_FP32
dims: [ 1, -1, -1]
},
{
name: "POST_INPUT_1"
data_type: TYPE_INT32
dims: [ 4 ]
},
{
name: "ORI_IMG"
data_type: TYPE_UINT8
dims: [ -1, -1, 3 ]
}
]
output [
{
name: "POST_OUTPUT_0"
data_type: TYPE_STRING
dims: [ -1, 1 ]
},
{
name: "POST_OUTPUT_1"
data_type: TYPE_FP32
dims: [ -1, 1 ]
},
{
name: "POST_OUTPUT_2"
data_type: TYPE_FP32
dims: [ -1, -1, 1 ]
}
]
instance_group [
{
count: 1
kind: KIND_CPU
}
]
deploy/fastdeploy/serving/fastdeploy_serving/models/det_preprocess/1/model.py
+107
View File
@@ -0,0 +1,107 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import numpy as np
import time
import fastdeploy as fd
# triton_python_backend_utils is available in every Triton Python model. You
# need to use this module to create inference requests and responses. It also
# contains some utility functions for extracting information from model_config
# and converting Triton input/output types to numpy types.
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
"""Your Python model must use the same class name. Every Python model
that is created must have "TritonPythonModel" as the class name.
"""
def initialize(self, args):
"""`initialize` is called only once when the model is being loaded.
Implementing `initialize` function is optional. This function allows
the model to intialize any state associated with this model.
Parameters
----------
args : dict
Both keys and values are strings. The dictionary keys and values are:
* model_config: A JSON string containing the model configuration
* model_instance_kind: A string containing model instance kind
* model_instance_device_id: A string containing model instance device ID
* model_repository: Model repository path
* model_version: Model version
* model_name: Model name
"""
# You must parse model_config. JSON string is not parsed here
self.model_config = json.loads(args['model_config'])
print("model_config:", self.model_config)
self.input_names = []
for input_config in self.model_config["input"]:
self.input_names.append(input_config["name"])
print("preprocess input names:", self.input_names)
self.output_names = []
self.output_dtype = []
for output_config in self.model_config["output"]:
self.output_names.append(output_config["name"])
dtype = pb_utils.triton_string_to_numpy(output_config["data_type"])
self.output_dtype.append(dtype)
print("preprocess output names:", self.output_names)
self.preprocessor = fd.vision.ocr.DBDetectorPreprocessor()
def execute(self, requests):
"""`execute` must be implemented in every Python model. `execute`
function receives a list of pb_utils.InferenceRequest as the only
argument. This function is called when an inference is requested
for this model. Depending on the batching configuration (e.g. Dynamic
Batching) used, `requests` may contain multiple requests. Every
Python model, must create one pb_utils.InferenceResponse for every
pb_utils.InferenceRequest in `requests`. If there is an error, you can
set the error argument when creating a pb_utils.InferenceResponse.
Parameters
----------
requests : list
A list of pb_utils.InferenceRequest
Returns
-------
list
A list of pb_utils.InferenceResponse. The length of this list must
be the same as `requests`
"""
responses = []
for request in requests:
data = pb_utils.get_input_tensor_by_name(request,
self.input_names[0])
data = data.as_numpy()
outputs, im_infos = self.preprocessor.run(data)
dlpack_tensor = outputs[0].to_dlpack()
output_tensor_0 = pb_utils.Tensor.from_dlpack(self.output_names[0],
dlpack_tensor)
output_tensor_1 = pb_utils.Tensor(
self.output_names[1], np.array(
im_infos, dtype=np.int32))
inference_response = pb_utils.InferenceResponse(
output_tensors=[output_tensor_0, output_tensor_1])
responses.append(inference_response)
return responses
def finalize(self):
"""`finalize` is called only once when the model is being unloaded.
Implementing `finalize` function is optional. This function allows
the model to perform any necessary clean ups before exit.
"""
print('Cleaning up...')
deploy/fastdeploy/serving/fastdeploy_serving/models/det_preprocess/config.pbtxt
+37
View File
@@ -0,0 +1,37 @@
# optional, If name is specified it must match the name of the model repository directory containing the model.
name: "det_preprocess"
backend: "python"
max_batch_size: 1
# Input configuration of the model
input [
{
# input name
name: "INPUT_0"
# input type such as TYPE_FP32、TYPE_UINT8、TYPE_INT8、TYPE_INT16、TYPE_INT32、TYPE_INT64、TYPE_FP16、TYPE_STRING
data_type: TYPE_UINT8
# input shape， The batch dimension is omitted and the actual shape is [batch, c, h, w]
dims: [ -1, -1, 3 ]
}
]
# The output of the model is configured in the same format as the input
output [
{
name: "OUTPUT_0"
data_type: TYPE_FP32
dims: [ 3, -1, -1 ]
},
{
name: "OUTPUT_1"
data_type: TYPE_INT32
dims: [ 4 ]
}
]
instance_group [
{
count: 1
kind: KIND_CPU
}
]
deploy/fastdeploy/serving/fastdeploy_serving/models/det_runtime/config.pbtxt
+52
View File
@@ -0,0 +1,52 @@
# optional, If name is specified it must match the name of the model repository directory containing the model.
name: "det_runtime"
backend: "fastdeploy"
max_batch_size: 1
# Input configuration of the model
input [
{
# input name
name: "x"
# input type such as TYPE_FP32、TYPE_UINT8、TYPE_INT8、TYPE_INT16、TYPE_INT32、TYPE_INT64、TYPE_FP16、TYPE_STRING
data_type: TYPE_FP32
# input shape， The batch dimension is omitted and the actual shape is [batch, c, h, w]
dims: [ 3, -1, -1 ]
}
]
# The output of the model is configured in the same format as the input
output [
{
name: "sigmoid_0.tmp_0"
data_type: TYPE_FP32
dims: [ 1, -1, -1 ]
}
]
# Number of instances of the model
instance_group [
{
# The number of instances is 1
count: 1
# Use GPU, CPU inference option is:KIND_CPU
kind: KIND_GPU
# The instance is deployed on the 0th GPU card
gpus: [0]
}
]
optimization {
execution_accelerators {
# GPU推理配置， 配合KIND_GPU使用
gpu_execution_accelerator : [
{
name : "paddle"
# 设置推理并行计算线程数为4
parameters { key: "cpu_threads" value: "4" }
# 开启mkldnn加速，设置为0关闭mkldnn
parameters { key: "use_mkldnn" value: "1" }
}
]
}
}
deploy/fastdeploy/serving/fastdeploy_serving/models/pp_ocr/config.pbtxt
+87
View File
@@ -0,0 +1,87 @@
name: "pp_ocr"
platform: "ensemble"
max_batch_size: 1
input [
{
name: "INPUT"
data_type: TYPE_UINT8
dims: [ -1, -1, 3 ]
}
]
output [
{
name: "rec_texts"
data_type: TYPE_STRING
dims: [ -1, 1 ]
},
{
name: "rec_scores"
data_type: TYPE_FP32
dims: [ -1, 1 ]
},
{
name: "det_bboxes"
data_type: TYPE_FP32
dims: [ -1, -1, 1 ]
}
]
ensemble_scheduling {
step [
{
model_name: "det_preprocess"
model_version: 1
input_map {
key: "INPUT_0"
value: "INPUT"
}
output_map {
key: "OUTPUT_0"
value: "infer_input"
}
output_map {
key: "OUTPUT_1"
value: "infos"
}
},
{
model_name: "det_runtime"
model_version: 1
input_map {
key: "x"
value: "infer_input"
}
output_map {
key: "sigmoid_0.tmp_0"
value: "infer_output"
}
},
{
model_name: "det_postprocess"
model_version: 1
input_map {
key: "POST_INPUT_0"
value: "infer_output"
}
input_map {
key: "POST_INPUT_1"
value: "infos"
}
input_map {
key: "ORI_IMG"
value: "INPUT"
}
output_map {
key: "POST_OUTPUT_0"
value: "rec_texts"
}
output_map {
key: "POST_OUTPUT_1"
value: "rec_scores"
}
output_map {
key: "POST_OUTPUT_2"
value: "det_bboxes"
}
}
]
}
deploy/fastdeploy/serving/fastdeploy_serving/models/rec_postprocess/1/model.py
+112
View File
@@ -0,0 +1,112 @@
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import numpy as np
import time
import os
import sys
import codecs
import fastdeploy as fd
# triton_python_backend_utils is available in every Triton Python model. You
# need to use this module to create inference requests and responses. It also
# contains some utility functions for extracting information from model_config
# and converting Triton input/output types to numpy types.
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
"""Your Python model must use the same class name. Every Python model
that is created must have "TritonPythonModel" as the class name.
"""
def initialize(self, args):
"""`initialize` is called only once when the model is being loaded.
Implementing `initialize` function is optional. This function allows
the model to intialize any state associated with this model.
Parameters
----------
args : dict
Both keys and values are strings. The dictionary keys and values are:
* model_config: A JSON string containing the model configuration
* model_instance_kind: A string containing model instance kind
* model_instance_device_id: A string containing model instance device ID
* model_repository: Model repository path
* model_version: Model version
* model_name: Model name
"""
# You must parse model_config. JSON string is not parsed here
self.model_config = json.loads(args['model_config'])
print("model_config:", self.model_config)
self.input_names = []
for input_config in self.model_config["input"]:
self.input_names.append(input_config["name"])
print("postprocess input names:", self.input_names)
self.output_names = []
self.output_dtype = []
for output_config in self.model_config["output"]:
self.output_names.append(output_config["name"])
dtype = pb_utils.triton_string_to_numpy(output_config["data_type"])
self.output_dtype.append(dtype)
print("postprocess output names:", self.output_names)
dir_name = os.path.dirname(os.path.realpath(__file__)) + "/"
file_name = dir_name + "ppocr_keys_v1.txt"
#self.label_list = load_dict()
self.postprocessor = fd.vision.ocr.RecognizerPostprocessor(file_name)
def execute(self, requests):
"""`execute` must be implemented in every Python model. `execute`
function receives a list of pb_utils.InferenceRequest as the only
argument. This function is called when an inference is requested
for this model. Depending on the batching configuration (e.g. Dynamic
Batching) used, `requests` may contain multiple requests. Every
Python model, must create one pb_utils.InferenceResponse for every
pb_utils.InferenceRequest in `requests`. If there is an error, you can
set the error argument when creating a pb_utils.InferenceResponse.
Parameters
----------
requests : list
A list of pb_utils.InferenceRequest
Returns
-------
list
A list of pb_utils.InferenceResponse. The length of this list must
be the same as `requests`
"""
responses = []
for request in requests:
infer_outputs = pb_utils.get_input_tensor_by_name(
request, self.input_names[0])
infer_outputs = infer_outputs.as_numpy()
results = self.postprocessor.run([infer_outputs])
out_tensor_0 = pb_utils.Tensor(
self.output_names[0], np.array(
results[0], dtype=np.object_))
out_tensor_1 = pb_utils.Tensor(self.output_names[1],
np.array(results[1]))
inference_response = pb_utils.InferenceResponse(
output_tensors=[out_tensor_0, out_tensor_1])
responses.append(inference_response)
return responses
def finalize(self):
"""`finalize` is called only once when the model is being unloaded.
Implementing `finalize` function is optional. This function allows
the model to perform any necessary clean ups before exit.
"""
print('Cleaning up...')
deploy/fastdeploy/serving/fastdeploy_serving/models/rec_postprocess/config.pbtxt
+30
View File
@@ -0,0 +1,30 @@
name: "rec_postprocess"
backend: "python"
max_batch_size: 128
input [
{
name: "POST_INPUT_0"
data_type: TYPE_FP32
dims: [ -1, 6625 ]
}
]
output [
{
name: "POST_OUTPUT_0"
data_type: TYPE_STRING
dims: [ 1 ]
},
{
name: "POST_OUTPUT_1"
data_type: TYPE_FP32
dims: [ 1 ]
}
]
instance_group [
{
count: 1
kind: KIND_CPU
}
]
deploy/fastdeploy/serving/fastdeploy_serving/models/rec_pp/config.pbtxt
+54
View File
@@ -0,0 +1,54 @@
name: "rec_pp"
platform: "ensemble"
max_batch_size: 128
input [
{
name: "x"
data_type: TYPE_FP32
dims: [ 3, 48, -1 ]
}
]
output [
{
name: "rec_texts"
data_type: TYPE_STRING
dims: [ 1 ]
},
{
name: "rec_scores"
data_type: TYPE_FP32
dims: [ 1 ]
}
]
ensemble_scheduling {
step [
{
model_name: "rec_runtime"
model_version: 1
input_map {
key: "x"
value: "x"
}
output_map {
key: "softmax_5.tmp_0"
value: "infer_output"
}
},
{
model_name: "rec_postprocess"
model_version: 1
input_map {
key: "POST_INPUT_0"
value: "infer_output"
}
output_map {
key: "POST_OUTPUT_0"
value: "rec_texts"
}
output_map {
key: "POST_OUTPUT_1"
value: "rec_scores"
}
}
]
}
deploy/fastdeploy/serving/fastdeploy_serving/models/rec_runtime/config.pbtxt
+52
View File
@@ -0,0 +1,52 @@
# optional, If name is specified it must match the name of the model repository directory containing the model.
name: "rec_runtime"
backend: "fastdeploy"
max_batch_size: 128
# Input configuration of the model
input [
{
# input name
name: "x"
# input type such as TYPE_FP32、TYPE_UINT8、TYPE_INT8、TYPE_INT16、TYPE_INT32、TYPE_INT64、TYPE_FP16、TYPE_STRING
data_type: TYPE_FP32
# input shape， The batch dimension is omitted and the actual shape is [batch, c, h, w]
dims: [ 3, 48, -1 ]
}
]
# The output of the model is configured in the same format as the input
output [
{
name: "softmax_5.tmp_0"
data_type: TYPE_FP32
dims: [ -1, 6625 ]
}
]
# Number of instances of the model
instance_group [
{
# The number of instances is 1
count: 1
# Use GPU, CPU inference option is:KIND_CPU
kind: KIND_GPU
# The instance is deployed on the 0th GPU card
gpus: [0]
}
]
optimization {
execution_accelerators {
# GPU推理配置， 配合KIND_GPU使用
gpu_execution_accelerator : [
{
name : "paddle"
# 设置推理并行计算线程数为4
parameters { key: "cpu_threads" value: "4" }
# 开启mkldnn加速，设置为0关闭mkldnn
parameters { key: "use_mkldnn" value: "1" }
}
]
}
}
deploy/fastdeploy/serving/fastdeploy_serving/ppocr.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB