The BMNET library is designed to convert the neural networks defined by CAFFE to target instructions. It seems like a compiler which translates high-level language into machine instruc- tions. It also contains three phase which are the front end, the optimizer and the back end. The front end parses source code, extracts network prototxt and weights. The optimizer is responsible for doing a broad variety of transformations to try to improve the code’s running time. The back end (also known as the code generator) then maps the code onto the target instruction set. In the BM1880 platform,we add a new feature called INT8 computation,it can provide better performance such as inference speedup. INT8 computation need an calibration table to modify network parameter,you can refer section 2 for how to generate a network’s calibration table. Refer this document,you can convert a network from FP32 to INT8 without significant accuracy loss.
General Description
We provide multiple utility tools to convert CAFFE models into machine instructions. These instructions, as well as model’s weights, would be packed into a file named bmodel (model file for BITMAIN targets), which can be executed in BITMAIN board directly. BMNet has implemented many common layers, the full list of build-in layers is in below table, and many more layers are in developing:
Activation
BatchNorm
Concat
Convolution
Eltwise
Flatten
InnerProduct
Join
LRN
Normalize
Permute
Pooling
PReLU
PriorBox
Reorg
Reshape
Scale
Split
Upsample
If layers of your network model are all supported in BMNet, it is very convenient to use command line to compile the network, otherwise you can refer to Chapter 3 to add customized layers.
Programing model
The BMNET library offers a set of API and tool to convert caffemodel into machine instructions, which is saved in bmodel file. The bmodel file also keeps more information of network model, such as network name, target name, shape, weight, etc.
Calibration Caffe Tool Guide
Introduction
● The tool would use caffe as inference framework and collect required statistics.
● Given a caffe prototxt and fp32 caffemodel can automatically generate both calibration table which contain calibration info and int8 caffemodel.
○ [net_name]: net name
○ [output_path]: output path of prototxt and int8 caffemodel.
○ [iteration batch size]: calibration batch size.
Input data layer
You can define the calibration data in the caffe prototxt. You can see
example in the deploy.prototxt.
The relative parameter is as following:
[data_list]: The file describes where to find the images. Each line is the path to image.
[h]: input data height
[w]: input data width
[color_format]: Specify which color format you want to use. Only
support RGB/BGR.
[r/g/b_mean]: RGB mean value. If > 0, all image will subtract the
mean value.
[scale]: The scale value of the data. It will multiply to the data after minus the r/g/b mean value. The default value is 1
[mirror]: Specify the data needs to mirror or not. The default value is 0.
0: no need to mirror
1: vertical and horizontal
2: vertical
3: horizontal
[transpose]: Specify the data transpose axises. The default is [2,0,1] (equal to [c,h,w] order).
[padding]: If true, padding is added when resize to the target size to keep the original aspect ratio.The default is false.
[debug]: Save each data as specific format in “debug” folder in the working directory. Note that it is saved before doing transpose. The default is None.
npy: Save as numpy array
image: Save as image
Custom model support
If you want to calibrate your own model, you can define ‘custom_model.json’. The relative parameter is as following:
[name]: net name
[in_prototxt]: input caffe prototxt file
[in_caffemodel]: input caffe model
The out file will be ‘bmnet_your_net_name_calibration_table.pb2’ and ‘bmnet_your_net_name_int8.caffemodel’
Example
Enter calibration_tool folder while you get the toolkit from
The bm_builder.bin combines frontend, optimizer and backend modules into one executable binary, and links to libbmnet.so. It takes network’s caffemodel and deploy.prototxt as inputs, and finally generates bmodel after compiled.
bm_builder.bin
-t or --target
Specify the name of BITMAIN target board (bm1880).
-n or --name
Specify the name of deep learning network.
-s or --shape
Specify the input shape of network. Dims should be separated by commas and no backspace is
allowed.
-u or --plugin
Specify the directories of cpu op plugins.
-c or --caffemodel
Specify the caffemodel generated by calibration_caffe.bin.
-m or --modified_proto
If you want to modify the prototxt, specify the modified deploy.prototxt of network.
-o or --out_model
Specify the output bmodel file.
-d or --in_ctable=input_ctable_file
Specify the calibration table generated by calibration_caffe.bin.
-e or --out_ctable=output_ctable_file
Specify the output of optimizer calibration table.
-p or --out_proto=output_prototxt_file
Specify the optimizer prototxt file of network.
--enable -weight -optimize=yes|no [no]
Specify the option(yes or no) to enable or disable optimization .
$ cd calibration_onnx_tool
$ export LD_LIBRARY_PATH=./lib/
$ bin/onnx_calibration.bin
-m ./Resnet50/resnet50.onnx
-d ./res/imagenet_partial/LSVRC-2012/
-o r50.cal.onnx
-i 10
-b 1
onnxmodel ./Resnet50/resnet50.onnx
dataset ./res/imagenet_partial/LSVRC-2012/
output model r50.cal.onnx
iteration 10
input batch size 1
parse arg ret: 0
<CMD> bin/onnx_calibration.bin -m ./Resnet50/resnet50.onnx -d ./res/imagenet_partial/LSVRC-2012/ -o r50.cal.onnx -i 10 -b 1
This version of onnx-caffe2 targets ONNX operator set version 7, but the model we are trying to import uses version 8. We will try to import it anyway, but if the model uses operators which had BC-breaking changes in the intervening versions, import will fail.
This version of onnx-caffe2 targets ONNX operator set version 7, but the model we are trying to import uses version 8. We will try to import it anyway, but if the model uses operators which had BC-breaking changes in the intervening versions, import will fail.
This version of onnx-caffe2 targets ONNX operator set version 7, but the model we are trying to import uses version 8. We will try to import it anyway, but if the model uses operators which had BC-breaking changes in the intervening versions, import will fail.
This version of onnx-caffe2 targets ONNX operator set version 7, but the model we are trying to import uses version 8. We will try to import it anyway, but if the model uses operators which had BC-breaking changes in the intervening versions, import will fail.
This version of onnx-caffe2 targets ONNX operator set version 7, but the model we are trying to import uses version 8. We will try to import it anyway, but if the model uses operators which had BC-breaking changes in the intervening versions, import will fail.
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0115 11:41:41.498390 21843 init.h:99] Caffe2 GlobalInit should be run before any other API calls.
W0115 11:41:41.500347 21843 init.h:99] Caffe2 GlobalInit should be run before any other API calls.
I0115 11:41:41.803261 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.803356 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.804275 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.804330 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.805279 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.805331 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.806298 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.806352 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.807260 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.807310 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.808189 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.808254 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.809103 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.809151 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.809998 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.810055 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.810925 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.810973 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
I0115 11:41:41.811820 21843 Calibration_legacy.cpp:111] pInputDims NCHW: 1 3 224 224
I0115 11:41:41.811872 21843 Calibration_legacy.cpp:119] datum CHW: 3 256 256
...
layer {
name: "res5c_1"
blob_param {
name: "res5c_1"
threshold_y: 33.7025
}
}
layer {
name: "res5c_relu_1"
blob_param {
name: "res5c_relu_1"
threshold_y: 33.7025
}
}
layer {
name: "pool5_1"
blob_param {
name: "pool5_1"
threshold_y: 14.885545
}
}
layer {
name: "OC2_DUMMY_106"
blob_param {
name: "OC2_DUMMY_106"
threshold_y: 14.885545
}
blob_param {
name: "OC2_DUMMY_1"
threshold_y: 0
}
}
layer {
name: "fc1000_1"
blob_param {
name: "fc1000_1"
threshold_y: 28.184326
}
}
layer {
name: "prob_1"
blob_param {
name: "prob_1"
threshold_y: 0.062530234
}
You can see the the file r50.cal.onnx be generated.
The bm_builder_onnx.bin combines frontend, optimizer and backend modulesinto one executable binary, and links to libbmnet.so. It takes network’scalibrated .onnx model as inputs, and finally generates bmodel after compiled.
