BMNet Compiler

Introduction

The BMNET library is designed to convert the neural networks defined by CAFFE to target instructions. It seems like a compiler which translates high-level language into machine instruc- tions. It also contains three phase which are the front end, the optimizer and the back end. The front end parses source code, extracts network prototxt and weights. The optimizer is responsible for doing a broad variety of transformations to try to improve the code’s running time. The back end (also known as the code generator) then maps the code onto the target instruction set. In the BM1880 platform,we add a new feature called INT8 computation,it can provide better performance such as inference speedup. INT8 computation need an calibration table to modify network parameter,you can refer section 2 for how to generate a network’s calibration table. Refer this document,you can convert a network from FP32 to INT8 without significant accuracy loss.

General Description

We provide multiple utility tools to convert CAFFE models into machine instructions. These instructions, as well as model’s weights, would be packed into a file named bmodel (model file for BITMAIN targets), which can be executed in BITMAIN board directly. BMNet has implemented many common layers, the full list of build-in layers is in below table, and many more layers are in developing:

Activation

BatchNorm

Concat


Convolution

Eltwise

Flatten

InnerProduct

Join

LRN

Normalize

Permute

Pooling

PReLU

PriorBox

Reorg

Reshape

Scale

Split

Upsample

If layers of your network model are all supported in BMNet, it is very convenient to use command line to compile the network, otherwise you can refer to Chapter 3 to add customized layers.

Programing model

The BMNET library offers a set of API and tool to convert caffemodel into machine instructions, which is saved in bmodel file. The bmodel file also keeps more information of network model, such as network name, target name, shape, weight, etc.

Calibration Caffe Tool Guide

Introduction

● The tool would use caffe as inference framework and collect required statistics.

● Given a caffe prototxt and fp32 caffemodel ​can automatically generate both​ calibration table which contain calibration info and int8 caffemodel.

Support network list

resnet18|resnet50|alexnet_winograd|unet|custom_model|alexnet|yolov3|yolov2|mobilenet|vgg16|googlenet|resnet50_winograd|lenet|det1|densenet|det2|resnet101|SSD_300x300|det3|googlenet_v3_winograd|googlenet_v3|resnet152|mobilenet_v2

Get calibration caffe toolkit

You can download the toolkit from:

and the files included in this toolkit as below, the CustomModel and the Resnet50 offered just for examples.

Usage

  • Put the prototxt and fp32 caffemodel in the directory of output_path.

  • The calibration table and int8 caffemodel would generate in the

    ouput_path, the script as following:

  • The relative parameter is as following:

    ○ [net_name]: net name ○ [output_path]: output path of prototxt and int8 caffemodel. ○ [iteration batch size]: calibration batch size.

Input data layer

  • You can define the calibration data in the caffe prototxt. You can see

    example in the deploy.prototxt.

  • The relative parameter is as following:

    • [data_list]: The file describes where to find the images. Each line is the path to image.

    • [h]: input data height

    • [w]: input data width

    • [color_format]: Specify which color format you want to use. Only

      support RGB/BGR.

    • [r/g/b_mean]: RGB mean value. If > 0, all image will subtract the

      mean value.

    • [scale]: The scale value of the data. It will multiply to the data after minus the r/g/b mean value. The default value is 1

    • [mirror]: Specify the data needs to mirror or not. The default value is 0.

      • 0: no need to mirror

      • 1: vertical and horizontal

      • 2: vertical

      • 3: horizontal

    • [transpose]: Specify the data transpose axises. The default is [2,0,1] (equal to [c,h,w] order).

    • [padding]: If true, padding is added when resize to the target size to keep the original aspect ratio.The default is false.

    • [debug]: Save each data as specific format in “debug” folder in the working directory. Note that it is saved before doing transpose. The default is None.

      • npy: Save as numpy array

      • image: Save as image

Custom model support

If you want to calibrate your own model, you can define ‘custom_model.json’. The relative parameter is as following:

  • [name]: net name

  • [in_prototxt]: input caffe prototxt file

  • [in_caffemodel]: input caffe model

The out file will be ‘bmnet_​your_net_name​_calibration_table.pb2’ and ‘bmnet_​your_net_name_​int8.caffemodel’

Example

Enter calibration_tool folder while you get the toolkit from

You can download the dataset ILSVRC2012_val here: http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_val.tar

Describe your jpeg dataset path in deploy.prototxt

Calibration toolkit also support lmdb dataset, describe your lmdb-dataset path in deploy.prototxt

and the input.txt describe the path of jpeg files, content as below:

Now, you can start the Resnet50 caffemodel calibration

  • In the path of ​Resnet50/, you can see bmnet_​resnet50_​ calibration_table.1x10.pb2 and bmnet_​resnet50_​int8.1x10.caffemodel.

bm_builder.bin

Description

The bm_builder.bin combines frontend, optimizer and backend modules into one executable binary, and links to libbmnet.so. It takes network’s caffemodel and deploy.prototxt as inputs, and finally generates bmodel after compiled.

Get the bm_builder toolkit：

you need install the BMNNSDK usb mode on your ubuntu x86-64 host，how to install BMNSDK usb mode.

bm_builder.bin in path:

End-user Options

Example

please pay attention that parameter bm_builder.bin used above should be one line

resnet50.bmodel can be generated.

bmodel testing

How to test the bmodel you can follow the instructions below If you use the Edge Development Board SoC mode,

For NNS or usb mode of Edge Development Board, you can follow the instruction below:

Calibration ONNX Tool Guide

Description

Calibration tool for onnx model.

Get calibration ONNX tool

You can download the toolkit from:

https://sophon-file.bitmain.com.cn/sophon-prod/drive/19/00/21/11/BM1880_Calibration_Tools.zip

and the files included in this toolkit as below, the CustomModel and the Resnet50 offered just for examples.

Usage

Example

You can see the the file r50.cal.onnx be generated.

bm_build_onnx.bin

Description

The bm_builder_onnx.bin combines frontend, optimizer and backend modulesinto one executable binary, and links to libbmnet.so. It takes network’scalibrated .onnx model as inputs, and finally generates bmodel after compiled.

End-User Options

Example

You can see the r50_batch_1.bmodel generated:

bmodel testing

The testing method is same as the method that caffe bmodel used, you can visit the section above.

PreviousCaffeNextHeaders

Last updated