Function Testing

Last updated 3 months ago

Basic function testing

The Basic function testing use the network model(minist) and data include in the installation package for Neural Network inference, the test results will be output to the out subdirectory of the current directory, please make sure that the user has the write rights to the current directory.

$ source /opt/bmtap2/bm1880-<Mode>-x.y.z /test/test_bmnet.sh
$ mkdir out
$ run_test_bmnet_bm1880

Test test results are as follow:

$ run_test_bmnet_bm1880
begin to test net:mnist mnist <4,1,28,28>,epsilon:0.00001
<CMD>test_bmnet_bmodel /opt/bmtap2/bm1880-pcie_1.0.1/test/mnist/mnist_input_4_1_28_28.bin /opt/bmtap2/bm1880-pcie_1.0.1/test/mnist/mnist_4_1_28_28.bmodel out//mnist_output.bin 4 1 28 28 0.00001
input size:12544
WARRING:Logging before InitgoogleLogging()is written to STDERR
# run_test_bmnet_bm1880
begin to test net: mtcnn det1 <1,3,12,12>
<CMD> test_bmnet_bmodel /opt/bmtap2/bm1880-usb_1.0.1/test/../models/mtcnn/det1_input_1_3_12_12.bin /opt/bmtap2/bm1880-usb_1.0.1/test/../models/mtcnn/det1_1_3_12_12.bmodel out//det1_output.bin 1 3 12 12
input size: 432
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1026 19:41:03.865362 2635 bm_firmware.cpp:76] check firmware status ...
I1026 19:41:03.865628 2635 bm_firmware.cpp:82] firmware loading ...
I1026 19:41:03.929674 2635 bm_firmware.cpp:47] firmware load success
I1026 19:41:08.937716 2635 bm_firmware.cpp:98] firmware is running
I1026 19:41:08.937757 2635 bm_device.cpp:104] device[0] opened,gmem_size : 0x40000000
output size:6
outputs[0]: [1,4,1,1], "conv4-2"
outputs[1]: [1,2,1,1], "conv4-1"
test_bmnet_bmodel: load 107 us, run 106 us, read 76 us
('/opt/bmtap2/bm1880-usb_1.0.1/test/../models/mtcnn/det1_output_1_3_12_12_ref.bin', ' vs. ', 'out//det1_output.bin')
Result matched @ epsilon = 0.000000
det1<1,3,12,12> passed
begin to test net: mobilenet mobilenet <1,3,224,224>
<CMD> test_bmnet_bmodel /opt/bmtap2/bm1880-usb_1.0.1/test/../models/mobilenet/mobilenet_input_1_3_224_224.bin /opt/bmtap2/bm1880-usb_1.0.1/test/../models/mobilenet/mobilenet_1_3_224_224.bmodel out//mobilenet_output.bin 1 3 224 224
input size: 150528
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1026 19:41:09.092887 2662 bm_firmware.cpp:76] check firmware status ...
I1026 19:41:09.093133 2662 bm_firmware.cpp:85] firmware is running
I1026 19:41:09.093152 2662 bm_device.cpp:104] device[0] opened,gmem_size : 0x40000000
output size:1000
outputs[0]: [1,1000,1,1], "fc7"
test_bmnet_bmodel: load 3773 us, run 2820 us, read 222 us
('/opt/bmtap2/bm1880-usb_1.0.1/test/../models/mobilenet/mobilenet_output_1_3_224_224_ref.bin', ' vs. ', 'out//mobilenet_output.bin')
Result matched @ epsilon = 0.000000
mobilenet<1,3,224,224> passed
test_bmnet_bm1880 success

Network model testing

Network model testing depends on the BM1880 bmodel network packet bm1880_bmodel.tar.gz released with the BMNNSDK.please make sure that the network model packet has been unzipped and put into the system path .tar.gz.

$ sudo mkdir –p /data/release
$ sudo tar xvf bm1880_bmodels.tar.gz -C /data/release/

Network model testing will perform the neural network inference for all BM1880 bmodel network models, and the test results will be output to the out subdirectory of the currnet directory, pleas ensure that the user has the write permission for the current directory.

$ source /data/release/bm1880_bmodels/test_bmnet.sh
$ mkdir out$ run_test_bmnet_bm1880

Test results are as follow:

begin to test net: alexnet alexnet <4,3,227,227>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/alexnet/alexnet_input_4_3_227_227.bin /data/release/bm1880_bmodels/alexnet/alexnet.bmodel out//alexnet_output.bin 4 3 227 227
input size: 618348
output size:4000
outputs[0]: [1,1,4,1000], "fc8"
test_bmnet_bmodel: load 13310 us, run 16482 us, read 264 us
*** Similarity: 1.0 (4000/4000)
alexnet<4,3,227,227> passed
begin to test net: alexnet alexnet <8,3,227,227>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/alexnet/alexnet_input_8_3_227_227.bin /data/release/bm1880_bmodels/alexnet/alexnet.bmodel out//alexnet_output.bin 8 3 227 227
input size: 1236696
output size:8000
outputs[0]: [1,1,8,1000], "fc8"
test_bmnet_bmodel: load 26076 us, run 25016 us, read 484 us
*** Similarity: 1.0 (8000/8000)
alexnet<8,3,227,227> passed
begin to test net: alexnet alexnet <16,3,227,227>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/alexnet/alexnet_input_16_3_227_227.bin /data/release/bm1880_bmodels/alexnet/alexnet.bmodel out//alexnet_output.bin 16 3 227 227
input size: 2473392
output size:16000
outputs[0]: [1,1,16,1000], "fc8"
test_bmnet_bmodel: load 51738 us, run 42346 us, read 673 us
*** Similarity: 1.0 (16000/16000)
alexnet<16,3,227,227> passed
begin to test net: alexnet alexnet <32,3,227,227>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/alexnet/alexnet_input_32_3_227_227.bin /data/release/bm1880_bmodels/alexnet/alexnet.bmodel out//alexnet_output.bin 32 3 227 227
input size: 4946784
output size:32000
outputs[0]: [1,1,32,1000], "fc8"
test_bmnet_bmodel: load 103408 us, run 76893 us, read 957 us
*** Similarity: 1.0 (32000/32000)
alexnet<32,3,227,227> passed
begin to test net: resnet50 resnet50 <4,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/resnet50/resnet50_input_4_3_224_224.bin /data/release/bm1880_bmodels/resnet50/resnet50.bmodel out//resnet50_output.bin 4 3 224 224
input size: 602112
output size:4000
outputs[0]: [1,1,4,1000], "fc1000"
test_bmnet_bmodel: load 13020 us, run 48089 us, read 319 us
*** Similarity: 1.0 (4000/4000)
resnet50<4,3,224,224> passed
begin to test net: resnet50 resnet50 <8,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/resnet50/resnet50_input_8_3_224_224.bin /data/release/bm1880_bmodels/resnet50/resnet50.bmodel out//resnet50_output.bin 8 3 224 224
input size: 1204224
output size:8000
outputs[0]: [1,1,8,1000], "fc1000"
test_bmnet_bmodel: load 25558 us, run 93614 us, read 377 us
*** Similarity: 1.0 (8000/8000)
resnet50<8,3,224,224> passed
begin to test net: resnet50 resnet50 <16,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/resnet50/resnet50_input_16_3_224_224.bin /data/release/bm1880_bmodels/resnet50/resnet50.bmodel out//resnet50_output.bin 16 3 224 224
input size: 2408448
output size:16000
outputs[0]: [1,1,16,1000], "fc1000"
test_bmnet_bmodel: load 50495 us, run 186083 us, read 646 us
*** Similarity: 1.0 (16000/16000)
resnet50<16,3,224,224> passed
begin to test net: resnet50 resnet50 <32,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/resnet50/resnet50_input_32_3_224_224.bin /data/release/bm1880_bmodels/resnet50/resnet50.bmodel out//resnet50_output.bin 32 3 224 224
input size: 4816896
output size:32000
outputs[0]: [1,1,32,1000], "fc1000"
test_bmnet_bmodel: load 100280 us, run 369638 us, read 1090 us
*** Similarity: 1.0 (32000/32000)
resnet50<32,3,224,224> passed
begin to test net: vgg16 vgg16 <4,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/vgg16/vgg16_input_4_3_224_224.bin /data/release/bm1880_bmodels/vgg16/vgg16.bmodel out//vgg16_output.bin 4 3 224 224
input size: 602112
output size:4000
outputs[0]: [1,1,4,1000], "fc8"
test_bmnet_bmodel: load 13250 us, run 156856 us, read 313 us
*** Similarity: 1.0 (4000/4000)
vgg16<4,3,224,224> passed
begin to test net: vgg16 vgg16 <8,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/vgg16/vgg16_input_8_3_224_224.bin /data/release/bm1880_bmodels/vgg16/vgg16.bmodel out//vgg16_output.bin 8 3 224 224
input size: 1204224
output size:8000
outputs[0]: [1,1,8,1000], "fc8"
test_bmnet_bmodel: load 25711 us, run 297896 us, read 466 us
*** Similarity: 1.0 (8000/8000)
vgg16<8,3,224,224> passed
begin to test net: vgg16 vgg16 <16,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/vgg16/vgg16_input_16_3_224_224.bin /data/release/bm1880_bmodels/vgg16/vgg16.bmodel out//vgg16_output.bin 16 3 224 224
input size: 2408448
output size:16000
outputs[0]: [1,1,16,1000], "fc8"
test_bmnet_bmodel: load 50484 us, run 579585 us, read 510 us
*** Similarity: 1.0 (16000/16000)
vgg16<16,3,224,224> passed
begin to test net: vgg16 vgg16 <32,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/vgg16/vgg16_input_32_3_224_224.bin /data/release/bm1880_bmodels/vgg16/vgg16.bmodel out//vgg16_output.bin 32 3 224 224
input size: 4816896
output size:32000
outputs[0]: [1,1,32,1000], "fc8"
test_bmnet_bmodel: load 101112 us, run 1143342 us, read 1093 us
*** Similarity: 1.0 (32000/32000)
vgg16<32,3,224,224> passed
begin to test net: googlenet googlenet <4,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/googlenet/googlenet_input_4_3_224_224.bin /data/release/bm1880_bmodels/googlenet/googlenet.bmodel out//googlenet_output.bin 4 3 224 224
input size: 602112
output size:4000
outputs[0]: [1,1,4,1000], "loss3/classifier"
test_bmnet_bmodel: load 13103 us, run 19363 us, read 322 us
*** Similarity: 1.0 (4000/4000)
googlenet<4,3,224,224> passed
begin to test net: googlenet googlenet <8,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/googlenet/googlenet_input_8_3_224_224.bin /data/release/bm1880_bmodels/googlenet/googlenet.bmodel out//googlenet_output.bin 8 3 224 224
input size: 1204224
output size:8000
outputs[0]: [1,1,8,1000], "loss3/classifier"
test_bmnet_bmodel: load 25783 us, run 38983 us, read 488 us
*** Similarity: 1.0 (8000/8000)
googlenet<8,3,224,224> passed
begin to test net: googlenet googlenet <16,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/googlenet/googlenet_input_16_3_224_224.bin /data/release/bm1880_bmodels/googlenet/googlenet.bmodel out//googlenet_output.bin 16 3 224 224
input size: 2408448
output size:16000
outputs[0]: [1,1,16,1000], "loss3/classifier"
test_bmnet_bmodel: load 50612 us, run 76853 us, read 686 us
*** Similarity: 1.0 (16000/16000)
googlenet<16,3,224,224> passed
begin to test net: googlenet googlenet <32,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/googlenet/googlenet_input_32_3_224_224.bin /data/release/bm1880_bmodels/googlenet/googlenet.bmodel out//googlenet_output.bin 32 3 224 224
input size: 4816896
output size:32000
outputs[0]: [1,1,32,1000], "loss3/classifier"
test_bmnet_bmodel: load 101090 us, run 152487 us, read 1108 us
*** Similarity: 1.0 (32000/32000)
googlenet<32,3,224,224> passed
begin to test net: mobilenet mobilenet <4,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/mobilenet/mobilenet_input_4_3_224_224.bin /data/release/bm1880_bmodels/mobilenet/mobilenet.bmodel out//mobilenet_output.bin 4 3 224 224
input size: 602112
output size:4000
outputs[0]: [4,1000,1,1], "fc7"
test_bmnet_bmodel: load 13157 us, run 8580 us, read 228 us
*** Similarity: 1.0 (4000/4000)
mobilenet<4,3,224,224> passed
begin to test net: mobilenet mobilenet <8,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/mobilenet/mobilenet_input_8_3_224_224.bin /data/release/bm1880_bmodels/mobilenet/mobilenet.bmodel out//mobilenet_output.bin 8 3 224 224
input size: 1204224
output size:8000
outputs[0]: [8,1000,1,1], "fc7"
test_bmnet_bmodel: load 25801 us, run 16232 us, read 467 us
*** Similarity: 1.0 (8000/8000)
mobilenet<8,3,224,224> passed
begin to test net: mobilenet mobilenet <16,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/mobilenet/mobilenet_input_16_3_224_224.bin /data/release/bm1880_bmodels/mobilenet/mobilenet.bmodel out//mobilenet_output.bin 16 3 224 224
input size: 2408448
output size:16000
outputs[0]: [16,1000,1,1], "fc7"
test_bmnet_bmodel: load 50939 us, run 31122 us, read 676 us
*** Similarity: 1.0 (16000/16000)
mobilenet<16,3,224,224> passed
begin to test net: mobilenet mobilenet <32,3,224,224>
<CMD> test_bmnet_bmodel /data/release/bm1880_bmodels/mobilenet/mobilenet_input_32_3_224_224.bin /data/release/bm1880_bmodels/mobilenet/mobilenet.bmodel out//mobilenet_output.bin 32 3 224 224
input size: 4816896
output size:32000
outputs[0]: [32,1000,1,1], "fc7"
test_bmnet_bmodel: load 101190 us, run 61165 us, read 1076 us
*** Similarity: 1.0 (32000/32000)
mobilenet<32,3,224,224> passed
test_bmnet_bm1880 success

Network compilation testing

Network compilation tesing depends on the network packet bmnet_models.tar.gz released with the BMNNSDK.please make sure that the network model packet has been unzipped and put into the system path: /data/release/:

$ sudo mkdir –p /data/release
$ sudo tar zxvf bmnet_models.tar.gz -C /data/release/

The network compilation tesing will compile the supported network model, generate bmodel, and perform neural network inference, the test results will be output to the out subdirectory of the currnet directory, pleas ensure that the user has the write permission for the current directory.

$ source /opt/bmtap2/bm1880-<Mode>-x.y.z /test/regression.sh
$ run_regression_bmnet_bm1880

Test results are as follow:

...
begin to test net: googlenet googlenet <4,3,224,224>
/home/cgitb1808230052/hinton/workspace/out /home/cgitb1808230052/hinton/workspace
<CMD> bm_builder.bin -t bm1880 -n googlenet -c /data/release/bmnet_models//googlenet/bmnet_googlenet_int8.1x10.caffemodel --in_ctable=/data/release/bmnet_models//googlenet/bmnet_googlenet_calibration_table.1x10.pb2 --out_ctable=googlenet_ctable_opt.pb2 --enable-weight-optimize=yes -u /usr/lib -s 4,3,224,224 -p googlenet_frontend_opt.proto -o googlenet_4_3_224_224.bmodel
tg_lrn param:
input_shape 4 192 56 56 output_shape 4 192 56 56
/data/release/bmnet_models//googlenet/bmnet_googlenet_calibration_table.1x10.pb2 loss3/classifier
/data/release/bmnet_models//googlenet/bmnet_googlenet_calibration_table.1x10.pb2 data
<CMD> test_bmnet_bmodel /data/release/bmnet_models//googlenet/googlenet_input_4_3_224_224.bin googlenet_4_3_224_224.bmodel googlenet_output_4_3_224_224_int8.bin 4 3 224 224
input size: 602112
output size:4000
outputs[0]: [1,1,4,1000], "loss3/classifier"
test_bmnet_bmodel: load 12746 us, run 19544 us, read 260 us
*** Similarity: 1.0 (4000/4000)
/home/cgitb1808230052/hinton/workspace
googlenet<4,3,224,224> passed
...

Pay attention please, as the performance, the bmnet tools can’t run on the SoC mode, so the network regression testing can’t run directly on the SoC development board, you need to generate the bmodel on x86 host firstly, then copy it to the development board and test.

Python API testing

BMNNSDK provide Python Wrapper,and you can use the python interface for deeplearning application development:

// bmnet inference test
$ python /opt/bmtap2/bm1880-<MODE>_x.y.z/python/test/test_mobilenet.py

The testing results are as follows:

# python /opt/bmtap2/bm1880-usb_1.0.1/python/test/test_mobilenet.py
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1103 11:35:07.977723 21141 bm_firmware.cpp:76] check firmware status ...
I1103 11:35:07.977876 21141 bm_firmware.cpp:85] firmware is running
I1103 11:35:07.977897 21141 bm_device.cpp:104] device[0] opened,gmem_size : 0x40000000
input size 150528
output size 1000