Neural Network Module

Introduction

Product Overview

The Bitmain Sophon Neural Network Module (NNM) is a USB module that designed for Deep Learning inference on various edge application. NNM is powered by high performance, low power Sophon BM1880 chip. BM1880 chip supports DNN/CNN/RNN/LSTM models or uniquely trained networks, and can perform facial detection, recognition, facial expression analysis, object detection, recognition, vehicle license plate recognition, voiceprint recognition, etc. The NNM enables traditional product with AI functions, and can be used in smart IPC, Robots, industrial PC, etc.

Product Features

  • Power and all data provided over USB 9pin (pin width: 1.25mm) interface

  • Supports DNN/CNN/RNN/LSTM models profiling, compiling and tuning

  • Real-time inference in edge device Quickly deploy existing

  • DNN/CNN/RNN/LSTM models or uniquely trained networks Features

  • Sophon BM1880 with energy efficient DNN/CNN/RNN/LSTM processing

Target AI Functions

  • Facial detection (Frame by Frame), recognition and expression analysis, such as Age, Gender, etc.

  • Human attributes and pose analysis

  • Object detection and recognition

  • Vehicle license plate recognition

  • Voice print recognition

Product Technical Specification

Processor

Sophon BM1880

Supported Framework

Caffe, ONNX, Tensorflow, Pytorch

Supported AI Models

ResNet50, Yolo V2, GoogleNet V1, MobileNet v1&v2, SSD300, AlexNet, VGG16

H.264 decoder, MJPEG encoder/decoder

1x 1080p @60fps or 2x 1080p @30fps H.264 decoder, 75fps for FHD images

Connectivity

USB 9pin (pin width: 1.25mm) interface

Dimensions

38*38 mm

Operating environmental temperature

0 – 40C (commercial level)

Hot plugin/plugoff

Yes

Certification

EMC/FCC/3C/CE

Minimum system requirement

X86_64 computer running Windows or Ubuntu USB3.0 or USB2.0 port

Toolkit

  • BMNet: Bitmain Compiler which can convert supported AI models to internal format accelerated by Sophon TPU.

  • ONNX: Compiler which can convert ONNX format to internal format accelerated by Sophon TPU.

  • Quantization Tool: Convert FP32 to INT8 and support calibration function.

Get Started

You can freely downloaded the SDK package at BitMain website https://sophon.cn/drive/36.html

SDK Content Released package includes:

  • bm_update.img

  • bm_programmer.zip

  • bm1880-system-sdk

  • Demo program

Please check below links for more information.

Setup Neural Network Module

Setup the Neural Network Module (NNM) by following the steps provided below.

Step 1: Preparation

Before you start, make sure you have the following:

Step 2: Hardware Installation

Connect BM1880 USB Module to the host via USB Type A interface, the HOST which is running Ubuntu 16.04 operation system will identify the USB module automatically, it will create one USB device node named /dev/ttyACM0 if connection succeed. You can check it by below command:

$ ls -al /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 十二 12 18:03 /dev/ttyACM0

Step 3: Bring Up the Module

Please connect the UART cable to module as below:

Before starting downloading, it must install pyusb first on Ubuntu 16.04.

$ sudo apt-get install python-pip
$ pip install pyusb

Now we can start to download image to BM1880 USB module.

$ mkdir bm_tool
$ cd bm_tool
$ wget https://sophon-file.bitmain.com.cn/sophon-prod/drive/18/11/14/17/bm_programmer.zip
$ unzip bm_programmer.zip
$ cd bm_programmer
$ wget https://sophon-file.bitmain.com.cn/sophon-prod/drive/18/11/14/17/bm_update.img.zip
$ unzip bm_update.img.zip
$ sudo python2.7 bm1880_usb_dl.py

It will show:

Waiting for BM1880 USB port: /​

Plug BM1880 into PC, and it will start downloading.

bm_dl_magic.bin is 128 bytes \
Send to address 0x4003000
--- 0.01 Seconds ---
--- 0.0062141418457 Seconds ---
prg.bin is 81272 bytes
Send to address 0x4003000
--- 0.15 Seconds ---
--- 0.1264564991 Seconds ---
u-boot.bin is 346159 bytest: \
Send to address 0x108000000
--- 0.0 Seconds ---
--- 8.70227813721e-05 Seconds ---
bm_update.img is 558704128 bytes
Send to address 0x110000000
--- 4.62 Seconds ---
--- 0.0663626194 Seconds ---
Download complete
--- Reconnect 0.0 Seconds ---
--- 23.78 Seconds ---

It can check with BM1880 UART output.

U-Boot 2017.07-00005-g08e7a7b (Dec 10 2018 - 10:06:31 +0800) bitmain_bm1880, Build: jenkins-bm1880_nnm-24
DRAM: 236 MiB
Relocation Offset is: 06b65000
Relocating to 10eb65000, new gd at 10e2e4e10, sp at 10e2e27e0
MMC: sdhc@50100000: 0, sdhc@50101000: 1
Using default environment
In: serial
Out: serial
Err: serial
Net: No ethernet found.
Hit any key to stop autoboot: 0
Total of 1 word(s) were the same
start_addr:0x110000000
header->magic:bm_pack
header->item_num:0x8
header->blksz:0x200
header->hdr_crc32:0xcfb9bc37
Check package header CRC...
calc crc32: 0xcfb9bc37
crc compare done!
item->name:GPT
item->size:0x4400
item->offset:0x200
item->md5:02f619c274558dbef989bddff3b66
GPT start...
GPT: blks:34, item_addr:0x110000200
done...
item->name:FIP
item->size:0x6a7c3
item->offset:0x4800
item->md5:bc6aa7753bfcd826c1dbb86afdc5452
FIP start...
sdhci_send_command: MMC: 0 busy timeout increasing to: 200 ms.
sdhci_send_command: MMC: 0 busy timeout increasing to: 400 ms.
done...
item->name:BOOT
item->size:0xb5136e
item->offset:0x6f000
item->md5:9c5515fefda35e5d5ecc06f5d6f834e
BOOT start...
done...
item->name:RECOVERY
item->size:0x4e5b56e
item->offset:0xbc0400
item->md5:f440e0ee24735f47a2446b75016c095
RECOVERY start...
done...
item->name:ROOTFS

After done, it will boot into recovery mode. Type below command to boot into normal mode.

$ reboot -f​

And it shows messages about the initialization of FD & FR

[ 10.953805] jpu: loading out-of-tree module taints kernel.
[ 10.959814] jpu_init
[ 10.962104] jpu_probe
[ 10.964402] [JPUDRV] : jpu base address get from platform driver base addr=0x50070000, virtualbase=0xffffff80089ea000, size=768
[ 10.975875] [JPUDRV] : jpu control reg address get from defined base addr=0x50008008, virtualbase=0xffffff80089ec008, size=4
[ 10.987093] [JPUDRV] : fail to get clock controller, but, do not treat as error,
[ 10.994573] jpu irq number = 107
[ 10.997803] [JPUDRV] : jpu irq number get from platform driver irq=0xf
[ 11.004489] [JPUDRV] success to probe jpu device with reserved video memory phys_addr=0x130000000, base = 0xffffff8010000000, size = 0x8000000
[ 11.017651] [JPUDRV] success to probe jpu device with reserved video memory phys_addr=0x130000000, base = 0xffffff8010000000
[ 11.029154] end jpu_init result=0x0
chown: unknown user/group app:app
jdi_init(): fd = 4
jdi_get_instance_pool
iflag: expect: 0, actual: 0
cflag: expect: 18bf, actual: 8bd
lflag: expect: ffffffe4, actual: ffffffe4
oflag: expect: fffffffe, actual: fffffffe
in_baud: expected: 4111, actual: 13
out_baud: expected: 4111, actual: 13
get dev handle successful!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0101 08:00:11.714221 135 BmivaBmodel.cpp:29] Bmiva Bmodel init done!
I0101 08:00:11.714693 135 BmivaDevice.cpp:117] Bmiva device init done!
I0101 08:00:11.716369 135 BmivaBmodel.cpp:42] BmivaBmodel::RegisterBmnet path/usr/data/bmodel/tiny_ssh.bmodel
I0101 08:00:11.747866 135 BmivaBmodel.cpp:42] BmivaBmodel::RegisterBmnet path/usr/data/bmodel/det3.bmodel
I0101 08:00:11.762814 135 BmivaBmodel.cpp:42] BmivaBmodel::RegisterBmnet path/usr/data/bmodel/bmface.bmodel
I0101 08:00:12.627129 135 BmivaRecognizer.cpp:44] Bmiva recognizer initilized, model inex 2

Build and Run Demo Program

Step 1: Install OpenCV 3.4.0

To build demo program, please make sure OpenCV 3.4.0 with libgtk2.0-dev is already installed. If not, follow below commands to finish OpenCV installation.

$ sudo apt-get install pkg-config
$ sudo apt-get install libgtk2.0-dev
$ mkdir opencv_src
$ cd opencv_src
$ wget https://github.com/opencv/opencv/archive/3.4.0.zip
$ unzip 3.4.0.zip
$ cd opencv-3.4.0/
$ mkdir build
$ cd build/
$ cmake ..
$ make -j4
$ sudo make install

Step 2: Setup BM1880_AI_SDK

To get BM1880_AI_SDK:

$ git clone https://github.com/BM1880-BIRD/bm1880-system-sdk.git

BM1880_AI_SDK is located in bm1880-sdk-v1.0/middleware.

Step 3: Build demo program

You can get demo source code by below command:

$ git clone https://github.com/BM1880-BIRD/bm1880-nnm-nne-development.git

Enter the folder of demo program and modify CMakeLists.txt.

$ cd bm1880-nnm-nne-development
$ vi CMakeLists.txt

Set AI_SDK_PATH to BM1880_AI_SDK folder, it should be bm1880-system-sdk/middleware.

For example:

set (AI_SDK_PATH /home/bmuser/bmprj/bm1880-system-sdk/middleware)

It is ready to build the application:

$ mkdir build
$ cmake -DTARGET=HOST . -Bbuild/host
$ make -C build/host

And the binary will be in build/host/bin.

Step 4: Run Demo Program

  • Plug UVC webcam into PC.

  • Plug BM1880 USB module into PC, it needs about 15 sec for FD&FR initialization.

  • In PC, check the file is created automatically

/dev/ttyACM0
  • Run command below, input an ID and press enter for the registration. After done, press Esc to exit.

sudo ./register_face
  • Run command below , And then check the screen. It show the results.

sudo ./usb_host