Dynamic Deep Learning Inference on the Edge

Project reference: 1914

Deep learning is an algorithmic technique that has found rapid adoption and application in a wide range of domains. In the deep learning workflow, the inference is one of the final steps where the trained models are deployed in the real world. For many applications, deep learning models are often deployed on the edge, i.e. on platforms that are remotely located. Furthermore, the inference devices may operate in low power envelopes and may not be readily accessible for updating the models deployed on them. Examples of such remote inference devices may be in industrial installations, solar/wind farms, weather monitoring stations or even satellites (in future). Therefore, it is essential that a method is developed to dynamically switch the deep learning models on the inference devices remotely.

The Intel Movidius Neural Compute Stick (NCS) a popular and powerful inference platform. The NCS is composed of a high-performance, energy-efficient inference processor (Myriad) that is packaged in a USB stick form factor. The Myriad processor is capable of real-time inference with a power envelope less than 1W. The NCS can be programmed using the Intel Movidius NCS Development Kit (NCSDK) or Intel OpenVINO primarily using Python.

In this project, we will develop a workflow and the core modules of an application that will offer the flexibility to dynamically switch deep learning models that are deployed on the Intel Movidius NCS. We will use pre-trained models that are available off-the-shelf. Models that are available include those for gender prediction from hand images, facial expression recognition and gesture recognition from hand images. The NCS will be interfaced with a Raspberry Pi kit, and the solution will be packaged into a publicly demonstrable end-user application.

A complete graph (see Section 9 below) (Publicly available image labeled for reuse).

Project Mentor: Venkatesh Kannan

Project Co-mentor: Myles Doyle

Site Co-ordinator: Simon Wong

Participant: Igor Kunjavskij

Learning Outcomes:

• Performing deep learning inference on the Intel Movidius Neural Computer Stick platform.
• Creating a publicly demonstrable end-user application with existing deep learning models.
• Exposure to larger projects in which the techniques and results of this project are integral.

Student Prerequisites (compulsory):

• Experience with Linux.
• Programming with Python.

Student Prerequisites (desirable):

• Basic knowledge of deep learning.
• Knowledge of working with Raspberry Pi.
• Creation of Docker images.

Training Materials:

1. Intel Movidius Neural Compute Stick          https://software.intel.com/en-us/movidius-ncs
2. Intel OpenVINO and Movidius Neural Compute SDK (NCSDK) https://software.intel.com/en-us/articles/transitioning-from-intel-movidius-neural-compute-sdk-to-openvino-toolkit

Workplan:

• Week 1: Induction, an overview of the project, introduction to relevant deep learning concepts, hardware, and software.
• Week 2: Example exercises for deep learning inference on the Intel Movidius Neural Compute Stick (NCS).
• Week 3: Design implementation methodology and write a project plan.
• Week 4: Implement an application to use a single deep learning model for inference on the NCS.
• Week 5: Extend the application to dynamically switch deep learning models on the NCS.
• Week 6: Package the application for portability (for example, Docker image) with publicly demonstrable user-interface.
• Week 7: Documentation of application and report writing.
• Week 8: Report writing and demonstration to ICHEC researchers.

Final Product Description:
A workflow and an end-user application that performs deep learning inference on the Intel Movidius Neural Compute Stick to dynamically switch the deep learning models.

Adapting the Project: Increasing the Difficulty:
The student could update/transmit the model to be loaded on the Intel Movidius NCS using a WiFi module on the Raspberry Pi kit instead of wired connectivity.

Adapting the Project: Decreasing the Difficulty:
The student could use a laptop/desktop to interface with the Intel Movidius NCS instead of the Raspberry Pi kit.

Resources:

1. Intel Movidius Neural Compute Stick (available at ICHEC)
2. Laptop (to be brought by the student)
3. Deep learning models and data sets (publicly available for use)
4. Optional Raspberry Pi kit (available at ICHEC)

Organisation:
Irish Centre for High-End Computing