Client Background

Client: A leading tech consulting firm in the USA

Industry Type: IT

Products & Services: IT Consulting, IT Support, SaaS

Organization Size: 100+

The Problem

The client aimed to explore the development and deployment of custom TensorFlow Lite (TFLite) models on VOXL2 hardware. The goal was to leverage the advanced GPU and NPU acceleration capabilities of VOXL2 to optimize and benchmark these models for efficient on-device inference. This project was not only about showcasing the potential of VOXL2 in enhancing machine learning performance but also about contributing to the broader understanding of deploying custom models on edge devices.

Our Solution

  1. Load Base Model in ONNX Format: Started with loading a base model like YOLOv7 or YOLOv8 in ONNX format.
  2. Convert ONNX Models to TFLite Format: Used the onnx-tf parser for conversion.
  3. Quantize Models for VOXL2 Chips: Quantized models to float16 format for compatibility with VOXL2 chips.
  4. Clone VOXL SDK Developer Environment: Cloned the VOXL SDK developer environment and set up ADB.
  5. Connect VOXL2 Chip to Your Computer: Connected the VOXL2 chip and verified the connection.
  6. Access VOXL Chip Shell: Accessed the shell of the VOXL chip for model deployment and configuration.
  7. Create DEB Packages with Custom TFLite Models: Cloned the voxl-tflite-server repository, copied TFLite files, and configured the model.
  8. Use Custom DEB Package on VOXL2: Deployed the DEB package and configured the VOXL chip to run the model.
  9. Run voxl-tflite-server: Executed voxl-tflite-server to start the inference process.
  10. Verify Model Execution: Ensured the model runs without errors on the VOXL chip.

Solution Architecture

Steps were followed as referred in the Modal AI documentation. No solution architecture was required here.

Deliverables

  • A Python script implementing the CVRP-TW model.
  • Test data and scripts for simulating different scenarios.
  • Documentation explaining how to use the model and interpret the results.

Tech Stack

  • Tools used
  • ONNX
  • TensorFlow Lite
  • VOXL SDK
  • Android Debug Bridge (ADB)
  • Language/techniques used
  • Python
  • Shell scripting
  • Models used
  • YOLOv7 or YOLOv8 in ONNX format
  • Mobilenet
  • Skills used
  • Machine Learning model conversion and optimization
  • Edge device deployment and configuration
  • Performance benchmarking

What are the technical Challenges Faced during Project Execution

  1. Converting ONNX models to TFLite format for compatibility with the TFLite runtime on VOXL chips.
  2. Quantizing models to float16 format for compatibility with the GPU and DPU delegations on VOXL chips.
  3. Setting up the VOXL SDK developer environment and ensuring ADB is correctly configured.
  4. Deploying custom TFLite models on the VOXL chip and configuring it to run the models.
  5. Benchmarking the model using the voxl-logger tool and encountering issues with the latest SDK build.

How the Technical Challenges were Solved

  1. Used the onnx-tf parser for model conversion, ensuring compatibility.
  2. Quantized models to float16 format, improving inference speed and reducing model size.
  3. Cloned the VOXL SDK developer environment and followed the documentation to set up ADB.
  4. Cloned the voxl-tflite-server repository, copied TFLite files, and configured the model for deployment on the VOXL chip.
  5. Consulted with the VOXL forums and developers for alternative methods of benchmarking due to SDK build issues.
  6. Business Impact The successful deployment and benchmarking of custom TFLite models on VOXL2 chips have significantly enhanced the client’s ability to optimize machine learning performance on edge devices. By leveraging the advanced GPU and NPU acceleration capabilities of VOXL2, the client has been able to achieve efficient on-device inference, showcasing the potential of VOXL2 in the machine learning domain.

Business Impact

This project has not only contributed to the broader understanding of deploying custom models on edge devices but has also provided valuable insights into the performance of these models on VOXL chips. The process of overcoming technical challenges has further solidified the client’s confidence in the capabilities of VOXL2 and the potential of deploying custom TFLite models on edge devices.

Overall, the Manu B VOXL project has been a success, demonstrating the potential of VOXL2 in enhancing machine learning performance on edge devices and contributing to the broader understanding of deploying custom models on edge devices. The project has highlighted the importance of overcoming technical challenges and the value of comprehensive guides for deploying and benchmarking TFLite models on VOXL chips.

Project Snapshots


Project website url

Forum :: https://forum.modalai.com/topic/3103/need-help-simulating-tflite-yolo-models-on-my-linux-machine/4?_=1711183909164

Report :: https://docs.google.com/document/d/17qVUzjCz3UKWB_-0fuBJzZAbdOULbp2a1U7b2Il3V78/edit 

Summarize

Summarized: https://blackcoffer.com/

This project was done by the Blackcoffer Team, a Global IT Consulting firm.

Contact Details

This solution was designed and developed by Blackcoffer Team
Here are my contact details:
Firm Name: Blackcoffer Pvt. Ltd.
Firm Website: www.blackcoffer.com
Firm Address: 4/2, E-Extension, Shaym Vihar Phase 1, New Delhi 110043
Email: ajay@blackcoffer.com
Skype: asbidyarthy
WhatsApp: +91 9717367468
Telegram: @asbidyarthy

RELATED ARTICLESMORE FROM AUTHOR