Benchmarking TF-TRT on the Raspberry Pi and Jetson Nano

By -

Trying to choose between the Pi 4B and the Jetson Nano for a Deep Learning project?

I recently posted some results from benchmarks I ran training and running TensorFlow networks on the Raspberry Pi 4 and Jetson Nano. They generated a lot of interest, but some readers questioned their relevance. They were'n interested in training networks on edge devices.

Most people expect to train on higher-power hardware and then deploy the trained networks on  the Pi and Nano. If they use TensorFlow for training, they have are several choices for deployment:

  1. Standard TensorFlow
  2. TensorFlow Lite
  3. TF-TRT (a TensorFlow wrapper around NVIDIA's TensorRT, or TRT)
  4. Raw TensorRT
In this post I'll focus on timing Standard TensorFlow and TF-TRT. In a later post I plan to cover TensorFlow Lite on the Pi with and without accelerators like the Coral EDGE TPU coprocessor and the Intel Compute Stick.

I've run a number of benchmarks, and the results have been much as I expected.
I did encounter one surprising result, though, which I'll talk about at the end of the post. It's a pitfall that could easily have invalidated the figures I'm about to share.

Benchmarking MobileNet V2


The results I'll report are based on running MobileNetV2 pre-trained with ImageNet data. I've adapted the code from the excellent DLology blog which covers deployment to the Nano. I've also deployed the model on the Pi using a hacked community build of TensorFlow, obtained from here.  That has a wheel containing TF-TRT for python3.7, which is the default for Raspbian Buster.

(The wheel seems to have a minor bug. This weekend I'll set up a GitHub repo with the sed script I used to work around that, along with the notebooks I used to run the tests.)

The Pi 4B has 4GB of RAM, running a freshly updated version of Raspbian buster.

So here are the results you've been waiting for, expressed in seconds per image and frames per second (FPS).

Platform        Software    Seconds/image   FPS
Raspberry Pi    TF          0.226            4.42
Raspberry Pi    TF-TRT      0.20             5.13
Jetson Nano     TF          0.082           12.2
Jetson Nano     TF-TRT      0.04            25.63

According to these figures, the Nano is three to five times faster than the Pi, and TF-TRT is about twice as fast as raw TensorFlow on the Nano.

TF-TRT is only slightly faster than raw TensorFlow on the Pi. I'm not sure why this should be, but the timings are pretty consistent. At some stage I'll run some other models, but those will have to do for now.

A benchmarking pitfall


jtop
I mentioned one pitfall. When I re-ran the tests for this blog post I got much  slower performance for the Nano using TF-TRT - around 5 fps.

Fortunately Raffaello Bonghi's excellent jtop package saved the day.  jtop is an enhanced version of top for Jetsons which shows real-time GPU and memory usage.

Looking at its output,  I realised that an earlier session on the Nano was still taking up memory. Once I'd closed the session down, a re-run gave me the 25 fps which I and others had seen before.

I continue to be impressed by the Pi 4 and the Nano.

While the Nano's GPU offers significantly faster performance on Deep Learning tasks, it cost almost twice as much. Both represent excellent value for money, and your choice will depend on the requirements for your project.

Comments

Post a Comment

Popular posts from this blog

Controlling a Raspberry Pi Pico remotely using PySerial

By -
Image
Update : I'm using thie code below in another project, and found that I had not correctly fixed the reported bug. The new version passes automated tests, and I am pretty sure it works OK. I have changed the name of the class to Talker  since it can both send and receive information. Apologies to all concerned for the bug! Introduction You can use a Raspberry Pi Pico as a powerful peripheral to a host - a Raspberry Pi, a Jetson Nano, a laptop or workstation. In this article you'll see how to interact with a Pico running MicroPython or CircuitPython by writing some Python code that runs on the host. The software is easy to use. It enables you to send a Python statement to the Pico and read the results. The statement can be any valid MicroPython code. Setting up the host and the Pico For this article I've used a Raspberry Pi as the host, but any computer running Windows, Linux or Mac OS will do so long as it has Python 3.5 or later installed. In particular, you can use this t
Read more

Five steps to connect Jetson Nano and Arduino

By -
Image
Yesterday's post showed how to link a micro:bit to the Jetson Nano. One of the members of the (unofficial) NVIDIA Jetson Nano group on Facebook asked about connecting an Arduino to the Jetson. Here's a simple recipe for getting data from the Arduino to the Jetson Nano. It should work on all the Jetson models, not just the Nano, but I only have Nanos to hand. On request, I've added a recipe at the end of this post which sends data from the Jetson Nano to the Arduino; it turns the default LED on the Arduino on or off. The recipe for sending data from the Arduino to the Jetson has just 5 stages: Program the Arduino. (I used the ASCIITable example). Connect the Arduino to the Jetson using a USB connector Install pyserial on the Jetson Download a three-line Python script Run the script.   Programming the Arduino I used an Arduino Uno, and checked it on a verteran Duemilanove (above), but any Arduino should work. You'll need to do this step usi
Read more

Raspberry Pi Pico project 2 - MCP3008

By -
Image
This post show you how to drive the MCP3008 8-channel ADC using a Raspberry Pi Pico. I'll start with a personally embarrassing and annoyingly relevant story. I'll also tell you about a minor 'gotcha' when using SPI on the Pico, and how you can avoid it. Finally, I've include the code and fritzing diagram I used to test the interface. First, the story. Back in 2012, when the Raspberry Pi was fresh and new, I was running a startup called Quick2Wire.  We made add-on boards for the Pi. Just before Christmas we sent out our first batch of boards to a small army of beta-testers. The beta boards didn't work. Somehow a single trace on the pcb had got deleted and one of the GPIO pins was isolated. The boards were still usable and it wasn't too hard to solder in a jumper wire to replace the missing trace, but it was very embarrassing. It wasn't just that we'd shipped boards with a defect. The really annoying thing was that I'd set up an  automated loop-bac
Read more