Description
This project demonstrates multi-camera YOLOv5 accelerated by the Hailo-8 on the Zynq UltraScale+. The setup has 4x Raspberry Pi cameras and can run on a few different Zynq UltraScale+ development platforms. Cameras are connected to the Zynq UltraScale+ development boards via the Opsero RPi Camera FMC. The Hailo-8 AI accelerator connects to the development board via the FPGA Drive FMC Gen4 or the M.2 M-key Stack FMC depending on the target design (see list of target designs).
A detailed description of this example design has also been written in the blog post Multi-camera YOLOv5 on Zynq UltraScale+ with Hailo-8 AI Acceleration.
Hardware Platforms
The hardware designs provided in this reference are based on Vivado and support a range of MPSoC evaluation boards. The repository contains all necessary scripts and code to build these designs for the supported platforms listed below:
Zynq UltraScale+ platforms
Target board |
FMC Slot Used |
Cameras |
Active M.2 Slots |
VCU |
|---|---|---|---|---|
LPC |
4 |
1 |
✅ |
|
HPC0+HPC1 |
4 |
1 |
✅ |
|
HPC0 |
4 |
2 |
✅ |
|
LPC |
2 |
1 |
❌ |
|
HPC |
4 |
2 |
✅ |
Software
These reference designs can be driven within a PetaLinux environment. The repository includes all necessary scripts and code to build the PetaLinux environment. The table below outlines the corresponding applications available in each environment:
Environment |
Available Applications |
|---|---|
PetaLinux |
Built-in Linux commands |
Architecture
The hardware design for these projects is built in Vivado and is composed of IP that implement the MIPI interface with the cameras and VCU as well as a display pipeline. For interface with the Hailo AI accelerator, the designs contain the XDMA IP. The main elements are:
4x Raspberry Pi cameras each with an independent MIPI capture pipeline that writes to the DDR
Video Mixer based display pipeline that writes to the DisplayPort live interface of the ZynqMP
Video Codec Unit (VCU)
DMA for PCIe (XDMA)
The block diagram below illustrates the design from the top level.
Capture pipeline
There are four main capture/input pipelines in this design, one for each of the 4x Raspberry Pi cameras. The capture pipelines are composed of the following IP, implemented in the PL of the Zynq UltraScale+:
The MIPI CSI-2 RX IP is the front of the pipeline and receives image frames from the Raspberry Pi camera over the 2-lane MIPI interface. The MIPI IP generates an AXI-Streaming output of the frames in RAW10 format. The ISP Pipeline IP performs BPC (Bad Pixel Correction), gain control, demosaicing and auto white balance, to output the image frames in RGB888 format. The Video Processing Subsystem IP performs scaling and color space conversion (when needed). The Frame Buffer Write IP then writes the frame data to memory (DDR). The image below illustrates the MIPI pipeline.
End-to-end pipeline
The end-to-end pipeline shows an example of the flow of image frames through the entire system, from source to sink. In the diagram, the image resolutions and pixel formats are shown at each interface between the image processing blocks. The resolution and pixel format can be dynamically changed at the output of the RPi camera and the scaler (Video Processing Subsystem IP).
Video Codec Unit
For some of the target boards, the Zynq UltraScale+ device contains a hardened Video Codec Unit (VCU) that can be used to perform video encoding and decoding of multiple video standards. On those target designs, we have included the VCU to enable these powerful features. Refer to the list of target designs to see which boards support this feature.