Developments in FPGA software, custom-built processors, single-board computers and compact vision systems offer designers a variety of choices when building embedded vision systems.
With the advent of faster processors, FPGA-based intellectual property, custom image processors, open-source software and modular compact vision systems, designers are faced with numerous choices when developing embedded vision systems. However, although the definition of what comprises an embedded vision system can be loosely described as one that combines an embedded computer and a vision system, this can vary depending on the final product manufactured.
Designers of smart cellular mobile devices or vision systems for automotive applications, for example, require highly integrated, rugged and compact vision processors and image sensors. Alternatively, those involved in the development of medical imaging systems such as borescopes, may require a less-compact, board-level based system that can be integrated into end-user product housings. For those integrating systems for the machine vision market, a smart camera may be viewed as such an embedded machine vision system. Many developers of machine vision systems, however, think of such systems as rugged, expandable products that allow a variety of camera interfaces to be deployed using off-the-shelf software.
In developing products for any type of embedded vision system, developers must be aware of the cross-pollination that exists between various hardware and software products that are available. Developers of mobile devices, for example, can take advantage of embedded vision processors that incorporate multiple processing elements to perform imaging tasks. Similarly, camera and frame grabber designers can leverage the power of an FPGA vendor's intellectual property (IP) to perform functions such as camera standards interface conversion, Bayer interpolation and lens distortion correction.
While embedded vision processors may incorporate multiple very long instruction word (VLIW), reduced instruction set (RISC) CPUs and custom hardware to perform image processing functions, their designers have realized that programming such devices must be easy to use. The EV61, EV62 and EV64 embedded vision processors from Synopsys (Mountain View, CA, USA;), for example, combine a 32-bit scalar core with a DSP and an optimized convolutional neural network (CNN) engine. To speed software development, the devices are supported by a software programming environment based on OpenCV (http://opencv.org) and OpenVX (/openvx), a royalty-free standard for cross-platform acceleration of computer vision applications designed by the Khronos Group (Beaverton, OR, USA; ).
Similarly, the Myriad 2 processor from Movidius (San Mateo, CA, USA;www.movidius.com) also contains hybrid processing elements including twelve 128-bit VLIW processors and two 32-bit RISC processors. As well as a software development kit (SDK) that enables developers to incorporate proprietary functions, the company also supplies a reference design that includes a camera and micro-electro-mechanical (MEM) sensors (Figure 1).
|Figure 1: For developers using the Movidius' Myriad 2 processor, the company supplies an SDK and a reference design that includes a camera and micro-electro-mechanical (MEM) sensors.|
While individual processors can be incorporated into embedded systems, there are other options to accelerate these tasks. These include embedding either embedded "soft-core" processors or IP into the FPGAs used in such systems. Such products can prove useful for those developing such products as smart cameras, drones and FPGA-based frame grabbers.
One example of such soft-core IP processors is the VectorBlox MXP Matrix Processor from CEVA (Mountain View, CA; USA;www.ceva-dsp.com). As a scalable soft-core processor designed for FPGAs, the IP can be implemented in an FPGA as a plug-in IP block that implements parallel vector processor algorithms on the 2D and 3D matrices commonly found in image processing. According to CEVA, the soft-core processor core can enhance the performance of standard MicroBlaze processors from Xilinx (San Jose, CA, USA; www.xilinx.com) by a minimum of 1-2 orders of magnitude. Taking advantage of this, Altek (Hsinchu, Taiwan; .tw) recently licensed CEVA's IP to perform object detection and tracking and 3D depth sensing in the company's drones and smart cameras.
Just as developers of cameras and frame grabbers can leverage the power of soft-core processors, they can also take advantage of FPGA IP libraries to perform dedicated image processing tasks. These include camera interfacing, image pre-processing functions such as Bayer interpolation, image compression, stereo vision, face detection and motion detection.
While making camera interfacing easier, the advent of high-density FPGAs also allows many image pre-processing functions to be incorporated. While pre-processing algorithms such as Bayer interpolation, bad pixel correction and color balancing can be performed on a CPU, they are more effectively performed in an FPGA (see "CCD cameras embed imaging algorithms,"Vision Systems Design, May 2007; http://bit.ly/VSD-0507). In the past, developers of cameras and frame grabbers needed to develop these algorithms. Today, however, vendors offer IP libraries to perform these tasks.
CoSynth (Oldenburg, Germany;www.cosynth.com) offers several IP cores optimized for use in FPGAs that support image processing functions such as Bayer pattern demosaicing, color-space conversion, image segmentation and morphological operations.
For use with camera based systems, IP cores for the integration of cameras via LVDS and Ethernet are available. Similarly, Logic Bricks from Xylon (Zagreb, Croatia;www.logicbricks.com) provide image processing functions ranging from Bayer decoding (demosiacing), perspective transformation and lens correction and MJPEG decoding.
Just as many IP vendors implement basic image processing functions such as these, other companies (including Xylon) have realized the need to develop more application specific FPGA IP to speed the time to market of their customers' products. As well as providing such basic functions, Xylon has developed IP that performs tasks such as face detection and tracking, vehicle detection and vehicle driver drowsiness detection, all targeted at niche markets.
Xylon is not the only company to have developed such IP. For motion detection of ground, surface and aerial objects, Riftek (Minsk, Republic of Belarus, https://riftek.com) offers FPGA IP that allows both tracking and motion detection of objects. The company's T-COR-30 tracking core, for example, can be used in both Xilinx and Altera FPGAs.
For stereo vision applications, both Nerian Vision Technologies (Leinfelden-Echterdingen, Germany; ) and Fujisoft (Kanagawa, Japan;) both offer FPGA IP to perform stereo object matching, albeit using different algorithms.
After images are first rectified to compensate for lens distortions and camera alignment errors, Nerian Vision Technologies stereo vision core performs stereo matching on two grayscale input images by applying a variation of the Semi Global Matching (SGM) algorithm, a method originally developed by Heiko Hirschmueller of the German Aerospace Center (DLR; Wessling, Germany;). His original 2005 paper entitled "Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information," can be found at .
Other stereo methods can be used to perform this task, most notably the sum of absolute difference (SAD) algorithm that has been used by Dr. Keiji Saneyoshi, Associate Professor of Tokyo Institute of Technology (Meguro, Japan;). In conjunction with Altera, Fujisoft deploys this algorithm in its Stereo Vision IP Suite allowing stereo matching to be implemented in a combination of FPGA hardware and software on a pair of ARM Cortex-A9 CPUs both contained in a single Altera Cyclone V. (see "Expanding the Applications of Stereo Machine Vision," ).
Rather than use individual IP cores, developers can instead use graphical programming interfaces for programing FPGAs. These allow a number of image processing functions to be implemented in a data-flow manner. Two examples of such graphical programming interfaces are Visual Applets software from Silicon Software (Mannheim, Germany; https://silicon.software) and National Instruments' (Austin, TX, USA;www.ni.com) Vision Development Module that can be used with the company's LabVIEW FPGA Module to process images on an FPGA. Using these graphical interfaces functions such as pixel manipulation, image transforms, color space conversion and morphological operations can be graphically pipelined and embedded into FPGAs.
Just as frame grabbers and smart cameras may incorporate FPGAs, designers of embedded systems using board-level products such as off-the-shelf CPUs, frame grabbers and I/O peripherals are faced with an even wider range of products from which to choose. Here, numerous board-level products based on standards ranging from OpenVPX, VME, CompactPCI, cPCI Express, PC 104, PC/104 Plus, EPIC, EBX and COM Express boards can all be used to build vision systems with different camera interfaces and I/O options.
For developers of embedded systems that choose to use board-level products, many standards organizations offer valuable information and manufacturers' product listings. These include the VITA Standards Organization (VITA; Oklahoma City, OK, USA,) that has defined and developed the VMEbus (and its various incarnations), VPX, and mezzanine card standards such as the PCI Mezzanine Card (PMC), switched mezzanine card (XMC) and FPGA Mezzanine Card (FMC), all primarily for military and aerospace applications. As the latest extension of the VME standard, the VPX standard retains both 6U and 3U Eurocard form factors and supports PMC and XMC mezzanine cards. A listing of companies that support this standard can be found at http:///vpx.
For over a decade, numerous frame grabber and camera interface boards have been developed to support PMC mezzanine cards (see "PMC frame grabbers attack military market,"Vision Systems Design, January 2005; http://bit.ly/VSD-PMC). With the emergence of so called "switched fabric" interfaces, many of these camera interface boards have migrated to XMC designs. Like older PMC based designs, these support numerous types of camera interfaces (see "Frame grabbers target embedded imaging," Vision Systems Design, April 2014, http://bit.ly/VSD-FG).
Many of these support well-established camera-to-computer interface standards such as Camera Link as well as more recent standards such as CoaXPress. The Condor 4100 XMC Series from EIZO Rugged Solutions (Altamonte Springs, FL, USA;www.eizorugged.com), for example, are XMC designs based on AMD's Radeon E8860 GPU that support camera input standards such as 3G-SDI, HDMI, NTSC and CoaXPress (Figure 2).
|Figure 2: Supporting the switched mezzanine card (XMC) format, the Condor 4100 XMC Series from EIZO Rugged Solutions is based on AMD's Radeon E8860 GPU and support camera input standards such as 3G-SDI, HDMI, NTSC and CoaXPress (CXP).|
Similarly, the Peripheral Component Interconnect Special Interest Group (PCI-SIG; Beaverton, OR, USA; https://pcisig.com) is a consortium responsible for specifying the Peripheral Component Interconnect (PCI), PCI-X, and PCI Express (PCIe) and PCI Express Mini Card standards. Add-in boards that implement the PCIe standard include those based on CompactPCI and computer on Module Express (COM Express) that are defined by the PCI Industrial Computer Manufacturers Group (PICMG; Wakefield, MA, USA;www.picmg.org).
While numerous companies support the PCIe standard to allow cameras with disparate outputs to be interfaced to host computers, embedded systems designers looking to reduce the cost and size of their systems can now leverage the power of a number of small form factor PCI Express Mini Cards with which to perform this task (see "Frame grabbers add features and formats",Vision Systems Design, June 2015; ).
Companies that produce such PCI Express Mini Card boards include the AVC8000nano, an eight-channel NTSC/PAL frame grabber from Advanced Micro Peripherals (Witchford, UK;www.ampltd.com), the CM313BW from AVerMedia (New Taipei City, Taiwan; www.avermedia.com ) that supports a single 3G-SDI input, the HD-SDI VCE-HDmPCIe01 from Imperx (Boca Raton, FL, USA; www.imperx.com) and the PIXCI EB1mini Base Camera Link card from EPIX (Buffalo Grove, IL, USA; www.epixinc.com).
Other options exist for designers of embedded systems. Depending on the application to be formed, the use of Computer-On-Module (COM) boards offer developers a means to both implement embedded single board computers or use such processor modules with a carrier board that contains other application specific I/O. Typically, such COM Express boards consist of a processor, memory, graphics display capability and numerous types of I/O including Ethernet, USB 3 and VGA capability.
Developers using USB3 or Ethernet-based cameras can use such boards to build relatively inexpensive imaging systems using these interface standards. One example of such a COM Express board is the SOM-6898 from Advantech (Milpitas, CA, USA;www.advantech.com), a Type 6 COM Express Compact Module that features an Intel i7 processor, up to 32GBytes of memory using a small outline dual in-line memory module), Ethernet, USB and graphics interfaces (Figure 3). Where other camera interface standards are required, COM Express carrier boards can increase the functionality of the embedded vision system.
|Figure 3: COM boards offer designers a means to implement embedded single board computers or use such processor modules with a carrier board to increase system functionality. Advantech's SOM-6898 Type 6 COM Express Compact Module features an Intel i7 processor, 32GBytes of memory, Ethernet, USB and graphics interfaces and can be used stand-alone or with such carrier boards.|
For example, the Express-BASE6, an ATX size COM Express Type 6 carrier board from ADLINK Technology (New Taipei City, Taiwan;www.adlinktech.com), can be used with COM Express Type 6 modules and allows PCI Express x 1, x 4 or x 16 add-in cards to be added as required (). More information on the COM Express standard can be found at http://bit.ly/VSD-COMX and http://bit.ly/VSD-COMX2.
Small form factors
Other popular add-in boards that incorporate the PCIe bus for use in embedded systems include those based on the PCI/104-Express form factor. Specifications for these boards are controlled by The PC/104 Consortium (Los Gatos, CA, USA; ). Here again, numerous frame grabbers are available to capture images from a variety of camera interfaces (see "Small-Form-Factor Frame Grabbers Enable Embedded Applications,"Vision Systems Design February 2013;http://bit.ly/VSD-0213).
Like XMC mezzanine cards, PCI/104-Express support numerous types of analog and digital inputs including Camera Link and the CoaXPress (CXP) standards. Perhaps the first to support the CXP standard in this form factor, the Coaxlink Duo PCIe/104-EMB board from Euresys (Angleur,Belgium;) features two CoaXPress CXP-6 channels each with 1250 MBytes/s bandwidth (Figure 4).
|Figure 4: PCI/104-Express add-in boards support many types of analog and digital camera and interfaces. One of the first to support the CXP standard, the Coaxlink Duo PCIe/104-EMB board from Euresys features two CoaXPress CXP-6 channels each with 1250MBytes/s bandwidth.|
Just as designers using COM Express processor boards can expand their systems using carrier boards, developers employing frame grabbers can add processing and I/O capability to PCI/104-Express based systems using multiple PCI/104 Express modules and carrier boards. Here, two single-board form factors known as EPIC (Embedded Platform for Industrial Computing) and EBX (Embedded Board, eXpandable) allow different types of PC/104 modules to be added to a host single-board computer.
As an example, the Atlas PCI/104-Express SBC board from Diamond Systems (Sunnyvale, CA, USA;www.diamondsystems.com) is a PCI/104-Express single board computer with an Intel Atom N2800 CPU that allows PCI-104 and PCIe/104 cards to be added. In this way, vision systems using PCIe/104 frame grabbers can incorporate USB 2.0, Gigabit Ethernet, SATA and digital I/O functionality.
While the choices of which type of processor, IP, add-in board or vision system may at first seem overwhelming, several organizations aim to make the process easier. While the Embedded Vision Alliance (Walnut Creek, CA, USA;) provides information that focuses on processors, cameras, vision algorithms and development tools used to build embedded vision systems, the Imaginghub () recently launched by Basler (Ahrensburg, Germany; ) offers developers the chance to participate and create embedded projects.
Examples of such projects include how to create a face detection script with OpenCV and Python 3 (www.python.org) on a AIIS-1200P vision system from Advantech using a Basler dart camera. As part of the Basler Imaginghub, Dream Chip Technologies (Garbsen, Germany; ) has developed a ready to use Cyclone V-based BCON camera interface IP for easy integration into customers FPGA designs.
By using custom processors, FPGA-based software, off-the-shelf board-level products and open-source and commercially-available software, developers can build products faster and more cost-effectively. Because of such developments, embedded systems will be deployed more rapidly in such disparate applications as factory automation, medical imaging, military and consumer applications.