One of the things I really enjoy is creating image processing solutions as you might notice from the projects I publish on Hackster. My most recent project looked at image processing from the photon to the display and how we could create a image processing pipelines.
There are many different image processing techniques which can be used in image processing. One of most commonly used techniques in image processing is to use a sliding window filter which sliders a n by n matrix across the image and preforms an operation on the centre pixel.
The operation which is performed on this image processing pixel can range from a simple delta function to more complex operations such as edge detection or edge enhancement.
I am a big fan of higher level design techniques such as High Level Synthesis and Matlab Simulink. For image processing they really accelerate the design and development of image processing algorithms as they automatically create the sliding window function which forms the basics of these filters.
As engineers while we leverage these higher level techniques we also need to be able to understand the under lying concepts and principals. Being able to implement aspects from scratch is also important when we are working with smaller more resource constrained devices.
One of the key elements of the sliding window filter is its ability to slide across, n by n pixels. This means if we are implementing a 3x3 sliding window filter we need to be able to buffer at least two full lines to be able to address filter sliding over the window.
In this blog we are going to look at how we can create a simple 3x3 window. The output from this sliding window will be the 9 pixel values of the grid. We can then look in another blog at what down stream processing on these values for example median filtering, edge enhancement etc.
As we want to make sure this works with our image processing designs we will be AXIS interfaces. We will also be developing the sliding window function to work with grey scale / pixel intensity only, not multiple channels for colour or YUV.
A common issue with sliding windows is how you address the pixel data around the edges of the image. To ensure the frame is the correct size we need to address them, one of the simplest methods is to pad with zero which is what I implemented. Other options include
We can write the RTL simply, and using generics we are able to make it flexible with image sizes.
To get started we can define the generics and the line buffers we will be using.
-- Line buffer type
type t_line_buffer is array (0 to g_IMG_WIDTH-1) of std_logic_vector(g_TDATA_WIDTH-1 downto 0);
-- Line buffers to store two rows of pixels signal s_line1, s_line2 : t_line_buffer;Once we get the line buffers set up the next step is to write the process, this process will provide the sliding functionality.
As we are using AXI Streaming, the module passes through the down stream back pressure. If the valid signal is asserted, and the start of a new frame is indicated by the user signal internal counters are reset.
The registers and line buffers are then updated correctly.
-- Shift window registers
s_r13 <= s_r12;
s_r12 <= s_r11;
s_r11 <= i_s_axis_tdata;
s_r23 <= s_r22;
s_r22 <= s_r21;
s_r21 <= s_line1(s_pixel_count);
s_r33 <= s_r32;
s_r32 <= s_r31;
s_r31 <= s_line2(s_pixel_count);If the pixel is the last one of a line, the pixel count for the line is reset.
if i_s_axis_tlast = '1' then
s_pixel_count <= 0;
if s_row_count = g_IMG_HEIGHT-1 then
s_row_count <= 0;
else
s_row_count <= s_row_count + 1;
end if;
else
s_pixel_count <= s_pixel_count + 1;
end if;The final element is then handling the edge cases where the pixel is padded 0x00 when the top or edges.
s_233 <= s_r22;
if s_row_count = 0 then
s_w11 <= (others => '0');
s_w12 <= (others => '0');
s_w13 <= (others => '0');
else
s_w11 <= s_r11 when s_pixel_count /= 0 else (others => '0');
s_w12 <= s_r12;
s_w13 <= s_r13 when s_pixel_count /= g_IMG_WIDTH-1 else (others => '0'); end if;
-- Middle row padding
s_w21 <= s_r21 when s_pixel_count /= 0 else (others => '0');
s_w23 <= s_r23 when s_pixel_count /= g_IMG_WIDTH-1 else (others => '0');
-- Bottom row padding
if s_row_count = g_IMG_HEIGHT-1 then
s_w31 <= (others => '0');
s_w32 <= (others => '0');
s_w33 <= (others => '0');
else
s_w31 <= s_r31 when s_pixel_count /= 0 else (others => '0');
s_w32 <= s_r32;
s_w33 <= s_r33 when s_pixel_count/= g_IMG_WIDTH-1 else (others => '0');
end if;With the RTL written the next stage is to create a test bench which will apply an image and check that it is passing through ok
When we do this we see very clearly the edges of the image being padded in simulation.
Now the output of this block are 9 pixels which are in a 3x3 grid, to be able to re constitute the image correctly we need to do some operation on these in another module for example edge enhancement or detection.
We will look at that in another blog soon, but now hopefully you know a little more about how image processing filters function and are developed.
The project is available on my github
Workshops and Webinars
If you enjoyed the blog why not take a look at the free webinars, workshops and training courses we have created over the years. Highlights include
Upcoming Webinars Timing, RTL Creation, FPGA Math and Mixed Signal
Professional PYNQ Learn how to use PYNQ in your developments
Introduction to Vivado learn how to use AMD Vivado
Ultra96, MiniZed & ZU1 three day course looking at HW, SW and PetaLinux
Arty Z7-20 Class looking at HW, SW and PetaLinux
Mastering MicroBlaze learn how to create MicroBlaze solutions
HLS Hero Workshop learn how to create High Level Synthesis based solutions
Perfecting Petalinux learn how to create and work with PetaLinux OS
Boards
Get an Adiuvo development board
Adiuvo Spartan 7 / RPi 2040 Embedded System Development Board
Adiuvo Spartan 7 Tile - Low Risk way to add a FPGA to your design.
Embedded System Book
Do you want to know more about designing embedded systems from scratch? Check out our book on creating embedded systems. This book will walk you through all the stages of requirements, architecture, component selection, schematics, layout, and FPGA / software design. We designed and manufactured the board at the heart of the book! The schematics and layout are available in Altium here Learn more about the board (see previous blogs on Bring up, DDR validation, USB, Sensors) and view the schematics here.
Comments