Fast reading from the raspberry camera with Python, Numpy, and OpenCV. See the comments for more details.

raspberry_fast_capture.py

# Fast reading from the raspberry camera with Python, Numpy, and OpenCV
# Allows to process grayscale video up to 124 FPS (tested in Raspberry Zero Wifi with V2.1 camera)
#
# Made by @CarlosGS in May 2017
# Club de Robotica - Universidad Autonoma de Madrid
# http://crm.ii.uam.es/
# License: Public Domain, attribution appreciated

import cv2
import numpy as np
import subprocess as sp
import time
import atexit

frames = [] # stores the video sequence for the demo
max_frames = 300

N_frames = 0

# Video capture parameters
(w,h) = (640,240)
bytesPerFrame = w * h
fps = 250 # setting to 250 will request the maximum framerate possible

# "raspividyuv" is the command that provides camera frames in YUV format
#  "--output -" specifies stdout as the output
#  "--timeout 0" specifies continuous video
#  "--luma" discards chroma channels, only luminance is sent through the pipeline
# see "raspividyuv --help" for more information on the parameters
videoCmd = "raspividyuv -w "+str(w)+" -h "+str(h)+" --output - --timeout 0 --framerate "+str(fps)+" --luma --nopreview"
videoCmd = videoCmd.split() # Popen requires that each parameter is a separate string

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE) # start the camera
atexit.register(cameraProcess.terminate) # this closes the camera process in case the python scripts exits unexpectedly

# wait for the first frame and discard it (only done to measure time more accurately)
rawStream = cameraProcess.stdout.read(bytesPerFrame)

print("Recording...")

start_time = time.time()

while True:
    cameraProcess.stdout.flush() # discard any frames that we were not able to process in time
    # Parse the raw stream into a numpy array
    frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
    if frame.size != bytesPerFrame:
        print("Error: Camera stream closed unexpectedly")
        break
    frame.shape = (h,w) # set the correct dimensions for the numpy array

    # The frame can be processed here using any function in the OpenCV library.

    # Full image processing will slow down the pipeline, so the requested FPS should be set accordingly.
    #frame = cv2.Canny(frame, 50,150)
    # For instance, in this example you can enable the Canny edge function above.
    # You will see that the frame rate drops to ~35fps and video playback is erratic.
    # If you then set fps = 30 at the beginning of the script, there will be enough cycle time between frames to provide accurate video.
    
    # One optimization could be to work with a decimated (downscaled) version of the image: deci = frame[::2, ::2]
    
    frames.append(frame) # save the frame (for the demo)
    #del frame # free the allocated memory
    N_frames += 1
    if N_frames > max_frames: break

end_time = time.time()

cameraProcess.terminate() # stop the camera


elapsed_seconds = end_time-start_time
print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")


print("Writing frames to disk...")
out = cv2.VideoWriter("slow_motion.avi", cv2.cv.CV_FOURCC(*"MJPG"), 30, (w,h))
for n in range(N_frames):
    #cv2.imwrite("frame"+str(n)+".png", frames[n]) # save frame as a PNG image
    frame_rgb = cv2.cvtColor(frames[n],cv2.COLOR_GRAY2RGB) # video codec requires RGB image
    out.write(frame_rgb)
out.release()

print("Display frames with OpenCV...")
for frame in frames:
    cv2.imshow("Slow Motion", frame)
    cv2.waitKey(1) # request maximum refresh rate

cv2.destroyAllWindows()

Introduction

For some reason every Raspberry Camera tutorial specifies the maximum camera frame rate at 90 FPS. This is not true!
With the proof-of-concept script above, the camera has been tested to work up to at least 120FPS. Now it is possible to have real time, ultra-low-latency video processing even in the Raspberry Zero. The form factor of this board makes it ideal for robotics.

These high FPS are possible thanks to the great work at Raspividyuv. The above script is the first one to connect Raspividyuv and Python to achieve ultra low latency results. I hope this example code can enable many people to integrate efficient computer vision algorithms into many kinds of robots. The scripts receives grayscale video only, though it could be extended to fetch color as well.

Maximum framerates at multiple resolutions

Download full spreadsheet here: https://www.dropbox.com/s/28vq8d9qx0tm81i/raspi_cameraV2_fpsResolution.ods

Example images at each resolution: https://www.dropbox.com/s/k0gzpt15jj0qbqd/raspberry_cameraV2_quality_FPS_comparison.zip (useful to compare the field-of-view in every mode). Note some of the large modes produce corrupt frames, this needs to be studied.

Videos (click to open)

Slow motion video example (~120fps played at 30fps, captured with the python script:

Real time Canny edge detection (30fps):

The videos are not only recorded, but also processed in real time in the Python script.

Now it is possible to have low cost vision for fast robots! Even if you don't actually use 120FPS, the lower time between frames will give you more cycle time to process each image.

Please share your progress too, so we can all learn :-)

Thank you! Needed "VideoWriter_fourcc" vs "cv.CV_FOURCC" to get this working for OpenCV3.3.0 on Pi3
Ex. out = cv2.VideoWriter("slow_motion.avi", cv2.VideoWriter_fourcc(*"MJPG"), 30, (w,h))

Sweet! Thanks for sharing how to run it on OpenCV 3.3, will come in handy for the future :)

Thank you! it's amazing. But how could I get colorful video?

Currently I have no plans to implement color version. To do this, it is necessary to remove the --luma option from raspividyuv and extract & combine the new chroma channels.

Hi,

Is there any c++ code for same speed ? or can we get more with c++ ?

Best

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

This error appears because in Python 3 pipe is buffered but unseekable (details).
So as a patch you need to add "bufsize=0" option, that is:

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0) # start the camera

Also in my case image has been distorted, and I changed resolution to 640x480 instead of 640x240.

Thank you for great piece of code! It really helped me out with my project.

I have one question. I need to get images from both cameras. I added '--stereo tb' / '--stereo sbs', changed resolution also, but I still get frames only from one camera.

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

try with sudo

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

This error appears because in Python 3 pipe is buffered but unseekable (details).
So as a patch you need to add "bufsize=0" option, that is:

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPEm, bufsize=0) # start the camera

Also in my case image has been distorted, and I changed resolution to 640x480 instead of 640x240.

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0) # start the camera` - works for me. PIPE without m

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0) # start the camera` - works for me. PIPE without m

Oops, sorry for this typo. I've fixed my comment!

Hi,

Thanks for the great piece of software.
I have camera V1.0, and I am not sure if this program should work with it or not. Maybe at limited speed and other conservative settings?

I got error message as follows:
Recording...
Traceback (most recent call last):
File "fast_fps.py", line 46, in
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: [Errno 29] Illegal seek

The OS is the latest Raspberry Pi OS, with OpenCV 4.5.1 compiled from source on Raspi 3B+, Python 3.7
the bufsize=0 is added: cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0)

Is this related to camera version or there is other reason ?
Thanks for your help in advance

Hi, can you check if raspividyuv is installed?

Yes, it is installed:
(cv) pi@raspberrypi:~ $ raspividyuv -v

_"raspividyuv" Camera App (commit 4a0a19b88b43 Tainted)

Camera Name ov5647
Width 1920, Height 1080, filename (null)
Using camera 0, sensor mode 0

GPS output Disabled

framerate 30, time delay 5000
Sub-image size 3133440 bytes in total.
Y pitch 1920, Y height 1088, UV pitch 960, UV Height 544
Wait method : Simple capture
Initial state 'record'

Preview Yes, Full screen Yes
Preview window 0,0,1024,768_

Then I'm not sure, can't offer you a solution :S

It is somehow related to "bufsize" topic, because when I set the bufsize=1 I got the message "obtaining file position failed".
I assume it is Python version 3.x issue, but could not find any working solution.

It does work with Python 2.7, however I needed to change from "cv2.cv.CV_FOURCC" to "cv2.VideoWriter_fourcc" in line 77 because of newer version of OpenCV

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

To fix this issue in Python3, you just need to replace:

frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)

with

frame = np.frombuffer(cameraProcess.stdout.read(bytesPerFrame), dtype=np.uint8)

Hi,
Thanks for the update, I had to modify few other things, this is the relevant part which works now in Python3:

print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")
fourcc = cv2.VideoWriter_fourcc(*'MJPG')

print("Writing frames to disk...")
out = cv2.VideoWriter("slow_motion.avi", fourcc, 30, (w,h))_**

Thanks a lot for sharing!! :)

Heyy, I had the same issues, I solved them but now I am not getting frame.size equal to bytesPerFrame

Recording... BytesPerFrame: 307200 Frame.size: 65536 Camera Process Stopped!~~ Writing frames to disk...

Please help me to solve this issue!!!

here`s the code with all fixes done:

# Fast reading from the raspberry camera with Python, Numpy, and OpenCV
# Allows to process grayscale video up to 124 FPS (tested in Raspberry Zero Wifi with V2.1 camera)
#
# Made by @CarlosGS in May 2017
# Club de Robotica - Universidad Autonoma de Madrid
# http://crm.ii.uam.es/
# License: Public Domain, attribution appreciated

import cv2
import numpy as np
import subprocess as sp
import time
import atexit

frames = [] # stores the video sequence for the demo
max_frames = 300

N_frames = 0

# Video capture parameters
(w,h) = (640,240)
bytesPerFrame = w * h
fps = 250 # setting to 250 will request the maximum framerate possible

# "raspividyuv" is the command that provides camera frames in YUV format
#  "--output -" specifies stdout as the output
#  "--timeout 0" specifies continuous video
#  "--luma" discards chroma channels, only luminance is sent through the pipeline
# see "raspividyuv --help" for more information on the parameters
videoCmd = "raspividyuv -w "+str(w)+" -h "+str(h)+" --output - --timeout 0 --framerate "+str(fps)+" --luma --nopreview"
videoCmd = videoCmd.split() # Popen requires that each parameter is a separate string

#cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE) # start the camera
cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=1)
atexit.register(cameraProcess.terminate) # this closes the camera process in case the python scripts exits unexpectedly

# wait for the first frame and discard it (only done to measure time more accurately)
rawStream = cameraProcess.stdout.read(bytesPerFrame)

print("Recording...")

start_time = time.time()

while True:
    cameraProcess.stdout.flush() # discard any frames that we were not able to process in time
    # Parse the raw stream into a numpy array
    #frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
    frame = np.frombuffer(cameraProcess.stdout.read(bytesPerFrame), dtype=np.uint8)
    if frame.size != bytesPerFrame:
        print("Error: Camera stream closed unexpectedly")
        break
    frame.shape = (h,w) # set the correct dimensions for the numpy array

    # The frame can be processed here using any function in the OpenCV library.

    # Full image processing will slow down the pipeline, so the requested FPS should be set accordingly.
    #frame = cv2.Canny(frame, 50,150)
    # For instance, in this example you can enable the Canny edge function above.
    # You will see that the frame rate drops to ~35fps and video playback is erratic.
    # If you then set fps = 30 at the beginning of the script, there will be enough cycle time between frames to provide accurate video.
    
    # One optimization could be to work with a decimated (downscaled) version of the image: deci = frame[::2, ::2]
    
    frames.append(frame) # save the frame (for the demo)
    #del frame # free the allocated memory
    N_frames += 1
    if N_frames > max_frames: break

end_time = time.time()
cameraProcess.terminate() # stop the camera


elapsed_seconds = end_time-start_time
print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")


print("Writing frames to disk...")
#out = cv2.VideoWriter("slow_motion.avi", cv2.cv.CV_FOURCC(*"MJPG"), 30, (w,h))
fourcc = cv2.VideoWriter_fourcc(*"MJPG")
out = cv2.VideoWriter("slow_motion.avi", fourcc, 30, (w,h))
for n in range(N_frames):
    #cv2.imwrite("frame"+str(n)+".png", frames[n]) # save frame as a PNG image
    frame_rgb = cv2.cvtColor(frames[n],cv2.COLOR_GRAY2RGB) # video codec requires RGB image
    out.write(frame_rgb)
out.release()

print("Display frames with OpenCV...")
for frame in frames:
    cv2.imshow("Slow Motion", frame)
    cv2.waitKey(1) # request maximum refresh rate

cv2.destroyAllWindows()

@zoldaten , you are my hero! Thank you very much for your fixes!

I made a C/C++ version of this script. My NoIR Pi Zero camera is limited to 90fps, so I could not gain any benefit from using raspividyuv. But maybe this is helpful for other users:

// License: Public Domain, attribution appreciated
#include <stdio.h>
#include <stdint.h>

#include <sstream>
#include <string>
#include <vector>

#include <opencv2/opencv.hpp>

int main(int argc, char** argv)
{
    int max_frames = 300;
    std::vector<cv::Mat> frames(max_frames); // stores the video sequence for the demo
    
    // Video capture parameters
    int width = 640;
    int height = 240;
    int bytesPerFrame = width*height;
    int fps = 250; // setting to 250 will request the maximum framerate possible
    
    // "raspividyuv" is the command that provides camera frames in YUV format
    //  "--output -" specifies stdout as the output
    //  "--timeout 0" specifies continuous video
    //  "--luma" discards chroma channels, only luminance is sent through the pipeline
    // see "raspividyuv --help" for more information on the parameters
    std::stringstream ss;
    ss << "/bin/raspividyuv -w " << std::to_string(width) << " -h " << std::to_string(height) << " --output - --timeout 0 --framerate " << std::to_string(fps) << " --luma --nopreview";
    std::string videoCmd = ss.str();
    
    // start the camera
    FILE *cameraProcess;

    if ((cameraProcess = popen(videoCmd.c_str(), "r")) == NULL) {
        printf("Error starting raspividyuv\n");
        return -1;
    }
    
    // create buffer for camera data
    char* buffer = new char[bytesPerFrame];
    cv::Mat frame(height, width, CV_8UC1, (unsigned char*)buffer);
    
    // wait for the first frame and discard it (only done to measure time more accurately)
    fread(buffer, bytesPerFrame, 1, cameraProcess);
    
    printf("Recording...\n");
    
    long long start_time = cv::getTickCount();
    
    for (int frameNo = 0; frameNo < max_frames; frameNo++)
    {
        // Parse the raw stream into our buffer
        fread(buffer, bytesPerFrame, 1, cameraProcess);
        
        // The frame can be processed here using any function in the OpenCV library.
        // ...
        
        frame.copyTo(frames[frameNo]); // save the frame (for the demo)
    }
    
    long long end_time = cv::getTickCount();
    
    pclose(cameraProcess);
    delete[] buffer;
    
    printf("Done! Result: %f fps\n", (cv::getTickFrequency() / (end_time - start_time))*max_frames);
    
    
    printf("Writing frames to disk...\n");
    
    cv::VideoWriter out("/home/mypi/slow_motion.avi", CV_FOURCC('M', 'J', 'P', 'G'), 30, cv::Size(width, height));
    
    cv::Mat rgbFrame;
    for (int frameNo = 0; frameNo < max_frames; frameNo++) {
        cvtColor(frames[frameNo], rgbFrame, cv::COLOR_GRAY2BGR); // video codec requires BGR image
        out.write(rgbFrame);
    }
    
    
    printf("Display frames with OpenCV...\n");
    
    for (int frameNo = 0; frameNo < max_frames; frameNo++) {
        cv::imshow("Slow Motion", frames[frameNo]);
        cv::waitKey(1); // request maximum refresh rate
    }
    
    return 0;
}

For resolution of 640x480 etc, the frames seem to get cropped to a smaller area at FPS above 40. This seems to be an issue after an update since it was working ok before the update.

1280x720 : Image ok & not cropped - fps at capped at ~50
960x480: Image ok & not cropped - fps at capped at ~50
640x480: Image cropped - fps at 120
640x480: Image NOT cropped - fps at 30
320x240: Image cropped - fps at 120
320x240: Image NOT cropped - fps at 30

Any idea how to get the full image instead of the cropped image at 640x480 back? Thanks.