Fast reading from the raspberry camera with Python, Numpy, and OpenCV. See the comments for more details.

raspberry_fast_capture.py

# Fast reading from the raspberry camera with Python, Numpy, and OpenCV
# Allows to process grayscale video up to 124 FPS (tested in Raspberry Zero Wifi with V2.1 camera)
#
# Made by @CarlosGS in May 2017
# Club de Robotica - Universidad Autonoma de Madrid
# http://crm.ii.uam.es/
# License: Public Domain, attribution appreciated

import cv2
import numpy as np
import subprocess as sp
import time
import atexit

frames = [] # stores the video sequence for the demo
max_frames = 300

N_frames = 0

# Video capture parameters
(w,h) = (640,240)
bytesPerFrame = w * h
fps = 250 # setting to 250 will request the maximum framerate possible

# "raspividyuv" is the command that provides camera frames in YUV format
#  "--output -" specifies stdout as the output
#  "--timeout 0" specifies continuous video
#  "--luma" discards chroma channels, only luminance is sent through the pipeline
# see "raspividyuv --help" for more information on the parameters
videoCmd = "raspividyuv -w "+str(w)+" -h "+str(h)+" --output - --timeout 0 --framerate "+str(fps)+" --luma --nopreview"
videoCmd = videoCmd.split() # Popen requires that each parameter is a separate string

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE) # start the camera
atexit.register(cameraProcess.terminate) # this closes the camera process in case the python scripts exits unexpectedly

# wait for the first frame and discard it (only done to measure time more accurately)
rawStream = cameraProcess.stdout.read(bytesPerFrame)

print("Recording...")

start_time = time.time()

while True:
    cameraProcess.stdout.flush() # discard any frames that we were not able to process in time
    # Parse the raw stream into a numpy array
    frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
    if frame.size != bytesPerFrame:
        print("Error: Camera stream closed unexpectedly")
        break
    frame.shape = (h,w) # set the correct dimensions for the numpy array

    # The frame can be processed here using any function in the OpenCV library.

    # Full image processing will slow down the pipeline, so the requested FPS should be set accordingly.
    #frame = cv2.Canny(frame, 50,150)
    # For instance, in this example you can enable the Canny edge function above.
    # You will see that the frame rate drops to ~35fps and video playback is erratic.
    # If you then set fps = 30 at the beginning of the script, there will be enough cycle time between frames to provide accurate video.
    
    # One optimization could be to work with a decimated (downscaled) version of the image: deci = frame[::2, ::2]
    
    frames.append(frame) # save the frame (for the demo)
    #del frame # free the allocated memory
    N_frames += 1
    if N_frames > max_frames: break

end_time = time.time()

cameraProcess.terminate() # stop the camera


elapsed_seconds = end_time-start_time
print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")


print("Writing frames to disk...")
out = cv2.VideoWriter("slow_motion.avi", cv2.cv.CV_FOURCC(*"MJPG"), 30, (w,h))
for n in range(N_frames):
    #cv2.imwrite("frame"+str(n)+".png", frames[n]) # save frame as a PNG image
    frame_rgb = cv2.cvtColor(frames[n],cv2.COLOR_GRAY2RGB) # video codec requires RGB image
    out.write(frame_rgb)
out.release()

print("Display frames with OpenCV...")
for frame in frames:
    cv2.imshow("Slow Motion", frame)
    cv2.waitKey(1) # request maximum refresh rate

cv2.destroyAllWindows()

Thank you! Needed "VideoWriter_fourcc" vs "cv.CV_FOURCC" to get this working for OpenCV3.3.0 on Pi3
Ex. out = cv2.VideoWriter("slow_motion.avi", cv2.VideoWriter_fourcc(*"MJPG"), 30, (w,h))

Sweet! Thanks for sharing how to run it on OpenCV 3.3, will come in handy for the future :)

Thank you! it's amazing. But how could I get colorful video?

Currently I have no plans to implement color version. To do this, it is necessary to remove the --luma option from raspividyuv and extract & combine the new chroma channels.

Hi,

Is there any c++ code for same speed ? or can we get more with c++ ?

Best

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

This error appears because in Python 3 pipe is buffered but unseekable (details).
So as a patch you need to add "bufsize=0" option, that is:

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0) # start the camera

Also in my case image has been distorted, and I changed resolution to 640x480 instead of 640x240.

Thank you for great piece of code! It really helped me out with my project.

I have one question. I need to get images from both cameras. I added '--stereo tb' / '--stereo sbs', changed resolution also, but I still get frames only from one camera.

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

try with sudo

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

This error appears because in Python 3 pipe is buffered but unseekable (details).
So as a patch you need to add "bufsize=0" option, that is:

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPEm, bufsize=0) # start the camera

Also in my case image has been distorted, and I changed resolution to 640x480 instead of 640x240.

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0) # start the camera` - works for me. PIPE without m

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0) # start the camera` - works for me. PIPE without m

Oops, sorry for this typo. I've fixed my comment!

Hi,

Thanks for the great piece of software.
I have camera V1.0, and I am not sure if this program should work with it or not. Maybe at limited speed and other conservative settings?

I got error message as follows:
Recording...
Traceback (most recent call last):
File "fast_fps.py", line 46, in
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: [Errno 29] Illegal seek

The OS is the latest Raspberry Pi OS, with OpenCV 4.5.1 compiled from source on Raspi 3B+, Python 3.7
the bufsize=0 is added: cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=0)

Is this related to camera version or there is other reason ?
Thanks for your help in advance

Hi, can you check if raspividyuv is installed?

Yes, it is installed:
(cv) pi@raspberrypi:~ $ raspividyuv -v

_"raspividyuv" Camera App (commit 4a0a19b88b43 Tainted)

Camera Name ov5647
Width 1920, Height 1080, filename (null)
Using camera 0, sensor mode 0

GPS output Disabled

framerate 30, time delay 5000
Sub-image size 3133440 bytes in total.
Y pitch 1920, Y height 1088, UV pitch 960, UV Height 544
Wait method : Simple capture
Initial state 'record'

Preview Yes, Full screen Yes
Preview window 0,0,1024,768_

Then I'm not sure, can't offer you a solution :S

It is somehow related to "bufsize" topic, because when I set the bufsize=1 I got the message "obtaining file position failed".
I assume it is Python version 3.x issue, but could not find any working solution.

It does work with Python 2.7, however I needed to change from "cv2.cv.CV_FOURCC" to "cv2.VideoWriter_fourcc" in line 77 because of newer version of OpenCV

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

To fix this issue in Python3, you just need to replace:

frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)

with

frame = np.frombuffer(cameraProcess.stdout.read(bytesPerFrame), dtype=np.uint8)

Hi,
Thanks for the update, I had to modify few other things, this is the relevant part which works now in Python3:

print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")
fourcc = cv2.VideoWriter_fourcc(*'MJPG')

print("Writing frames to disk...")
out = cv2.VideoWriter("slow_motion.avi", fourcc, 30, (w,h))_**

Thanks a lot for sharing!! :)

Heyy, I had the same issues, I solved them but now I am not getting frame.size equal to bytesPerFrame

Recording... BytesPerFrame: 307200 Frame.size: 65536 Camera Process Stopped!~~ Writing frames to disk...

Please help me to solve this issue!!!

here`s the code with all fixes done:

# Fast reading from the raspberry camera with Python, Numpy, and OpenCV
# Allows to process grayscale video up to 124 FPS (tested in Raspberry Zero Wifi with V2.1 camera)
#
# Made by @CarlosGS in May 2017
# Club de Robotica - Universidad Autonoma de Madrid
# http://crm.ii.uam.es/
# License: Public Domain, attribution appreciated

import cv2
import numpy as np
import subprocess as sp
import time
import atexit

frames = [] # stores the video sequence for the demo
max_frames = 300

N_frames = 0

# Video capture parameters
(w,h) = (640,240)
bytesPerFrame = w * h
fps = 250 # setting to 250 will request the maximum framerate possible

# "raspividyuv" is the command that provides camera frames in YUV format
#  "--output -" specifies stdout as the output
#  "--timeout 0" specifies continuous video
#  "--luma" discards chroma channels, only luminance is sent through the pipeline
# see "raspividyuv --help" for more information on the parameters
videoCmd = "raspividyuv -w "+str(w)+" -h "+str(h)+" --output - --timeout 0 --framerate "+str(fps)+" --luma --nopreview"
videoCmd = videoCmd.split() # Popen requires that each parameter is a separate string

#cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE) # start the camera
cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE, bufsize=1)
atexit.register(cameraProcess.terminate) # this closes the camera process in case the python scripts exits unexpectedly

# wait for the first frame and discard it (only done to measure time more accurately)
rawStream = cameraProcess.stdout.read(bytesPerFrame)

print("Recording...")

start_time = time.time()

while True:
    cameraProcess.stdout.flush() # discard any frames that we were not able to process in time
    # Parse the raw stream into a numpy array
    #frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
    frame = np.frombuffer(cameraProcess.stdout.read(bytesPerFrame), dtype=np.uint8)
    if frame.size != bytesPerFrame:
        print("Error: Camera stream closed unexpectedly")
        break
    frame.shape = (h,w) # set the correct dimensions for the numpy array

    # The frame can be processed here using any function in the OpenCV library.

    # Full image processing will slow down the pipeline, so the requested FPS should be set accordingly.
    #frame = cv2.Canny(frame, 50,150)
    # For instance, in this example you can enable the Canny edge function above.
    # You will see that the frame rate drops to ~35fps and video playback is erratic.
    # If you then set fps = 30 at the beginning of the script, there will be enough cycle time between frames to provide accurate video.
    
    # One optimization could be to work with a decimated (downscaled) version of the image: deci = frame[::2, ::2]
    
    frames.append(frame) # save the frame (for the demo)
    #del frame # free the allocated memory
    N_frames += 1
    if N_frames > max_frames: break

end_time = time.time()
cameraProcess.terminate() # stop the camera


elapsed_seconds = end_time-start_time
print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")


print("Writing frames to disk...")
#out = cv2.VideoWriter("slow_motion.avi", cv2.cv.CV_FOURCC(*"MJPG"), 30, (w,h))
fourcc = cv2.VideoWriter_fourcc(*"MJPG")
out = cv2.VideoWriter("slow_motion.avi", fourcc, 30, (w,h))
for n in range(N_frames):
    #cv2.imwrite("frame"+str(n)+".png", frames[n]) # save frame as a PNG image
    frame_rgb = cv2.cvtColor(frames[n],cv2.COLOR_GRAY2RGB) # video codec requires RGB image
    out.write(frame_rgb)
out.release()

print("Display frames with OpenCV...")
for frame in frames:
    cv2.imshow("Slow Motion", frame)
    cv2.waitKey(1) # request maximum refresh rate

cv2.destroyAllWindows()

@zoldaten , you are my hero! Thank you very much for your fixes!

I made a C/C++ version of this script. My NoIR Pi Zero camera is limited to 90fps, so I could not gain any benefit from using raspividyuv. But maybe this is helpful for other users:

// License: Public Domain, attribution appreciated
#include <stdio.h>
#include <stdint.h>

#include <sstream>
#include <string>
#include <vector>

#include <opencv2/opencv.hpp>

int main(int argc, char** argv)
{
    int max_frames = 300;
    std::vector<cv::Mat> frames(max_frames); // stores the video sequence for the demo
    
    // Video capture parameters
    int width = 640;
    int height = 240;
    int bytesPerFrame = width*height;
    int fps = 250; // setting to 250 will request the maximum framerate possible
    
    // "raspividyuv" is the command that provides camera frames in YUV format
    //  "--output -" specifies stdout as the output
    //  "--timeout 0" specifies continuous video
    //  "--luma" discards chroma channels, only luminance is sent through the pipeline
    // see "raspividyuv --help" for more information on the parameters
    std::stringstream ss;
    ss << "/bin/raspividyuv -w " << std::to_string(width) << " -h " << std::to_string(height) << " --output - --timeout 0 --framerate " << std::to_string(fps) << " --luma --nopreview";
    std::string videoCmd = ss.str();
    
    // start the camera
    FILE *cameraProcess;

    if ((cameraProcess = popen(videoCmd.c_str(), "r")) == NULL) {
        printf("Error starting raspividyuv\n");
        return -1;
    }
    
    // create buffer for camera data
    char* buffer = new char[bytesPerFrame];
    cv::Mat frame(height, width, CV_8UC1, (unsigned char*)buffer);
    
    // wait for the first frame and discard it (only done to measure time more accurately)
    fread(buffer, bytesPerFrame, 1, cameraProcess);
    
    printf("Recording...\n");
    
    long long start_time = cv::getTickCount();
    
    for (int frameNo = 0; frameNo < max_frames; frameNo++)
    {
        // Parse the raw stream into our buffer
        fread(buffer, bytesPerFrame, 1, cameraProcess);
        
        // The frame can be processed here using any function in the OpenCV library.
        // ...
        
        frame.copyTo(frames[frameNo]); // save the frame (for the demo)
    }
    
    long long end_time = cv::getTickCount();
    
    pclose(cameraProcess);
    delete[] buffer;
    
    printf("Done! Result: %f fps\n", (cv::getTickFrequency() / (end_time - start_time))*max_frames);
    
    
    printf("Writing frames to disk...\n");
    
    cv::VideoWriter out("/home/mypi/slow_motion.avi", CV_FOURCC('M', 'J', 'P', 'G'), 30, cv::Size(width, height));
    
    cv::Mat rgbFrame;
    for (int frameNo = 0; frameNo < max_frames; frameNo++) {
        cvtColor(frames[frameNo], rgbFrame, cv::COLOR_GRAY2BGR); // video codec requires BGR image
        out.write(rgbFrame);
    }
    
    
    printf("Display frames with OpenCV...\n");
    
    for (int frameNo = 0; frameNo < max_frames; frameNo++) {
        cv::imshow("Slow Motion", frames[frameNo]);
        cv::waitKey(1); // request maximum refresh rate
    }
    
    return 0;
}

For resolution of 640x480 etc, the frames seem to get cropped to a smaller area at FPS above 40. This seems to be an issue after an update since it was working ok before the update.

1280x720 : Image ok & not cropped - fps at capped at ~50
960x480: Image ok & not cropped - fps at capped at ~50
640x480: Image cropped - fps at 120
640x480: Image NOT cropped - fps at 30
320x240: Image cropped - fps at 120
320x240: Image NOT cropped - fps at 30

Any idea how to get the full image instead of the cropped image at 640x480 back? Thanks.