Sending a cv::Mat from C++ to a Python script

main.cpp

/*
This requires adding the "include" directory of your Python installation to the include diretories 
of your project, e.g., in Visual Studio you'd add `C:\Program Files\Python36\include`. 
You also need to add the 'include' directory of your NumPy package, e.g. 
`C:\Program Files\PythonXX\Lib\site-packages\numpy\core\include`.

Additionally, you need to link your "python3#.lib" library, e.g. `C:\Program Files\Python3X\libs\python3X.lib`. 
*/

// python bindings
#include "Python.h"
#include "numpy/arrayobject.h"

#include <iostream>

#include "opencv2/opencv.hpp"

// references to all the functions
PyObject *m_PyDict, *m_PyFooBar;

// reference to the Pyhton module
PyObject* m_PyModule;

int main(int argc, char** argv) {
  // initialize Python embedding
  Py_Initialize(); 
  
  // set the command line arguments (can be crucial for some python-packages, like tensorflow)
  PySys_SetArgv(argc, (wchar_t**) argv);
  
  // add the current folder to the Python's PATH
  PyObject *sys_path = PySys_GetObject("path");
  PyList_Append(sys_path, PyUnicode_FromString("."));  
  
  // this macro is defined by NumPy and must be included
  import_array1(-1);
  
  // load our python script (see the gist at the bottom)
  // use the script's filename, sans the extension 
  m_PyModule = PyImport_ImportModule("pythonScript"); 
  
  if (m_PyModule != NULL)
  {
    // get dictionary of all available entities in the module
    m_PyDict = PyModule_GetDict(m_PyModule);

    // grab the functions we are interested in (using its name)
    m_PyFooBar = PyDict_GetItemString(m_PyDict, "foo_bar");
    
    // execute the function    
    if (m_PyFooBar != NULL)
    {
      // take a cv::Mat object from somewhere (we'll just create an empty one)
      cv::Mat img = cv::Mat::zeros(480, 640, CV_8U);
      
      // total number of elements (here it's a greyscale 640x480)
      const unsigned int nElem = 640 * 480;
      
      // create an array of apropriate datatype
      uchar* m = new uchar[nElem];
      
      // copy the data from the cv::Mat object into the array
      std::memcpy(m, img.data, nElem * sizeof(uchar));
      
      // the dimensions of the matrix
      npy_intp mdim[] = { 480, 640 };
      
      // convert the cv::Mat to numpy.array
      PyObject* mat = PyArray_SimpleNewFromData(2, mdim, NPY_UINT8, (void*) m);
      
      // create a Python-tuple of arguments for the function call
      // "()" means "tuple". "O" means "object"
      PyObject* args = Py_BuildValue("(O)", mat);
      // if we want several arguments, we can write ("i" means "integer"):
      // PyObject* args = Py_BuildValue("(OOi)", mat, mat, 123);
      // to send two images and an integer, equivalent to Python's (mat, mat, 123) tuple
      // see detailed explanation here: https://docs.python.org/2.0/ext/buildValue.html 
      
      // execute the function
      PyObject* result = PyEval_CallObject(m_PyFooBar, args);
      
      // process the result
      // ...
      
      // decrement the object references
      Py_XDECREF(mat);
      Py_XDECREF(result);
      Py_XDECREF(args);

      delete[] m;
    }
  }
  else
  {
    std::cerr << "Failed to load the Python module!" << std::endl;
    PyErr_Print();
  }
  
  return 0;
}

main2.cpp

// same example, but for a 3-channel RGB image.

// python bindings
#include "Python.h"
#include "numpy/arrayobject.h"

#include <iostream>

#include "opencv2/opencv.hpp"

// for the references to all the functions
PyObject *m_PyDict, *m_PyFooBar;

// for the reference to the Pyhton module
PyObject* m_PyModule;

int main(int argc, char** argv) {
  // initialize everything (see gist above for details)
  Py_Initialize(); 
  PySys_SetArgv(argc, (wchar_t**)argv);  
  PyObject *sys_path = PySys_GetObject("path");
  PyList_Append(sys_path, PyUnicode_FromString("."));    
  import_array1(-1);  
  m_PyModule = PyImport_ImportModule("pythonScript"); 
  
  if (m_PyModule != NULL)
  {
    // get dictionary of available items in the module
    m_PyDict = PyModule_GetDict(m_PyModule);

    // grab the functions we are interested in
    m_PyFooBar = PyDict_GetItemString(m_PyDict, "foo_bar");
    
    // execute the function    
    if (m_PyFooBar != NULL)
    {
      // take a cv::Mat object from somewhere (we'll just create one)
      cv::Mat img = cv::Mat::zeros(480, 640, CV_8UC3);
      
      // total number of elements (here it's an RGB image of size 640x480)
      const unsigned int nElem = 640 * 480 * 3;
      
      // create an array of apropriate datatype
      uchar* m = new uchar[nElem];
      
      // copy the data from the cv::Mat object into the array
      std::memcpy(m, img.data, nElem * sizeof(uchar));
      
      // the dimensions of the matrix
      npy_intp mdim[] = { 480, 640, 3 };
      
      // convert the cv::Mat to numpy.array
      PyObject* mat = PyArray_SimpleNewFromData(3, mdim, NPY_UINT8, (void*) m);
      
      // create a Python-tuple of arguments for the function call
      // "()" means "tuple". "O" means "object"
      PyObject* args = Py_BuildValue("(O)", mat);
      
      // execute the function
      PyObject* result = PyEval_CallObject(m_PyFooBar, args);
      
      // process the result
      // ...
      
      // decrement the object references
      Py_XDECREF(mat);
      Py_XDECREF(result);
      Py_XDECREF(args);

      delete[] m;
    }
  }
  else
  {
    std::cerr << "Failed to load the Python module!" << std::endl;
    PyErr_Print();
  }
  
  return 0;
}

pythonScript.py

import cv2
import numpy as np

# define the function that we will call from the C++ code
def foo_bar(img=None):  
    if img is not None:
       cv2.imshow("image in python", img)

Thanks Guys Xonxt for this script. It's very helpfull. I run this code on Visual Studio 2017 version 15.9.40 and i don't meet any problem.

But I meet the problem: ModuleNotFoundError: No module named 'numpy'. This problem occur when i using Visual Studio 2019 version 16.11.5. The line code import_array1(-1) is the reason for that.
Can you help me? I don't know why MSVC 2019 imcompatible with numpy API of Python.
thanks for your reading !

Thanks Guys Xonxt for this script. It's very helpfull. I run this code on Visual Studio 2017 version 15.9.40 and i don't meet any problem.

But I meet the problem: ModuleNotFoundError: No module named 'numpy'. This problem occur when i using Visual Studio 2019 version 16.11.5. The line code import_array1(-1) is the reason for that. Can you help me? I don't know why MSVC 2019 imcompatible with numpy API of Python. thanks for your reading !

Did you install the 'numpy' module in your Python and import the C:\Program Files\PythonXX\Lib\site-packages\numpy\core\include into your MSVC project?
If that doesn't work, then I don't know, sorry. I haven't worked with this stuff for a while. Ever since I posted this to be exact :D.

Yes. I already installed numpy package in python. That why i can run your code in VS 2017. Now i want run your code in VS 2019 but i have the problem.
Thanks you response for me.

Yes. I already installed numpy package in python. That why i can run your code in VS 2017. Now i want run your code in VS 2019 but i have the problem. Thanks you response for me.

I have more information, now i return VS 2017 and this code have the same problem :)) . I install two Visual Studio together 2017 and 2019. First 2017 and second 2019.

https://gist.github.com/Xonxt/26d2a9ac6c56505d0896822ede99a646#gistcomment-3723333

I tested it again and I think it didn't have any overhead anymore. thanks for the reply.
@Xonxt

Hi guys, long time no reply :)
I have a new question for you. How i can choose enviromnent for python? It's mean when i call python from c++ script? How to choose env to run python script.
Example: I using ubuntu 18.04 OS . I have 2 virtual environment ( anaconda and venv) and 1 real environment python3.6.9 default of ubuntu (python default). How to i know what env i run python code when call from c++ and how to change it? is it based on python.h (python.a in linux). I think this is similar on window with venv and anaconda and python when install from offcial web site of python.