Sending a cv::Mat from C++ to a Python script

main.cpp

/*
This requires adding the "include" directory of your Python installation to the include diretories 
of your project, e.g., in Visual Studio you'd add `C:\Program Files\Python36\include`. 
You also need to add the 'include' directory of your NumPy package, e.g. 
`C:\Program Files\PythonXX\Lib\site-packages\numpy\core\include`.

Additionally, you need to link your "python3#.lib" library, e.g. `C:\Program Files\Python3X\libs\python3X.lib`. 
*/

// python bindings
#include "Python.h"
#include "numpy/arrayobject.h"

#include <iostream>

#include "opencv2/opencv.hpp"

// references to all the functions
PyObject *m_PyDict, *m_PyFooBar;

// reference to the Pyhton module
PyObject* m_PyModule;

int main(int argc, char** argv) {
  // initialize Python embedding
  Py_Initialize(); 
  
  // set the command line arguments (can be crucial for some python-packages, like tensorflow)
  PySys_SetArgv(argc, (wchar_t**) argv);
  
  // add the current folder to the Python's PATH
  PyObject *sys_path = PySys_GetObject("path");
  PyList_Append(sys_path, PyUnicode_FromString("."));  
  
  // this macro is defined by NumPy and must be included
  import_array1(-1);
  
  // load our python script (see the gist at the bottom)
  // use the script's filename, sans the extension 
  m_PyModule = PyImport_ImportModule("pythonScript"); 
  
  if (m_PyModule != NULL)
  {
    // get dictionary of all available entities in the module
    m_PyDict = PyModule_GetDict(m_PyModule);

    // grab the functions we are interested in (using its name)
    m_PyFooBar = PyDict_GetItemString(m_PyDict, "foo_bar");
    
    // execute the function    
    if (m_PyFooBar != NULL)
    {
      // take a cv::Mat object from somewhere (we'll just create an empty one)
      cv::Mat img = cv::Mat::zeros(480, 640, CV_8U);
      
      // total number of elements (here it's a greyscale 640x480)
      const unsigned int nElem = 640 * 480;
      
      // create an array of apropriate datatype
      uchar* m = new uchar[nElem];
      
      // copy the data from the cv::Mat object into the array
      std::memcpy(m, img.data, nElem * sizeof(uchar));
      
      // the dimensions of the matrix
      npy_intp mdim[] = { 480, 640 };
      
      // convert the cv::Mat to numpy.array
      PyObject* mat = PyArray_SimpleNewFromData(2, mdim, NPY_UINT8, (void*) m);
      
      // create a Python-tuple of arguments for the function call
      // "()" means "tuple". "O" means "object"
      PyObject* args = Py_BuildValue("(O)", mat);
      // if we want several arguments, we can write ("i" means "integer"):
      // PyObject* args = Py_BuildValue("(OOi)", mat, mat, 123);
      // to send two images and an integer, equivalent to Python's (mat, mat, 123) tuple
      // see detailed explanation here: https://docs.python.org/2.0/ext/buildValue.html 
      
      // execute the function
      PyObject* result = PyEval_CallObject(m_PyFooBar, args);
      
      // process the result
      // ...
      
      // decrement the object references
      Py_XDECREF(mat);
      Py_XDECREF(result);
      Py_XDECREF(args);

      delete[] m;
    }
  }
  else
  {
    std::cerr << "Failed to load the Python module!" << std::endl;
    PyErr_Print();
  }
  
  return 0;
}

main2.cpp

// same example, but for a 3-channel RGB image.

// python bindings
#include "Python.h"
#include "numpy/arrayobject.h"

#include <iostream>

#include "opencv2/opencv.hpp"

// for the references to all the functions
PyObject *m_PyDict, *m_PyFooBar;

// for the reference to the Pyhton module
PyObject* m_PyModule;

int main(int argc, char** argv) {
  // initialize everything (see gist above for details)
  Py_Initialize(); 
  PySys_SetArgv(argc, (wchar_t**)argv);  
  PyObject *sys_path = PySys_GetObject("path");
  PyList_Append(sys_path, PyUnicode_FromString("."));    
  import_array1(-1);  
  m_PyModule = PyImport_ImportModule("pythonScript"); 
  
  if (m_PyModule != NULL)
  {
    // get dictionary of available items in the module
    m_PyDict = PyModule_GetDict(m_PyModule);

    // grab the functions we are interested in
    m_PyFooBar = PyDict_GetItemString(m_PyDict, "foo_bar");
    
    // execute the function    
    if (m_PyFooBar != NULL)
    {
      // take a cv::Mat object from somewhere (we'll just create one)
      cv::Mat img = cv::Mat::zeros(480, 640, CV_8UC3);
      
      // total number of elements (here it's an RGB image of size 640x480)
      const unsigned int nElem = 640 * 480 * 3;
      
      // create an array of apropriate datatype
      uchar* m = new uchar[nElem];
      
      // copy the data from the cv::Mat object into the array
      std::memcpy(m, img.data, nElem * sizeof(uchar));
      
      // the dimensions of the matrix
      npy_intp mdim[] = { 480, 640, 3 };
      
      // convert the cv::Mat to numpy.array
      PyObject* mat = PyArray_SimpleNewFromData(3, mdim, NPY_UINT8, (void*) m);
      
      // create a Python-tuple of arguments for the function call
      // "()" means "tuple". "O" means "object"
      PyObject* args = Py_BuildValue("(O)", mat);
      
      // execute the function
      PyObject* result = PyEval_CallObject(m_PyFooBar, args);
      
      // process the result
      // ...
      
      // decrement the object references
      Py_XDECREF(mat);
      Py_XDECREF(result);
      Py_XDECREF(args);

      delete[] m;
    }
  }
  else
  {
    std::cerr << "Failed to load the Python module!" << std::endl;
    PyErr_Print();
  }
  
  return 0;
}

pythonScript.py

import cv2
import numpy as np

# define the function that we will call from the C++ code
def foo_bar(img=None):  
    if img is not None:
       cv2.imshow("image in python", img)

@Xonxt
i embeded my script function in c++, and pass cv::mat to it ang get result back
but
my script (without embedding) just take 30 ms
but when i use it in embedding mode and send cv::mat from c++ to python it takes about 80-130 ms

why and how can i solve this problem?

for more details:
PyObject_CallObject(pFunc, pArgs); takes 130 ms
but when i check time of my function in python (that im called) it is about 30 ms

for more details:
PyObject_CallObject(pFunc, pArgs); takes 130 ms
but when i check time of my function in python (that im called) it is about 30 ms

Hi. I'm not sure, since I haven't worked with C++/Python-communication since I'd first posted this gist, but I suspect this is an unavoidable "overhead" that you get when trying to communicate between those two.
An obvious solution would be to just switch to working purely in Python (which is what I did), or purely in C++, or maybe put your C++ code into a library and write a proper Python-wrapper for it.

Thanks Guys Xonxt for this script. It's very helpfull. I run this code on Visual Studio 2017 version 15.9.40 and i don't meet any problem.

But I meet the problem: ModuleNotFoundError: No module named 'numpy'. This problem occur when i using Visual Studio 2019 version 16.11.5. The line code import_array1(-1) is the reason for that.
Can you help me? I don't know why MSVC 2019 imcompatible with numpy API of Python.
thanks for your reading !

Thanks Guys Xonxt for this script. It's very helpfull. I run this code on Visual Studio 2017 version 15.9.40 and i don't meet any problem.

But I meet the problem: ModuleNotFoundError: No module named 'numpy'. This problem occur when i using Visual Studio 2019 version 16.11.5. The line code import_array1(-1) is the reason for that. Can you help me? I don't know why MSVC 2019 imcompatible with numpy API of Python. thanks for your reading !

Did you install the 'numpy' module in your Python and import the C:\Program Files\PythonXX\Lib\site-packages\numpy\core\include into your MSVC project?
If that doesn't work, then I don't know, sorry. I haven't worked with this stuff for a while. Ever since I posted this to be exact :D.

Yes. I already installed numpy package in python. That why i can run your code in VS 2017. Now i want run your code in VS 2019 but i have the problem.
Thanks you response for me.

Yes. I already installed numpy package in python. That why i can run your code in VS 2017. Now i want run your code in VS 2019 but i have the problem. Thanks you response for me.

I have more information, now i return VS 2017 and this code have the same problem :)) . I install two Visual Studio together 2017 and 2019. First 2017 and second 2019.

https://gist.github.com/Xonxt/26d2a9ac6c56505d0896822ede99a646#gistcomment-3723333

I tested it again and I think it didn't have any overhead anymore. thanks for the reply.
@Xonxt

Hi guys, long time no reply :)
I have a new question for you. How i can choose enviromnent for python? It's mean when i call python from c++ script? How to choose env to run python script.
Example: I using ubuntu 18.04 OS . I have 2 virtual environment ( anaconda and venv) and 1 real environment python3.6.9 default of ubuntu (python default). How to i know what env i run python code when call from c++ and how to change it? is it based on python.h (python.a in linux). I think this is similar on window with venv and anaconda and python when install from offcial web site of python.