As a developer writing high-performance C++ libraries, I often find myself needing to integrate them into Python for easier scripting and flexibility. Given the speed and efficiency of C++, it’s crucial for me to maintain that performance while making the functionality accessible from Python.
I frequently turn to pybind11 for this task. It’s a lightweight solution that, once set up, is simple to use and doesn’t add unnecessary overhead.
I want to share some of the tips and tricks I’ve picked up along the way, and show how you can leverage pybind11 to bridge the gap between C++ and Python in your own projects.
Calling a function through pybind11 is straightforward. Once your function is defined, especially if marked with extern "C" for compatibility, the binding is simple. Here’s an example of how to expose a function to Python using pybind11:
#include "binding_function.h"
int add(int i, int j)
{
retun i + j;
}
PYBIND11_MODULE(pybind11_iface, m)
{
// Expose the 'add' function to Python
m.def("add", &add, "A function which adds two numbers", pybind11::arg("i") = 1, pybind11::arg("j") = 1);
}
In this case, the function add is a simple C++ function that takes two arguments and adds them together. By using pybind11::def, the function becomes callable directly from Python. The pybind11::arg declarations allow you to specify default arguments, making it even easier to call from Python. Once bound, the function behaves as a regular Python function while keeping the performance benefits of the underlying C++ implementation.
Calling a C++ class from Python is also quite straightforward when using pybind11. If your class has public members and methods, they can be easily exposed to Python. Here’s an example of how to bind a class with a function to Python:
class CppClass
{
public:
CppClass();
~CppClass();
uint32_t class_function();
};
PYBIND11_MODULE(pybind11_iface, m)
{
// Binding the C++ class to Python
pybind11::class_<CppClass>(m, "CppClass")
.def(pybind11::init<>()) // Bind constructor
.def("class_function", &CppClass::class_function); // Bind method
}
In this example, CppClass is exposed to Python with its constructor and the class_function method. Once bound, you can create an instance of CppClass and call its methods directly from Python. This allows you to seamlessly interact with your C++ logic within Python while keeping the full speed of C++. I will cover more complex topics like handling std::vector later.
pybind11 offers the ability to define C++ classes that can manipulate Python objects, for example, exposing numpy.array memory that can be overwritten in C++. However, since numpy arrays and std::vector are not directly compatible, it’s necessary to copy data back and forth between them. Below is an example where I define a class in C++ that inherits from CppClass and handles numpy.array manipulation.
class PyClass : public CppClass
{
public:
using CppClass::CppClass;
pybind11::memoryview numbers();
void UpdateVectorNumpy(py::array_t<double> array);
};
pybind11::memoryview PyClass::numbers()
{
return pybind11::memoryview::from_memory(n, sizeof(uint32_t) * 4, false);
}
void PyClass::UpdateVectorNumpy(py::array_t<double> array)
{
py::buffer_info buf = array.request();
double* ptr = static_cast<double*>(buf.ptr); // Get the pointer to numpy array memory
size_t size = static_cast<size_t>(buf.size);
// Use the existing C++ function but operate directly on the memory
std::vector<double> vec(ptr, ptr + size); // Create a vector from numpy memory
// Call the original C++ function that modifies the vector
CppClass::UpdateVector(vec);
// Directly copy the modified vector back into the numpy array (in-place update)
for (size_t i = 0; i < size; ++i)
{
ptr[i] = vec[i]; // Reflect changes in the numpy array
}
}
PYBIND11_MODULE(pybind11_iface, m)
{
pybind11::class_<PyClass>(m, "PyClass")
.def(pybind11::init<>())
.def("class_function", &PyClass::class_function)
.def("numbers", &PyClass::numbers)
.def("update_vector", &PyClass::UpdateVector)
.def("update_vector_numpy", &PyClass::UpdateVectorNumpy);
}
In this example, the PyClass class inherits from CppClass and exposes the numbers and UpdateVectorNumpy functions to Python. The numbers function returns a memoryview from a C++ array, and the UpdateVectorNumpy function allows modifying a numpy.array directly from C++, reflecting the changes back into the Python side.
One common issue when working with std::vector in C++ and numpy.array in Python is the memory overhead caused by copying data back and forth between the two. When it’s necessary to return a Python object from C++ without wasting memory, you can define a direct binding to the C++ std::vector<double>. This allows you to allocate the vector directly from Python, manipulate it in C++, and avoid unnecessary data copies.
Below is an example of how to bind std::vector<double> and expose it to Python:
void CppClass2::UpdateVector2(std::vector<double>& vv)
{
for (size_t i = 0; i < vv.size(); ++i) {
vv[i] = (static_cast<double>(i) + 1) * 1.1;
}
}
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
PYBIND11_MAKE_OPAQUE(std::vector<double>)
PYBIND11_MODULE(pybind11_iface, m)
{
pybind11::class_<CppClass2>(m, "CppClass2")
.def(pybind11::init<>())
.def_readwrite("vv_", &CppClass2::vv_) // Expose vv_ as read/write
.def("set_vv", [](CppClass2& self, pybind11::array_t<double> arr) {
// Check that the numpy array is contiguous
pybind11::buffer_info buf = arr.request();
if (buf.ndim != 1) throw std::runtime_error("Expected a 1-dimensional array");
// Convert numpy array to std::vector<double>
self.vv_ = std::vector<double>(static_cast<double*>(buf.ptr), static_cast<double*>(buf.ptr) + buf.size);
})
.def("update_vector2", &CppClass2::UpdateVector2);
// Bind std::vector<double> directly to Python as VectorDouble
pybind11::bind_vector<std::vector<double>>(m, "VectorDouble");
}
With this setup, you can now create and manipulate a std::vector<double> directly from Python:
import py_iface
# Create a VectorDouble from Python
v = py_iface.VectorDouble([0.0, 0.0, 0.0])
# Assign the vector to the class
cpp_obj = py_iface.CppClass2()
cpp_obj.vv_ = v
# Modify the vector from C++ without copying
cpp_obj.update_vector2()
# Access the updated vector from Python
print(cpp_obj.vv_) # Output: [1.1, 2.2, 3.3]
By directly binding std::vector<double>, you avoid the memory overhead of copying data between numpy and std::vector, and you can safely manipulate the data from both Python and C++.
A full example to show the different way of interfacing Python with C/C++ libraries.
Header file binding_function.h.
#ifndef _CPP_FUNCTION_H_6B07A2839E3A49C186BE43F44A0BD10F_
#define _CPP_FUNCTION_H_6B07A2839E3A49C186BE43F44A0BD10F_
#include <iostream>
extern "C" int add(int i = 1, int j = 1);
#endif
Source file binding_function.cpp.
#include "binding_function.h"
int add(int i, int j)
{
return i + j;
}
Header file cpp_class.h.
#ifndef _CPP_CLASS_H_9B674BA51D7C4A1B996665618F54FC51_
#define _CPP_CLASS_H_9B674BA51D7C4A1B996665618F54FC51_
#include <iostream>
#include <vector>
class CppClass
{
public:
CppClass();
~CppClass();
uint32_t class_function();
void UpdateVector(std::vector<double>& vv);
protected:
int n[3]{1, 2, 3};
static constexpr uint32_t _n2 = 4;
};
class CppClass2
{
public:
CppClass2();
~CppClass2();
std::vector<double> vv_;
void UpdateVector2(std::vector<double>& vv);
};
#endif
Source file cpp_class.cpp.
#include "cpp_class.h"
CppClass::CppClass() {};
CppClass::~CppClass() {};
uint32_t CppClass::class_function()
{
std::cout << "Class Member Function Called" << '\n';
return _n2;
};
void CppClass::UpdateVector(std::vector<double>& vv)
{
for (size_t i = 0; i < vv.size(); ++i) { vv[i] = (static_cast<double>(i) + 1) * 1.1; }
}
CppClass2::CppClass2() {};
CppClass2::~CppClass2() {};
void CppClass2::UpdateVector2(std::vector<double>& vv)
{
for (size_t i = 0; i < vv.size(); ++i) { vv[i] = (static_cast<double>(i) + 1) * 1.1; }
}
Header file pybinding_class.h.
#ifndef _PYBINDING_CLASS_H_553A1609016E4FFE9D58E317669E67CB1_
#define _PYBINDING_CLASS_H_553A1609016E4FFE9D58E317669E67CB1_
#include <vector>
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include "binding_class.h"
namespace py = pybind11;
class PyClass : public CppClass
{
public:
using CppClass::CppClass;
pybind11::memoryview numbers();
void UpdateVectorNumpy(py::array_t<double> array);
};
#endif
Source file pybinding_class.cpp.
#include "pybinding_class.hpp"
pybind11::memoryview PyClass::numbers()
{
return pybind11::memoryview::from_memory(n, sizeof(uint32_t) * 4, false);
};
void PyClass::UpdateVectorNumpy(py::array_t<double> array)
{
py::buffer_info buf = array.request();
double* ptr = static_cast<double*>(buf.ptr); // Get the pointer to numpy array memory
size_t size = static_cast<size_t>(buf.size);
// Use the existing C++ function but operate directly on the memory
std::vector<double> vec(ptr, ptr + size); // Create a vector from numpy memory
// Call the original C++ function that modifies the vector
CppClass::UpdateVector(vec);
// Directly copy the modified vector back into the numpy array (in-place update)
for (size_t i = 0; i < size; ++i)
{
ptr[i] = vec[i]; // Reflect changes in the numpy array
}
};
Source file py_iface.cpp.
#include <iostream>
#include <vector>
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include "binding_function.h"
#include "pybinding_class.h"
PYBIND11_MAKE_OPAQUE(std::vector<double>)
PYBIND11_MODULE(pybind11_iface, m)
{
m.doc() = "pybind11 example plugin";
m.attr("test_integer") = 1;
pybind11::object one_string = pybind11::cast("This is a string");
m.attr("test_string") = one_string;
m.def("add", &add, "A function which adds two numbers", pybind11::arg("i") = 1, pybind11::arg("j") = 1);
pybind11::class_<PyClass>(m, "PyClass")
.def(pybind11::init<>())
.def("class_function", &PyClass::class_function)
.def("numbers", &PyClass::numbers)
.def("update_vector", &PyClass::UpdateVector)
.def("update_vector_numpy", &PyClass::UpdateVectorNumpy);
pybind11::class_<CppClass>(m, "CppClass")
.def(pybind11::init<>())
.def("class_function", &CppClass::class_function)
.def("update_vector", &CppClass::UpdateVector);
pybind11::class_<CppClass2>(m, "CppClass2")
.def(pybind11::init<>())
.def_readwrite("vv_", &CppClass2::vv_) // Expose vv_ as read/write
.def("set_vv", [](CppClass2& self, pybind11::array_t<double> arr) {
// Check that the numpy array is contiguous
pybind11::buffer_info buf = arr.request();
if (buf.ndim != 1) throw std::runtime_error("Expected a 1-dimensional array");
// Convert numpy array to std::vector<double>
self.vv_ = std::vector<double>(static_cast<double*>(buf.ptr), static_cast<double*>(buf.ptr) + buf.size);
}).def("update_vector2", &CppClass2::UpdateVector2);
pybind11::bind_vector<std::vector<double>>(m, "VectorDouble");
}
The relevant compilation part of CMakeList.txt.
cmake_minimum_required(VERSION 3.13.4)
project(pybind11_iface)
set (PROJECT_VERSION "1.0")
project(${PROJECT_NAME} VERSION ${PROJECT_VERSION})
# Other C++ or CMake flags
if(WIN32)
# On Windows, the pybind11 path is different
set(CMAKE_PREFIX_PATH "$ENV{VIRTUAL_ENV}/Lib/site-packages/pybind11/share/cmake/pybind11")
else()
# On macOS/Linux, use Python to detect the Python version and set the path
set(PYTHON_EXECUTABLE "python3")
# Get the Python version from the current virtual environment
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')"
OUTPUT_VARIABLE PYTHON_VERSION
OUTPUT_STRIP_TRAILING_WHITESPACE
)
# Set the CMAKE_PREFIX_PATH using the detected Python version
set(CMAKE_PREFIX_PATH "$ENV{VIRTUAL_ENV}/lib/python${PYTHON_VERSION}/site-packages/pybind11/share/cmake/pybind11")
endif()
set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 REQUIRED)
pybind11_add_module(pybind11_iface pybind11_iface.cpp pybinding_class.cpp binding_class.cpp binding_function.cpp)
Source file py_bindings.py.
import pybind11_iface as py_iface
import numpy as np
import struct
print('call a function:', py_iface.add(1, 2))
t = py_iface.PyClass()
print('int from a cpp member function:', t.class_function(), " - expected 4")
mm = t.numbers().tobytes()
_d = struct.unpack('i', mm[4:8])
print("unpack n[1]:", _d[0], " - expected 2")
vec = np.zeros(6)
t.update_vector_numpy(vec)
print (f"vec after update with dedicated interface {vec} [working]")
t2 = py_iface.CppClass()
vec = py_iface.VectorDouble([0., 0., 0.])
print (f"vec before update {vec}")
t2.update_vector(vec)
print (f"vec after update with C++ vector<double> {vec} [working]")
t3 = py_iface.CppClass2()
t3.set_vv(np.zeros(6))
print (f"vec before update {t3.vv_}")
t3.update_vector2(t3.vv_)
print (f"vec after update with C++ vector<double> {t3.vv_} [working]")
Results from python py_bindings.py.
python py_bindings.py
call a function: 3
Class Member Function Called
int from a cpp member function: 4 - expected 4
unpack n[1]: 2 - expected 2
vec after update with dedicated interface [1.1 2.2 3.3 4.4 5.5 6.6]
vec before update VectorDouble[0, 0, 0]
vec after update with C++ vector<double> VectorDouble[1.1, 2.2, 3.3]
vec before update VectorDouble[0, 0, 0, 0, 0, 0]
vec after update with C++ vector<double> VectorDouble[1.1, 2.2, 3.3, 4.4, 5.5, 6.6]