Create a basic redistributable OpenVINO GenAI application

_README.md

This gist shows how to install OpenVINO GenAI from archives, create a basic GenAI C++ application, and install that application including required DLLs in a directory. For the same example with building OpenVINO and OpenVINO GenAI from source, see this gist for building OpenVINO 2024.3 or this gist for building the master branch.

The example runs inference on CPU. To run on GPU, change CPU to GPU in example.cpp and add the following libraries to the install(FILES) section of CmakeLists.txt: openvino_intel_gpu_plugin.dll, opencl.dll (C:\Windows\System32\opencl.dll) and cache.json (.\runtime\bin\intel64\Release\cache.json).

See https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp for more OpenVINO GenAI example code.

This code was tested on Windows 11 with Microsoft Visual Studio 2022, CMake 3.28.0-msvc1 (shipped with Visual Studio), with a Developer Command Prompt. When using PowerShell, replace setupvars.bat with setupvars.ps1.

System requirements

• Microsoft Visual Studio 2019 or higher, version 16.3 or later
• CMake 3.23 or higher (Visual Studio 2022 version is fine)

Install OpenVINO GenAI

Download the OpenVINO GenAI archive from https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/2024.3/windows/openvino_genai_windows_2024.3.0.0_x86_64.zip and extract the contents to C:\Users\USER\tools\openvino_20243 (when using a different directory, adapt the paths in CMakeLists.txt).

Note that some unzip tools create an extra directory. C:\Users\USER\tools\openvino_20243 should contain setupvars.bat, it should not be in a subdirectory.

For example (in cmd.exe):

mkdir %USERPROFILE%/tools/openvino_20243
curl https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/2024.3/windows/openvino_genai_windows_2024.3.0.0_x86_64.zip --output openvino_20243.zip
tar -xf openvino_20243.zip -C %USERPROFILE%/tools/openvino_20243 --strip-components=1

Build the sample application

Note: Make sure to always run %USERPROFILE%\tools\openvino\setupvars.bat before building the application. For PowerShell, use setupvars.ps1.

Put CMakeLists.txt and example.cpp from this gist in a directory and cd to that directory. Then run:

%USERPROFILE%\tools\openvino_20243\setupvars.bat
mkdir build && cd build
cmake ..
cmake --build . --config Release
cmake --install . --prefix %USERPROFILE%\tools

Run the example

Running the example code requires an OpenVINO LLM model. To export an example model, run optimum-cli export openvino -m gpt2 gpt2-ov after installing the dependencies with: pip install --upgrade --upgrade-strategy eager optimum[openvino]. These dependencies are only needed for exporting the model. See the documentation for more options.

NOTE: It is recommended to use a separate command prompt for exporting the model, to prevent potential environment issues between the Python environment with OpenVINO installed, and the build environment.

cd %USERPROFILE%\tools\GenaiExample
example gpt2-ov

The directory C:\Users\USER\GenaiExample contains all the necessary files for running the sample application. You can copy the directory (including the model) to another computer and run the example there. Note that the other computer needs to have Visual C++ Redistributable installed.

CMakeLists.txt

cmake_minimum_required(VERSION 3.10)
project(GenaiExampleProject)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED True)
find_package(OpenVINOGenAI REQUIRED)
# Replace backslashes in %USERPROFILE%
string(REPLACE "\\" "/" USERPROFILE_FIXED $ENV{USERPROFILE})
include_directories("${USERPROFILE_FIXED}/tools/openvino_20243/runtime/include")
link_directories("${USERPROFILE_FIXED}/tools/openvino_20243/runtime/lib/intel64/Release")
add_executable(example example.cpp)
target_link_libraries(example PRIVATE openvino::genai)
install(TARGETS example RUNTIME DESTINATION GenaiExample)
install(FILES
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/openvino.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/openvino_genai.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/openvino_tokenizers.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/core_tokenizers.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/openvino_ir_frontend.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/icuuc70.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/icudt70.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/openvino_tensorflow_frontend.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/bin/intel64/Release/openvino_intel_cpu_plugin.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/3rdparty/tbb/bin/tbb12.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/3rdparty/tbb/bin/tbbmalloc.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/3rdparty/tbb/bin/tbbmalloc_proxy.dll"
        "${USERPROFILE_FIXED}/tools/openvino_20243/runtime/3rdparty/tbb/bin/tbbbind_2_5.dll"
        DESTINATION GenaiExample)
if(WIN32)
    add_definitions(-DWINDOWS_PLATFORM)
endif()

example.cpp

#include "openvino/genai/llm_pipeline.hpp"
#include <iostream>

int main(int argc, char* argv[]) {
   std::string model_path = argv[1];
   ov::genai::LLMPipeline pipe(model_path, "CPU");
   ov::genai::GenerationConfig config = pipe.get_generation_config();
   config.max_new_tokens = 32;
   std::cout << pipe.generate("The Sun is yellow because", config);
}

What I tried is to build directly at my_path/openvino.genai, then with the above commands, everything worked very well. I actually didn't "Put CMakeLists.txt and example.cpp from this gist in a directory and cd to that directory.".

The goal of this gist is to build the standalone sample application in example.cpp, and install with the required OpenVINO libraries. You would need to put these files in some directory to do that :-)

Also, building from "my_path/openvino.genai", I only changed "CPU" to "GPU" in chat_sample.cpp and built again to make the inference working on GPU, without taking this step "To run on GPU, change CPU to GPU in example.cpp and add the following libraries to the install(FILES) section of CmakeLists.txt: openvino_intel_gpu_plugin.dll, opencl.dll (C:\Windows\System32\opencl.dll) and cache.json (.\runtime\bin\intel64\Release\cache.json)."

You probably tried it on a computer where you already had OpenVINO installed and you had run setupvars before?