# Execution Settings and Hardware Support (CPU/GPU) ## Execution Settings Training or running a **Deep Learning** tool is done through an ***engine*** that supports one or more ***devices***. For inference, an engine and device can support one or more ***inference precision***. #### List of engines | **Engine name** | **Supported devices** | **Supported platforms** | **Training** | **Inference** | **Inference precision** | |---------------------------------------|---------------------------------------------------|---------------------------|----------------|-----------------|------------------------------| | **default** | CPU

NVIDIA GPU[1](#nvidiagpu) | All | Yes | Yes | `FLOAT32` | | **OpenVINO**[2](#openvino) | CPU Intel

GPU

NPU | All | No | Yes | `FLOAT16`

`FLOAT32` | | **TensorRT**[3](#tensorrt) | NVIDIA GPU | All | No | Yes | `FLOAT16`

`FLOAT32` | * The main engine is named `default`: * It is always available (it does not require the installation of any additional package) and always supports the CPU of the computer on which it is running. * It always supports both inference and training. * It supports NVIDIA GPU when the package `deep-learning-redist` is installed and the NVIDIA drivers are correctly installed and configured. * The other engines: * Use them to increase the inference speed. * The other engines and the support for a GPU are provided by the package named `deep-learning-redist`. * The inference precision is about the underlying data type used to perform the computation * `FLOAT32` means single-precision floating point values as defined in the IEEE 754 standard. Each number occupies 32 bits. This is the default value and the one used for training. * `FLOAT16` means half-precision floating point values as defined in the IEEE 754 standard. Each number occupies 16 bits. It can increase the inference speed at the cost of a small difference in results compared to using `FLOAT32`. #### NOTE The NVIDIA GPU devices, OpenVINO and TensorRT engines requires the `deep-learning-redist` package. ### Deep Learning Studio In **Deep Learning Studio**, there are two sets of `Execution settings`: * For training, select one or more `Training device`. The **default** engine will be used for training. See [Using Multiple GPUs](#easydeeplearning-hardware-multi-gpu) for more information about multi GPU training. ![image](assets/images/easydeeplearning/hardware__support_1.webp) * For inference, select the `Engine`, `Device` and `Precision` that you want to use. ![image](assets/images/easydeeplearning/hardware__support_6.webp) ### API * Running (inference) or training a **Deep Learning** tool is done through an engine. The engines are specified through their names. * To set an engine for a tool, use [`EDeepLearningTool::SetEngine`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a39c6edce5907037f36587b61daaa29b9) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a0fc61e38444304067006cbe82001b81e) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ae9dee33e5db15d0a07982475b5f9311e) * To list the available engines, use [`EDeepLearningTool::GetAvailableEngines`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a9c0f9da841fe660ad78c647f4933dac3) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a4e2c9f95cc25e227a66588154f856b15) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a8d9202198ae350d4282eaa253eadcd7a) * An engine can support different devices: * To list the detected devices for the current engine, use [`EDeepLearningTool::GetAvailableDevices`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#abe83f5b510a316f2c6d8ddb663d5a616) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a0e4ec5010e79fee8d508cbad7b084224) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ac1c7b724440288aa08912ca03a0664b4) * To list the detected devices for another engine, use [`EDeepLearningTool::GetAvailableDevicesForEngine`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a14b369d6e2064fc904897af30879ac76) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#afaf5671cb675b00e34c9a83fb27cd74c) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a403446e6b3e5ed7d993ab3aa7fea3f55). Note that this actually loads the engine in memory. * The devices are represented by the class [`EDeepLearningDevice`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html) or directly through their name ([`EDeepLearningDevice::GetName`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#a59da8c499110c529a781e6a1e62d47a6) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#aeaada39d280aa6f79f6b1280dd99bd99) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#a77ac4e2c69fbffc1ebe9770ec821be9c)) that are guaranteed to be unique for a given engine. * To set a device, use [`EDeepLearningTool::SetActiveDevices`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a098a0f5ecca5ddffa67c2e7b64ead1c1) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a3085d15de296e477be1995213c3b2570) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#adb34aba126cd0678b531da24ddfc8808) or [`EDeepLearningTool::SetActiveDeviceByName`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#adb1c4d7bb85f870ede914a61b956b5d1) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a9832f9b8927c3fa56f8dcdcc52858ffc) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a281def0b6fbb2a9afbd1887e5d5cb890). You can pass multiple devices for multi-GPU training for example. * To know if a device supports inference and/or training, use [`EDeepLearningDevice::HasInferenceCapability`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#af123cc7890ed560e4909bdd064ea0133) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#ad95045df471bb74aa0eebde38948929d) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#a6b3a9d5f62694ef4971ee01471e92d71) and [`EDeepLearningDevice::HasTrainingCapability`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#a00f2115dc2b16091c939831ecf42c306) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#ab5ab8cbe046b6f6d527cffa949f958d7) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningDevice.html#a089fa66294a128c8b3434b80d7db51e0). Check the matrix below for each engine and device type support. * A device or an engine can support various inference precisions (`FLOAT32` or `FLOAT16`). * By default, the precision is `FLOAT32`. * Use [`EDeepLearningTool::SetInferencePrecision`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#af8c6634f9dd02fe86d709c0fb18dd400) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ad857dc25039b180862f2a67ad5ce0714) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#af58b405b6461c0f9b7e284e585e50975) to set or retrieve the current precision. * A lower precision (for example `FLOAT16` instead of `FLOAT32`) can improve the speed. ### Engine Initialization The initialization of some engines can take quite a long time (for example the `TensorRT` engine can take up to a few minutes to initialize). To avoid having to perform this initialization each time you load the **Deep Learning** tool when your execution device, inference precision and batch size do not change, use the ***engine setup data***. In the API: * To save the “engine setup data” along with your **Deep Learning** tool, save your tool after calling [`EDeepLearningTool::InitializeInference`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a5a70b323809870d9a2512d46442718b0) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a0c889e8f70fedd86e77496edfd55a3ad) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a841de09ff474b83dd9774f49cdf7d32c) or after making an inference with it. * Check whether the tool has ***engine setup data*** with [`EDeepLearningTool::HasEngineSetupData`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ae6b2342f9c8d665add4825af9f7f63ec) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ac9ba52e7ab9c1a7648151cae0767114b) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ad953a52838eb7a366db74caf1d166dfc). * Check the execution settings for which the ***engine setup data*** are for with: * For the device: [`EDeepLearningTool::GetEngineSetupDataDevice`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#aa109e24120edeeee4e99a1049e03b5d8) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a72e766b27be0a35a37b494cbc886880a) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a505927849b1d0d537a4df45dfa31e58f) * For the inference precision: [`EDeepLearningTool::GetEngineSetupDataInferencePrecision`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a074b533b0b9865e1fdc94d07408a238c) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ad4cd2254e7fd2969551b22094676f233) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#ae542ee725e77577291969d8a393b7127) * For the batch size: [`EDeepLearningTool::GetEngineSetupDataBatchSize`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a6a19bab44ff40e3f28355770087c4d00) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#af348c71b03a8984e46f283d3a08dcbba) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a7919fb9dc38f5903b05850796a60560c) #### WARNING For some **NVIDIA** GPUs or **Jetson Orin** boards, using a NVIDIA GPU can trigger a long initialization (over a minute), independently of the engine initialization presented above. To avoid this initialization each time you use a NVIDIA GPU, we recommend to increase the **CUDA** cache max size to 1024 MB: * On Linux, set the environment variable `CUDA_CACHE_MAXSIZE` to `1073741824` before launching or your program. In a terminal, this means executing: ```bash $ CUDA_CACHE_MAXSIZE=1073741824 /path/to/your/program ``` * On Windows, launch the `Advanced System settings`, go to the `Environment` variables dialog and add or modify the `CUDA_CACHE_MAXSIZE` variable with the value `1073741824`. ### Multicore Processing The deep learning tools support multicore processing with the engines `default` and `EasyDeepLeanringEngine_OpenVINO` (see [Multicore Processing](../advanced_operations/multicore_processing.html.md)): * In the API, use the multicore processing helper function from (that is [`Easy::SetMaxNumberOfProcessingThreads`](../../doxygen_cpp/classeVision_1_1Easy.html#a07daba7dfbeff40db5e1d9e4404d4722) [.NET](../../doxygen_cs/classeVision_1_1Easy.html#ae729364026373ca376e818d4b82cd978) [Python](../../doxygen_python/classeVision_1_1Easy.html#ae819b646aea6778997c6155442a0ccee) with a value greater than 1). * In **Deep Learning Studio**, click on the `Configure` button below the `Execution profile` control and select `CPU Settings` in the menu. ![image](assets/images/easydeeplearning/hardware__support_4.webp) ### Additional Consideration for NVIDIA CUDA® GPU #### NOTE Using a recent NVIDIA GPU greatly accelerates the processing speeds. * To use an NVIDIA GPU with the tools, install the `deep-learning-redist` package for your operating system. In this version of **eVision**, the GPU acceleration is based on: * **NVIDIA CUDA® Toolkit** version v12.9. * **NVIDIA CUDA® Deep Neural Network** library (**cuDNN**) v9.10.2. * **NVIDIA TensorRT** library v10.12 on **x64** platforms and v10.6 on **aarch64** platforms. #### NOTE For **Linux ARM** (**aarch64 Jetson** platforms), the following version has been tested: - **JetPack 6** (tested on version 6.2 (L4T 36.4.3)) - **JetPack 7** (tested on version 7.2 (L4T 39.2.1)) * For **Windows** and **Linux x64**, check that you have or install the up-to-date **NVIDIA** drivers. GPU from compute capability 7.5 (**Turing/RTX2000** series) to 12.1 (**Blackwell/RTX5000** series) are supported. Refer to your GPU documentation to install recent drivers for your operating system. * For **Linux ARM** (**aarch64 Jetson** platforms), install the **NVIDIA JetPack SDK** version 6.2 or higher. * Use the engine `default` for training and inference or the engine `EasyDeepLearningEngine_TensorRT` for inference only. ### Using Multiple GPUs You can use multiple **NVIDIA** GPUs with the default engine for the training and the batch classification by specifying multiple **NVIDIA** GPUs devices with [`EDeepLearningTool::SetActiveDevices`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a098a0f5ecca5ddffa67c2e7b64ead1c1) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a3085d15de296e477be1995213c3b2570) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#adb34aba126cd0678b531da24ddfc8808): * Using multiple GPUs increases the training and batch classification speed only if these GPUs are **Quadro** or **Tesla** models with the TCC driver model. See [https://docs.nvidia.com/gameworks/content/developertools/
\\\\ desktop/nsight/tesla_compute_cluster.htm](https://docs.nvidia.com/gameworks/content/developertools/desktop/nsight/tesla_compute_cluster.htm) * Using multiple **GeForce** GPUs does not yield the same performance gain. #### NOTE Be careful to only set multiple GPUs that have similar performances as the performances are limited by the slowest device. * In **Deep Learning Studio**, to choose the training devices, check all the devices that you want to use for training. * You can configure these execution profiles to match your needs. ## Memory The **Deep Learning** tools can consume a lot of memory, especially during training. * Use [`EDeepLearningTool::Clean`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a1ee5b9e1b7150137af6f2bec6da08f5f) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a2d0616f1f162830993b43ec0a7e6b51e) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#aacf8bb1ec3cb65a375f03599d8223c07) to clean up any memory allocated by the selected engine. #### NOTE This does not clean all CPU/GPU memory: some memory includes program data that is statically allocated for the whole program and not for a specific model or tool. The next inference after a call to [`EDeepLearningTool::Clean`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a1ee5b9e1b7150137af6f2bec6da08f5f) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a2d0616f1f162830993b43ec0a7e6b51e) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#aacf8bb1ec3cb65a375f03599d8223c07) takes longer since the engine must reinitialize itself and allocate its memory again (unless [`EDeepLearningTool::InitializeInference`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a5a70b323809870d9a2512d46442718b0) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a0c889e8f70fedd86e77496edfd55a3ad) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a841de09ff474b83dd9774f49cdf7d32c) was called). * For training, use the **image cache** to reserve a part of your main memory for storing images. * The default size is 1 GB. * The training speed is higher when the image cache is big enough to hold all the images of your dataset. * To specify the cache size in bytes: * In the API, use the [`EDeepLearningTool::SetImageCacheSize`](../../doxygen_cpp/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#af722250b3a227a7ee953a2e89ba6c9b7) [.NET](../../doxygen_cs/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a02e9fd244e561e6546fe68e28eb1b2a4) [Python](../../doxygen_python/classeVision_1_1EasyDeepLearning_1_1EDeepLearningTool.html#a9a2702200cd35448d5cfd593d4c327ce) method. * In **Deep Learning Studio**, click on the `Hardware settings` button in the `Execution settings` panel and select `Image cache` in the menu. ![](assets/images/easydeeplearning/hardware__support_2.webp)![](assets/images/easydeeplearning/hardware__support_3.webp) * When there is enough memory to increase the image cache so that it can hold all the images in the dataset, **Deep Learning Studio** displays a recommendation next to the training button. Click on the recommendation to change the image cache size and improve the training speed. ![image](assets/images/easydeeplearning/hardware__support_5.webp) #### HINT With a dataset too big to fit in the image cache, we recommend using a SSD drive to hold your images and project files as a SSD drive improves the training speed. --- * **[1]** The **default** engine requires the `deep-learning-redist` package to work with NVIDIA GPUs. * **[2]** The full name of the OpenVINO engine is `EasyDeepLearningEngine_OpenVINO`. It is available in the `deep-learning-redist` package. * **[3]** The full name of the TensorRT engine is `EasyDeepLearningEngine_TensorRT`. It is available in the `deep-learning-redist` package.