Building NVIDIA Triton Inference Server from Scratch for TensorFlow Backend
I hope you are here because you can’t or don’t want to use existing triton inference server docker image. I had the same reason which made me to put my 3 weeks of effort to build triton inference server from scratch on RHEL7 platform. I found it very laborious as there is no enough documentation for it. It was test and trial kind of effort and it is very time consuming. I am sharing my experience, so it can save your time. Please feel free to give your suggestion and feedback.
Please refer to https://github.com/triton-inference-server/server/blob/main/docs/build.md#building-on-unsupported-platforms before going through my page. My blog helps you to build it without Docker. I have created my own docker image of it. I have build this for only for tensorflow backend. If you want for other backend then you may you have to try yourself.
Add Essential Yum Repo: cuda-rhel7
yum-config-manager — add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo
Install Require Libraries through yum:
yum install -y git vim autoconf automake boost-devel make openssl-devel python3-devel wget bzip2 patchelf bison gcc gcc-c++ libnccl libnccl-devel libnccl-static docker
We need GCC 9.3.0; I did not find it centos7 repo so I have installed it myself.
mkdir gcc-build && cd gcc-build && GCC_VERSION=9.3.0 && wget https://ftp.gnu.org/gnu/gcc/gcc-${GCC_VERSION}/gcc-${GCC_VERSION}.tar.gz && tar xzvf gcc-${GCC_VERSION}.tar.gz && mkdir obj.gcc-${GCC_VERSION} && cd gcc-${GCC_VERSION} && ./contrib/download_prerequisites && cd ../obj.gcc-${GCC_VERSION} && ../gcc-${GCC_VERSION}/configure — disable-multilib — enable-languages=c,c++&& make -j 3 && make install
We need cmake-3.19 version. The default one is a lesser version. We need 3.19 version of cmake to install triton
wget https://github.com/Kitware/CMake/releases/download/v3.19.8/cmake-3.19.8.tar.gz && tar -xf cmake-3.19.8.tar.gz && cd cmake-3.19.8 && ./bootstrap && make && make install
You require re2 of latest version. Download and install it.
git clone https://github.com/google/re2.git && cd re2 && make && make test && make install && make
git checkout the triton code. Go to the branch r21.04
git clone https://github.com/triton-inference-server/server.git && cd server && git checkout -b r21.04 origin/r21.04
Install the triton using build script:
./build.py — cmake-dir=/server/build<this is the path to build directory of inside tritonserver code parent directoy> — build-dir=/tmp/citritonbuild — no-container-build — endpoint=http — endpoint=grpc — repo-tag=common:r21.04 — repo-tag=core:r21.04 — repo-tag=backend:r21.04 — repo-tag=thirdparty:r21.04 — backend=tensorflow1:r21.04 — enable-logging — enable-stats — enable-tracing
We need glib of 2.29 version. Be default centos rhel7 has 2.17.
we need make-4.3 to install glib 2.29, so we download and install make-4.3 version. Default version is smaller in centos7.
wget https://ftp.gnu.org/gnu/make/make-4.3.tar.gz && tar xf make-4.3.tar.gz && cd make-4.3 && ./configure — prefix=/usr/local && make && make install
It is good to uninstall the older one and then download and install the glib2.29
yum remove make
wget https://ftp.gnu.org/gnu/glibc/glibc-2.29.tar.gz && tar xf glibc-2.29.tar.gz && cd glibc-2.29 && mkdir glibc-build && cd glibc-build && ../configure — prefix=/usr && /usr/local/bin/make && /usr/local/bin/make install
Directory /tmp/citritonbuild contains the installation of triton and pytorch backend. Move the /tmp/citritonbuild/opt/tritonserver directory to /opt directory
cp -r /tmp/citritonbuild/opt/tritonserver /opt
Set the path:
export PATH=$PATH:/opt/tritonserver/bin
If you don’t want to do this yourself then feel free to download this docker image from docker hub.
https://hub.docker.com/repository/docker/chandramohanmeena/centos7-tritonserver-tensorflow