编译MediaPipe框架

编译MediaPipe框架。

最近开始研究自己的大创项目，一个关于动作捕捉的小玩意儿，第一步就是抄袭开源代码借鉴他人优秀成果。在众多的项目中，我看上了这个Google开源的优秀框架，先把这个项目在本地上跑起来再说。这篇文章就记录了我编译这个框架的过程。

在我写完这篇文章之后，我就从WSL润到了Arch Linux，然而我还是有编译MediaPipe的需求，所以这篇文章就增加了Arch Linux下编译MediaPipe的过程。

首先是在Arch Linux下编译的过程。

编译环境概述

使用Arch Linux，需要注意的是Arch Linux采用滚动更新，目前的安装方法可能在不久的将来就不适用了。

编译

环境准备

首先安装bazelisk，由于我安装了pnpm，直接使用pnpm install -g @bazel/bazelisk安装。

然后使用pacman安装：

sudo pacman -S opencv ffmpeg jdk-openjdk git

还有编译的过程中可能会涉及到python中的numpy软件包，我由于已经安装了conda来管理python环境，于是就采用conda install numpy在base环境中安装。

克隆MediaPipe仓库：

git clone https://github.com/google/mediapipe.git

由于pacman仓库中安装的opencv版本是最新的opencv4，我们需要修改MediaPipe中的配置文件来适配，修改third_party/opencv_linux.BUILD：

# Description:
#   OpenCV libraries for video/image processing on Linux

licenses(["notice"])  # BSD license

exports_files(["LICENSE"])

# The following build rule assumes that OpenCV is installed by
# 'apt-get install libopencv-core-dev libopencv-highgui-dev \'
# '                libopencv-calib3d-dev libopencv-features2d-dev \'
# '                libopencv-imgproc-dev libopencv-video-dev'
# on Debian Buster/Ubuntu 18.04.
# If you install OpenCV separately, please modify the build rule accordingly.
cc_library(
    name = "opencv",
    hdrs = glob([
        # For OpenCV 4.x
        #"include/aarch64-linux-gnu/opencv4/opencv2/cvconfig.h",
        #"include/arm-linux-gnueabihf/opencv4/opencv2/cvconfig.h",
        #"include/x86_64-linux-gnu/opencv4/opencv2/cvconfig.h",
        # 将下面这行取消注释
        "include/opencv4/opencv2/**/*.h*",
    ]),
    includes = [
        # For OpenCV 4.x
        #"include/aarch64-linux-gnu/opencv4/",
        #"include/arm-linux-gnueabihf/opencv4/",
        #"include/x86_64-linux-gnu/opencv4/",
        # 将下面这行取消注释
        "include/opencv4/",
    ],
    linkopts = [
        "-l:libopencv_core.so",
        "-l:libopencv_calib3d.so",
        "-l:libopencv_features2d.so",
        "-l:libopencv_highgui.so",
        "-l:libopencv_imgcodecs.so",
        "-l:libopencv_imgproc.so",
        "-l:libopencv_video.so",
        "-l:libopencv_videoio.so",
    ],
    visibility = ["//visibility:public"],
)

编译

跑一下实例中的Hello, World。

首先设置一个环境变量：

$ export GLOG_logtostderr=1

然后一把梭哈：

bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
    mediapipe/examples/desktop/hello_world:hello_world

在第一次编译的时候会下载大量的依赖文件，如果遇到网络错误可以多试几次，我试了三次才完成。

Starting local Bazel server and connecting to it...
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'com_google_absl' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'com_google_benchmark' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'flatbuffers' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'pybind11_bazel' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'com_googlesource_code_re2' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'com_google_protobuf' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'com_google_googletest' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'com_github_gflags_gflags' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'zlib' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'rules_python' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'build_bazel_rules_apple' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'build_bazel_rules_swift' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'build_bazel_apple_support' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'xctestrunner' because it already exists.
DEBUG: /home/ricardo/.cache/bazel/_bazel_ricardo/9d4c3ea39592a9aa5075148d2a9caf3e/external/org_tensorflow/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'pybind11' because it already exists.
WARNING: /home/ricardo/Documents/code/cpp/mediapipe/mediapipe/framework/tool/BUILD:184:24: in cc_library rule //mediapipe/framework/tool:field_data_cc_proto: target '//mediapipe/framework/tool:field_data_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /home/ricardo/Documents/code/cpp/mediapipe/mediapipe/framework/BUILD:54:24: in cc_library rule //mediapipe/framework:calculator_cc_proto: target '//mediapipe/framework:calculator_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
INFO: Analyzed target //mediapipe/examples/desktop/hello_world:hello_world (84 packages loaded, 1747 targets configured).
INFO: Found 1 target...
Target //mediapipe/examples/desktop/hello_world:hello_world up-to-date:
  bazel-bin/mediapipe/examples/desktop/hello_world/hello_world
INFO: Elapsed time: 3.962s, Critical Path: 0.31s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
I20230115 20:26:29.880736 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.880805 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.880817 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.880888 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.880957 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.881028 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.881096 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.881157 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.881198 47247 hello_world.cc:57] Hello World!
I20230115 20:26:29.881268 47247 hello_world.cc:57] Hello World!

编译Pose解决方案的Android Archieve

我的需求有一个是在安卓手机上使用MediaPipe的Pose解决方案，需要编译出一个AAR安卓打包文件。

首先安装需要的依赖，Android SDK和Android NDK。这里使用Android Studio安装。

然后创建一个bazel编译的目标文件：在目录mediapipe/examples/android/src/java/com/google/mediapipe/apps/下创建一个文件夹pose_tracking_aar，在其中创建BUILD文件，写入：

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")

mediapipe_aar(
    name = "mediapipe_pose_tracking",
    calculators = ["//mediapipe/graphs/pose_tracking:pose_tracking_gpu_deps"],
)

使用命令编译：

bazel build -c opt --strip=ALWAYS \
    --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
    --fat_apk_cpu=arm64-v8a,armeabi-v7a \
    --legacy_whole_archive=0 \
    --features=-legacy_whole_archive \
    --copt=-fvisibility=hidden \
    --copt=-ffunction-sections \
    --copt=-fdata-sections \
    --copt=-fstack-protector \
    --copt=-Oz \
    --copt=-fomit-frame-pointer \
    --copt=-DABSL_MIN_LOG_LEVEL=2 \
    --linkopt=-Wl,--gc-sections,--strip-all \
    //mediapipe/examples/android/src/java/com/google/mediapipe/apps/pose_tracking_aar:mediapipe_pose_tracking.aar

如果在编译的过程中提示缺失dx.jar这个文件而且你用的SDK版本还是高于31的，那可能是SDK中缺失了这个文件，可以将SDk降级到30就含有这个文件了。我使用的解决办法比较离奇，我是将30版本的SDK文件中的这个文件软链接过来，解决了这个问题。

编译消耗的时间可能比较的长，耐心等待即可。

为了在手机上使用，我们还需要编译出binarypb文件，从Google的服务器上下载tflite文件。

编译binarypb的过程比较的简单，编译目标在mediapipe/graphs/pose_tracking中，名称是pose_tracking_gpu_binary_graph，使用下列指令编译：

bazel build -c opt //mediapipe/graphs/pose_tracking:pose_tracking_gpu_binary_graph

在这里，Google默认添加了一个input side packet打开人体遮罩，如果不需要这个效果，需要删除mediapipe/graphs/pose_tracking/pose_tracking_gpu.pbtxt文件中的以下内容：

# Generates side packet to enable segmentation.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:enable_segmentation"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { bool_value: true }
    }
  }
}

然后还需要从服务器上下载tflite文件，Pose Tracking这个解决方案需要两个tflite文件，第一个是pose_detection.tflite，第二个文件则有三个不同的选择，分别对于解决方案中提供的三个质量版本：

下载地址是pose_landmark_full.tflite，pose_landmark_heavy.tflite和pose_landmark_lite.tflite。

下面是原来使用WSL编译的过程。

编译环境概述

我使用的基于WSL2的Ubuntu 22.04编译。主要是参考官方的安装文档。

编译

环境准备

首先安装两个基础性的包：

sudo apt install build-essential git

然后安装MediaPipe的编译管理工具bazel，这里我是通过npm安装：

pnpm add -g @bazel/bazelisk

通过运行bazel version验证安装成功：

$ bazel version
Bazelisk version: v1.13.2
WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a WORKSPACE file).
Extracting Bazel installation...
Build label: 5.3.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Oct 19 18:22:12 2022 (1666203732)
Build timestamp: 1666203732
Build timestamp as int: 1666203732

安装Miniconda，再在环境中安装numpy。编译中依赖于Python和numpy，这里网上的资料汗牛充栋，我就不过多赘述。

在准备玩上面的环境之后，我们就可以用git将MediaPipe的源代码克隆下来了。

git clone https://github.com/google/mediapipe.git

安装Opencv和FFmpeg

我这里选择的是手动编译安装opencv，安装的步骤参考官方的安装脚本，但是脚本中的不少内容已经过时。

首先安装必要的依赖库和编译管理工具：

sudo apt install cmake ffmpeg libavformat-dev libdc1394-dev libgtk2.0-dev \
                       libjpeg-dev libpng-dev libswscale-dev libtbb2 libtbb-dev \
                       libtiff-dev

注意：官方脚本中要求安装libdc1394-22-dev这个包，但是按照这篇回答，这个包已经被libdc1392-dev取代了。

在临时文件夹中创建一个文件专门用来编译：

cd /tmp
mkdir opencv
cd opencv/

使用git下载Opencv的源代码：

git clone https://github.com/opencv/opencv_contrib.git
git clone https://github.com/opencv/opencv.git

在仓库中签出到指定的版本分支：

cd opencv
git checkout 3.4
cd ../opencv_contrib
git checkout 3.4

创建编译文件，使用指定的cmake参数生成编译文件：

cd ../opencv
mkdir release
cd release
cmake .. -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local \
          -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_opencv_ts=OFF \
          -DOPENCV_EXTRA_MODULES_PATH=/tmp/opencv/opencv_contrib/modules \
          -DBUILD_opencv_aruco=OFF -DBUILD_opencv_bgsegm=OFF -DBUILD_opencv_bioinspired=OFF \
          -DBUILD_opencv_ccalib=OFF -DBUILD_opencv_datasets=OFF -DBUILD_opencv_dnn=OFF \
          -DBUILD_opencv_dnn_objdetect=OFF -DBUILD_opencv_dpm=OFF -DBUILD_opencv_face=OFF \
          -DBUILD_opencv_fuzzy=OFF -DBUILD_opencv_hfs=OFF -DBUILD_opencv_img_hash=OFF \
          -DBUILD_opencv_js=OFF -DBUILD_opencv_line_descriptor=OFF -DBUILD_opencv_phase_unwrapping=OFF \
          -DBUILD_opencv_plot=OFF -DBUILD_opencv_quality=OFF -DBUILD_opencv_reg=OFF \
          -DBUILD_opencv_rgbd=OFF -DBUILD_opencv_saliency=OFF -DBUILD_opencv_shape=OFF \
          -DBUILD_opencv_structured_light=OFF -DBUILD_opencv_surface_matching=OFF \
          -DBUILD_opencv_world=OFF -DBUILD_opencv_xobjdetect=OFF -DBUILD_opencv_xphoto=OFF \
          -DCV_ENABLE_INTRINSICS=ON -DWITH_EIGEN=ON -DWITH_PTHREADS=ON -DWITH_PTHREADS_PF=ON \
          -DWITH_JPEG=ON -DWITH_PNG=ON -DWITH_TIFF=ON

注意安装自己下载源代码的地址修改-DOPENCV_EXTRA_MUDULES_PATH的值

安装过程中还会下载一系列的依赖包，请注意自己的网络环境

使用make指令进行编译和安装

make -j 16
sudo make install

编辑链接器的配置：

sudo touch /etc/ld.so.conf.d/mp_opencv.conf
sudo bash -c  "echo /usr/local/lib >> /etc/ld.so.conf.d/mp_opencv.conf"
sudo ldconfig -v

然后进行MediaPipe的目录，用脚本进行配置文件的修改：

./setup_opencv.sh config_only

运行首个例子：

export GLOG_logtostderr=1
bazel run --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world:hello_world

在等待一段时间的下载依赖和编译之后，我们可以看见：

I20221110 22:00:50.899885 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.899948 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.899955 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.899960 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.899962 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.899982 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.900000 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.900025 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.900030 14357 hello_world.cc:57] Hello World!
I20221110 22:00:50.900193 14357 hello_world.cc:57] Hello World!

如果出现了各种奇怪的报错，那可以执行这条命令重新安装依赖再编译试试：

bazel clean --expunge