Commit fbfc5af1 authored by Dmitry Petrov's avatar Dmitry Petrov
Browse files

Initial version of meta-coral-edgetpu


Signed-off-by: Dmitry Petrov's avatarDmitry Petrov <dpetrov@dev.rtsoft.ru>
parents
Yocto BSP layer for Google Coral Edge TPU
=========================================
This layer adds the support of Coral Edge TPU to Yocto:
- [Coral Edge TPU devices](https://coral.ai/products/#production-products)
By providing following software packages:
- [Edge TPU runtime library](https://github.com/google-coral/libedgetpu)
- [PyCoral API](https://github.com/google-coral/pycoral)
- [Edge TPU simple camera examples](https://github.com/google-coral/examples-camera)
## 1. Prerequisite(s)
Yocto layer dependencies
```
URI: git://git.yoctoproject.org/poky
branch: zeus
revision: cba967414370e195d109353e51510bd829aa86c3
URI: git://github.com/openembedded/meta-openembedded.git
layers: meta-python, meta-oe
branch: zeus
revision: 9e60d30669a2ad0598e9abf0cd15ee06b523986b
```
## 2. Installation instructions
1. **Integration of meta-coral-edgetpu layer**
To integrate the provided *meta-coral-edgetpu* layer to project, following
lines should be added to *conf/bblayers.conf*:
```
BBLAYERS ?= " \
...
/path/to/sources/meta-openembedded/meta-gnome \
/path/to/sources/meta-openembedded/meta-python \
/path/to/sources/meta-coral-edgetpu \
"
```
2. **Selection of packages**
Out of box, standalone packages, which represent Google Coral support when
combined, are not integrated into the resulted rootfs image. They should be
manually specified in corresponding *conf/local.conf* recipe in advance:
```
# Coral Edge TPU
IMAGE_INSTALL_append = " \
python3-edgetpuvision \
pycoral \
pycoral-tests \
pycoral-examples \
pycoral-benchmarks \
examples-camera \
"
```
3. After meta-coral-edgetpu is included into the project and corresponding
packages are enabled, rebuild Yocto images according to standard procedure.
## 3. Evaluation of the functionality
1. To launch the demo, which processes video stream from camera via
gstreamer, execute following commands:
```
# cd /opt/edgetpu/examples-camera/gstreamer/
# python3 detect.py --videosrc /dev/video1
```
*Note: to make this, connect a display and camera (CSI, USB) to the board.*
2. To launch remaining demos, which do not depend on gstreamer and which can
be found in /opt/edgetpu/pycoral on rootfs, use following commands:
```
# cd /opt/edgetpu/pycoral
# python3 examples/classify_image.py \
--model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \
--labels test_data/inat_bird_labels.txt \
--input test_data/parrot.jpg
```
For more information about pycoral examples, refer to:
* corresponding github repo, e.g.:
[Pycoral classify_image.py example](https://github.com/google-coral/pycoral/blob/master/examples/classify_image.py)
* official Coral website:
[Coral Examples](https://coral.ai/examples/)
## 4. Prevention of overheating
According to [Manage the PCIe module
temperature](https://coral.ai/docs/pcie-parameters/#configure-the-shutdownwarning-temperatures),
Edge TPU modules are prone to overheating which can damage both the module and
the board. Because of that, a special script, which sets temperature
thresholds for Coral module to prevent overheating, is provided along the layer
and can be found at `/opt/edgetpu/set_temp.sh`.
Moreover, special udev rule (`/etc/udev/rules.d/99-edgetpu-temp.rules`) has
been created to launch this script after Edge TPU device is enabled.
If needed, the threshold temperatures can be adjusted by changing
*TEMP_WARN_1* and *TEMP_WARN_2* variables in *set_temp.sh*.
For more information, please refer to: [Configure the shutdown/warning
temperatures](https://coral.ai/docs/pcie-parameters/#configure-the-shutdownwarning-temperatures)
## 5. Known issues
* Although an edgetpu-demo package is provided along with this layer, it
does not work and caused Weston server to crash
* Particular patches for gstreamer1.0-plugins-bad, which are provided by
Google, have not been integrated as they completely break gstreamer
* A third-party [Bazel](https://bazel.build/) build system is used by
libedgetpu and pycoral packages out of box. And as Bazel is not fully supported
by vanilla Yocto, so some build issues may occur. In most cases, they can be
fixed by rebuilding libedgetpu from scratch:
`bitbake -c clean libedgetpu && bitbake libedgetpu`
export JAVA_HOME="${STAGING_LIBDIR_NATIVE}/jvm/openjdk-8-native"
BAZEL_JOBS ??= "4"
# Memory 4GB
BAZEL_MEM ??= "4096"
TS_DL_DIR ??= "${DL_DIR}"
CCACHE_DISABLE = "1"
DEPENDS += "bazel-native \
openjdk-8-native \
"
DEPENDS_append_class-target = " python3"
inherit bazel-base
BAZEL_DIR ?= "${WORKDIR}/bazel"
BAZEL_OUTPUTBASE_DIR ?= "${BAZEL_DIR}/output_base"
export BAZEL_ARGS="--output_user_root=${BAZEL_DIR}/user_root \
--output_base=${BAZEL_OUTPUTBASE_DIR} \
--bazelrc=${S}/bazelrc \
--batch \
"
BAZEL ?= "${BAZEL_DIR}/bazel"
do_prepare_recipe_sysroot[postfuncs] += "do_install_bazel"
do_install_bazel() {
mkdir -p ${BAZEL_DIR}
install -m 0755 ${STAGING_BINDIR_NATIVE}/bazel ${BAZEL_DIR}
create_cmdline_wrapper ${BAZEL} \$BAZEL_ARGS
zip -A ${BAZEL}.real
}
def bazel_get_target_flags(d):
flags = ""
for i in d.getVar("CC").split()[1:]:
flags += "# From CC\n"
flags += "build --conlyopt=%s --cxxopt=%s --linkopt=%s\n" % (i, i, i)
for i in d.getVar("CFLAGS").split():
if i == "-g":
continue
flags += "# From CFLAGS\n"
flags += "build --conlyopt=%s\n" % i
for i in d.getVar("BUILD_CFLAGS").split():
flags += "# From BUILD_CFLAGS\n"
flags += "build --host_conlyopt=%s\n" % i
for i in d.getVar("CXXFLAGS").split():
if i == "-g":
continue
flags += "# From CXXFLAGS\n"
flags += "build --cxxopt=%s\n" % i
for i in d.getVar("BUILD_CXXFLAGS").split():
flags += "# From BUILD_CXXFLAGS\n"
flags += "build --host_cxxopt=%s\n" % i
for i in d.getVar("CPPFLAGS").split():
if i == "-g":
continue
flags += "# From CPPFLAGS\n"
flags += "build --conlyopt=%s --cxxopt=%s\n" % (i, i)
for i in d.getVar("BUILD_CPPFLAGS").split():
flags += "# From BUILD_CPPFLAGS\n"
flags += "build --host_conlyopt=%s --host_cxxopt=%s\n" % (i, i)
for i in d.getVar("LDFLAGS").split():
if i == "-Wl,--as-needed":
continue
flags += "# From LDFLAGS\n"
flags += "build --linkopt=%s\n" % i
for i in d.getVar("BUILD_LDFLAGS").split():
if i == "-Wl,--as-needed":
continue
flags += "# From BUILD_LDFLAGS\n"
flags += "build --host_linkopt=%s\n" % i
for i in d.getVar("TOOLCHAIN_OPTIONS").split():
if i == "-Wl,--as-needed":
continue
flags += "# From TOOLCHAIN_OPTIONS\n"
flags += "build --linkopt=%s\n" % i
return flags
def bazel_get_flags(d):
flags = ""
if d.getVar("BAZEL_JOBS"):
flags += "# From BAZEL_JOBS\n"
flags += "build --jobs=%s --local_cpu_resources=%s\n" % (d.getVar("BAZEL_JOBS"), d.getVar("BAZEL_JOBS"))
if d.getVar("BAZEL_MEM"):
flags += "# From BAZEL_MEM\n"
flags += "build --local_ram_resources=%s\n" % (d.getVar("BAZEL_MEM"))
return flags
bazel_do_configure () {
cat > "${S}/bazelrc" <<-EOF
build --verbose_failures
build --spawn_strategy=standalone --genrule_strategy=standalone
test --verbose_failures --verbose_test_summary
test --spawn_strategy=standalone --genrule_strategy=standalone
build --linkopt=-Wl,--no-as-needed
build --host_linkopt=-Wl,--no-as-needed
build --host_conlyopt=-D_PYTHON_INCLUDE_NATIVE --host_cxxopt=-D_PYTHON_INCLUDE_NATIVE
build --conlyopt=-D_PYTHON_INCLUDE_TARGET --cxxopt=-D_PYTHON_INCLUDE_TARGET
build --strip=never
build --python_path=python3
fetch --distdir=${TS_DL_DIR}
build --distdir=${TS_DL_DIR}
${@bazel_get_flags(d)}
EOF
}
bazel_do_configure_append_class-target () {
cat >> "${S}/bazelrc" <<-EOF
# FLAGS begin
${@bazel_get_target_flags(d)}
# FLAGS end
build --linkopt=-Wl,-latomic
EOF
#sed -i "s:${WORKDIR}:${BAZEL_OUTPUTBASE_DIR}/external/crosstool:g" ${S}/bazelrc
}
EXPORT_FUNCTIONS do_configure
PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/bazel"
inherit unsupportarch
#export YOCTO_NATIVE_SYSROOT="${BAZEL_OUTPUTBASE_DIR}/external/crosstool/recipe-sysroot-native"
export YOCTO_NATIVE_SYSROOT="${WORKDIR}/recipe-sysroot-native"
do_rm_work[prefuncs] += "clean_bazel"
do_clean[prefuncs] += "clean_bazel"
clean_bazel() {
if [ -d ${S} ]; then
cd ${S}
if [ -e ${BAZEL} ] && [ -e ${S}/bazelrc ]; then
${BAZEL} clean
${BAZEL} clean --expunge
fi
fi
rm ${BAZEL_DIR} -rf
}
BAZEL_TARGET_CPU ??= ""
BAZEL_TARGET_CPU_x86 = "x86"
BAZEL_TARGET_CPU_x86-64 = "k8"
BAZEL_TARGET_CPU_arm = "arm"
BAZEL_TARGET_CPU_aarch64 = "aarch64"
python __anonymous() {
if not d.getVar("BAZEL_TARGET_CPU"):
target_arch = d.getVar("TARGET_ARCH")
raise bb.parse.SkipPackage("BAZEL_TARGET_CPU is not set\nTensorFlow does not support Target Arch '%s'" % target_arch)
}
# We have a conf and classes directory, add to BBPATH
BBPATH .= ":${LAYERDIR}"
# We have a packages directory, add to BBFILES
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
${LAYERDIR}/recipes-*/*/*.bbappend"
BBFILE_COLLECTIONS += "coral-edgetpu"
BBFILE_PATTERN_coral-edgetpu := "^${LAYERDIR}/"
BBFILE_PRIORITY_coral-edgetpu = "6"
LAYERSERIES_COMPAT_coral-edgetpu = "zeus"
LAYERDEPENDS_coral-edgetpu = "core freescale-layer"
PREFERRED_VERSION_gstreamer1.0-plugins-base_mx8 = "1.16.2.imx"
PREFERRED_VERSION_gstreamer1.0-plugins-bad_mx8 = "1.16.2.imx"
PREFERRED_VERSION_gstreamer1.0-plugins-good_mx8 = "1.16.2.imx"
PREFERRED_VERSION_gstreamer1.0_mx8 = "1.16.2.imx"
PREFERRED_VERSION_gstreamer1.0-libav_mx8 = "1.16.2"
PREFERRED_VERSION_gstreamer1.0-python_mx8 = "1.16.2"
PREFERRED_VERSION_gstreamer1.0-plugins-ugly_mx8 = "1.16.2"
DESCRIPTION = "Bazel build and test tool"
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
SRC_URI[md5sum] = "cc255121586e849c4de2483a8a5814b6"
SRC_URI[sha256sum] = "c9244e5905df6b0190113e26082c72d58b56b1b0dec66d076f083ce4089b0307"
SRC_URI = "https://github.com/bazelbuild/bazel/releases/download/${PV}/bazel-${PV}-dist.zip \
file://0001-HttpDownloader-save-download-tarball-to-distdir.patch \
file://0001-fix-unzip-command-not-found.patch \
file://0001-python3.patch \
file://0001-add-Yocto-native-sysroot-dir-to-the-default-Bazel-to.patch \
"
inherit native python3native
INHIBIT_SYSROOT_STRIP = "1"
CCACHE_DISABLE = "1"
DEPENDS = "coreutils-native \
zip-native \
openjdk-8-native \
"
S="${WORKDIR}"
inherit bazel-base
EXTRA_BAZEL_ARGS = " \
--host_javabase=@local_jdk//:jdk \
--python_path=python3 \
${@oe.utils.conditional("BAZEL_JOBS", "", "", "--jobs=${BAZEL_JOBS}", d )} \
${@oe.utils.conditional("BAZEL_JOBS", "", "", "--local_cpu_resources=${BAZEL_JOBS}", d )} \
${@oe.utils.conditional("BAZEL_MEM", "", "", "--local_ram_resources=${BAZEL_MEM}", d )} \
"
do_compile () {
TMPDIR="${TOPDIR}/bazel" \
VERBOSE=yes \
EXTRA_BAZEL_ARGS="${EXTRA_BAZEL_ARGS}" \
./compile.sh
}
do_install () {
install -d ${D}${bindir}
install -m 0755 ${S}/output/bazel ${D}${bindir}
}
# Explicitly disable uninative
UNINATIVE_LOADER = ""
From 7225fc4a62a06b654fe7d04a2446a594888a7b8c Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
Date: Tue, 8 Dec 2020 10:58:52 +0800
Subject: [PATCH] HttpDownloader: save download tarball to distdir
It is helpful for collecting tarball url which supports offline build.
Upstream-Status: Inappropriate [oe specific]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../downloader/DownloadManager.java | 62 ++++++++++++-------
1 file changed, 38 insertions(+), 24 deletions(-)
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
index a89077e..ccaadb3 100755
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java
@@ -152,33 +152,36 @@ public class DownloadManager {
} else if (!dir.isDirectory()) {
eventHandler.handle(Event.warn("distdir " + dir + " is not a directory"));
} else {
- for (String name : candidateFileNames) {
+ for (String candidateFileName : candidateFileNames) {
boolean match = false;
- Path candidate = dir.getRelative(name);
- try {
- eventHandler.post(
- new CacheProgress(
- mainUrl.toString(), "Checking " + cacheKeyType + " of " + candidate));
- match = RepositoryCache.getChecksum(cacheKeyType, candidate).equals(cacheKey);
- } catch (IOException e) {
- // Not finding anything in a distdir is a normal case, so handle it absolutely
- // quietly. In fact, it is common to specify a whole list of dist dirs,
- // with the assumption that only one will contain an entry.
- } finally {
- eventHandler.post(new CacheProgress(mainUrl.toString()));
- }
- if (match) {
- if (isCachingByProvidedChecksum) {
- try {
- repositoryCache.put(cacheKey, candidate, cacheKeyType, canonicalId);
- } catch (IOException e) {
- eventHandler.handle(
- Event.warn("Failed to copy " + candidate + " to repository cache: " + e));
+ String[] names = {candidateFileName, candidateFileName+"_"+cacheKey};
+ for (String name: names) {
+ Path candidate = dir.getRelative(name);
+ try {
+ eventHandler.post(
+ new CacheProgress(
+ mainUrl.toString(), "Checking " + cacheKeyType + " of " + candidate));
+ match = RepositoryCache.getChecksum(cacheKeyType, candidate).equals(cacheKey);
+ } catch (IOException e) {
+ // Not finding anything in a distdir is a normal case, so handle it absolutely
+ // quietly. In fact, it is common to specify a whole list of dist dirs,
+ // with the assumption that only one will contain an entry.
+ } finally {
+ eventHandler.post(new CacheProgress(mainUrl.toString()));
+ }
+ if (match) {
+ if (isCachingByProvidedChecksum) {
+ try {
+ repositoryCache.put(cacheKey, candidate, cacheKeyType, canonicalId);
+ } catch (IOException e) {
+ eventHandler.handle(
+ Event.warn("Failed to copy " + candidate + " to repository cache: " + e));
+ }
}
+ FileSystemUtils.createDirectoryAndParents(destination.getParentDirectory());
+ FileSystemUtils.copyFile(candidate, destination);
+ return destination;
}
- FileSystemUtils.createDirectoryAndParents(destination.getParentDirectory());
- FileSystemUtils.copyFile(candidate, destination);
- return destination;
}
}
}
@@ -200,6 +203,17 @@ public class DownloadManager {
eventHandler.handle(Event.info("SHA256 (" + urls.get(0) + ") = " + newSha256));
}
+ for (Path dir : distdir) {
+ if (!dir.exists())
+ FileSystemUtils.createDirectoryAndParents(dir);
+
+ if (dir.isDirectory()) {
+ Path dl_mirror = dir.getRelative(destination.getBaseName()+"_"+checksum.get().toString());
+ if (!dl_mirror.exists())
+ FileSystemUtils.copyFile(destination, dl_mirror);
+ }
+ }
+
return destination;
}
--
2.18.2
From 34c4dc1a10140addf75d3503d4b9f427303fe212 Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
Date: Mon, 14 Dec 2020 16:45:31 +0800
Subject: [PATCH] add Yocto native sysroot dir to the default Bazel toolchain
While using the default Bazel C++ toolchain to build Yocto native tools
(bazel build --host_crosstool_top=@bazel_tools//tools/cpp:toolchain),
it failed `bazel references a path outside of the execution root',
Add Yocto native sysroot dir (YOCTO_NATIVE_SYSROOT) to
builtin_include_directories could fix the issue
If not set YOCTO_NATIVE_SYSROOT, use NOT_SET_YOCTO_NATIVE_SYSROOT
to replace as a warning
Upstream-Status: Inappropriate [oe specific]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
tools/cpp/unix_cc_configure.bzl | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tools/cpp/unix_cc_configure.bzl b/tools/cpp/unix_cc_configure.bzl
index d48485b..0d297bf 100755
--- a/tools/cpp/unix_cc_configure.bzl
+++ b/tools/cpp/unix_cc_configure.bzl
@@ -443,6 +443,9 @@ def configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools):
paths["@bazel_tools//tools/cpp:generate_system_module_map.sh"],
))
+ # Customize for Yocto
+ builtin_include_directories.append(get_env_var(repository_ctx,"YOCTO_NATIVE_SYSROOT", "NOT_SET_YOCTO_NATIVE_SYSROOT"))
+
write_builtin_include_directory_paths(repository_ctx, cc, builtin_include_directories)
repository_ctx.template(
"BUILD",
--
2.18.2
From 23e2aff67a03127572641c7286e306c2a20990e2 Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
Date: Tue, 8 Dec 2020 11:05:13 +0800
Subject: [PATCH] fix unzip: command not found
If host does not provide unzip, build bazel will fail even though
Yocto native sysroot does have unzip.
The reason is var-PATH was not passed to bazel build in some cases.
Remove hardcoded /bin and /usr/bin from PATH
Upstream-Status: Inappropriate [wr-installer specific]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../devtools/build/lib/bazel/rules/BazelRuleClassProvider.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java b/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java
index 16252df..e7cd609 100755
--- a/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java
@@ -469,7 +469,7 @@ public class BazelRuleClassProvider {
// Note that --action_env does not propagate to the host config, so it is not a viable
// workaround when a genrule is itself built in the host config (e.g. nested genrules). See
// #8536.
- return "/bin:/usr/bin:/usr/local/bin";
+ return System.getenv("PATH");
}
String newPath = "";
--
2.18.2
From aa31b751f9b2e5263e4510c8719f29050b8ce0de Mon Sep 17 00:00:00 2001
From: Hongxu Jia <hongxu.jia@windriver.com>
Date: Mon, 14 Dec 2020 16:43:45 +0800
Subject: [PATCH] set python3 interpreter
Since many distrobution choose python3, and drop python2,
If host does not provides `python', python rules failed
...
/usr/bin/env: 'python': No such file or directory
...
set python3 interpreter to shebang
Upstream-Status: Inappropriate [wr-installer specific]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
.../build/lib/bazel/rules/python/BazelPythonSemantics.java | 2 +-
.../build/lib/bazel/rules/python/python_stub_template.txt | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java
index 9be9ef3..dec19e5 100755
--- a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java
@@ -247,7 +247,7 @@ public class BazelPythonSemantics implements PythonSemantics {
PathFragment shExecutable = ShToolchain.getPathOrError(ruleContext);
// TODO(#8685): Remove this special-case handling as part of making the proper shebang a
// property of the Python toolchain configuration.
- String pythonExecutableName = OS.getCurrent() == OS.OPENBSD ? "python3" : "python";
+ String pythonExecutableName = OS.getCurrent() == OS.OPENBSD ? "python3" : "python3";
// NOTE: keep the following line intact to support nix builds
String pythonShebang = "#!/usr/bin/env " + pythonExecutableName;
ruleContext.registerAction(
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt
index 29f043f..c83891a 100755
--- a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt
+++ b/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# This script must retain compatibility with a wide variety of Python versions
# since it is run for every py_binary target. Currently we guarantee support
--
2.18.2
SUMMARY = "Edge TPU demo script"
HOMEPAGE = "https://coral.googlesource.com/edgetpu"
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://debian/copyright;md5=06632c7e5964209e848079cc1ffba7b9"
SRC_URI = "git://coral.googlesource.com/edgetpudemo;protocol=https;branch=master"
SRCREV = "ba9d1c0b9356b23733d0e3860a96da5287a5a1ef"
S = "${WORKDIR}/git"
RDEPENDS_${PN} = "\