How to install dependencies from requirements.txt in a Yocto recipe for a local Python project

yocto mega manual
yocto python
yocto add recipe
image_fstypes
yocto packages
yocto overview
yocto packageconfig
bitbake server
What I should have:

I want my Yocto Project to build a package for my Python project with all dependencies inside. The project has to run out of box on the resulting read-only sdcard image. It simply should install all requirements in the required version to the package.

What I tried without luck:
Calling pip in do_install():

"pip/pip3 is not found", even it's in RDEPENDS. Anyway, I really prefer this way.

With inherit pypi:

When trying with inherit pypi, it tries to get also my local sources (my pyton project) from pypi. And I have always to copy the requirements to the recipe. This is not my preferred way.

Calling pip in pkg_postinst():

It tries to install the modules on first start and fails, because the system has no internet connection and it's a read-only system. It must run out of the box without installation on first boot time. Does its stuff to late.

Where I'll get around:

There should be no need to change anything in the recipes when something changes in requirements.txt.

Background information

I'm working with Yocto Rocko in a Linux environment. In the Hostsystem, there is no pip installed. I want to run this one installed from RDEPENDS in the target system.

Building the Package (only this recipe) with:

bitbake myproject

Building the whole sdcard image:

bitbake myProject-image-base
The recipe:

myproject.bb (relevant lines):

RDEPENDS_${PN} = "python3 python3-pip"

APP_SOURCES_DIR := "${@os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../../../../app-sources')}"

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI = " \
    file://${APP_SOURCES_DIR}/myProject \
    ...
"

inherit allarch  # tried also with pypi and setuptools3 for the pypi way.

do_install() {  # Line 116
    install -d -m 0755 ${D}/myProject
    cp -R --no-dereference --preserve=mode,links -v ${APP_SOURCES_DIR}/myProject/* ${D}/myProject/
    pip3 install -r ${APP_SOURCES_DIR}/myProject/requirements.txt
    # Tried also python ${APP_SOURCES_DIR}/myProject/setup.py install
}

# Tried also this, but it's no option because the data MUST be included in the Package:
# pkg_postinst_${PN}() {
#     #!/bin/sh -e
#     pip3 install -r /myProject/requirements.txt
# }

FILES_${PN} = "/myProject/*"
Resulting Errors:

Expected to install the listed modules from requirements.txt into the myProject package, so that the python app will run directly on the resulting readonly sdcard image.

With pip, I get:

| /*/tmp/work/*/myProject/0.1.0-r0/temp/run.do_install: 116: pip3: not found
| WARNING: exit code 127 from a shell command.
| ERROR: Function failed: do_install ...

When using pypi:

404 Not Found
ERROR: myProject-0.1.0-r0 do_fetch: Fetcher failure for URL: 'https://files.pythonhosted.org/packages/source/m/myproject/myproject-0.1.0.tar.gz'. Unable to fetch URL from any source.

=> But it should not fetch myProject, since it is already local and nowhere remote.

Any ideas? What would be the best way to reach to a ready to use sdcard image without the need to change recipes when requirements.txt changes?

You should use RDEPENDS_${PN} to take care of your dependencies for your app in the recipe.

For example, assuming your python app needs aws-iot-device-sdk-python module, you should add it to RDEPENDS in the recipe. In your case, it would be like this:

RDEPENDS_${PN} = "python3 \
                  python3-pip \
                  python3-aws-iot-device-sdk-python \
                 "

Here's the link showing the Python modules supported by OpenEmbedded Layer. https://layers.openembedded.org/layerindex/branch/master/layer/meta-python/

If the modules you need are not there, you will likely need to create recipes for the modules.

[yocto] Installing Python app from source, In a Python project with a setup.py file, you can actually do "pip install ." Of course you still need internet access for dependencies, which I have during the build. In this case your app is local so need to have those files first > built That part which is confusing to me in that recipe is that the SRC_URI  If you want to install the dependencies in a virtual environment, create and activate that environment first, then use the Install from requirements.txt command. For more information on creating a virtual environment, see Use virtual environments.

Yocto Project Reference Manual, Local. 13.2. Recipes. 13.2.1. Required; 13.2.2. Dependencies; 13.2.3. sudo zypper install python gcc gcc-c++ git chrpath make wget python-xml \ diffstat and run it on a machine that does not meet the minimal Git, tar, and Python requirements. build-id.txt: Human-readable information about the build configuration and  This version of the Yocto Project Reference Manual is for the 2.4.4 release of the Yocto Project. To be sure you have the latest version of the manual for this release, go to the Yocto Project documentation page and select the manual from that site.

You try installing pip?

Debian

apt-get install python-pip
apt-get install python3-pip

Centos

yum install python-pip

Toaster User Manual, Matching Branch Requirement for Git Fetching¶ a headless system and using a default local.conf file, you file dependencies are satisfied (e.g. package contains a The build history feature now writes build-id.txt Consequently, the separate python-argparse recipe is  This manual provides reference information for the current release of the Yocto Project. The Yocto Project is an open-source collaboration project focused on embedded Linux developers. Amongst other things, the Yocto Project uses the OpenEmbedded build system, which is based on the Poky project, to construct complete Linux images.

How to work with Python applications and modules in Yocto Project , Additional Information About the Local Yocto Project Release; 3.7.3. Building a Browse images, recipes, and machines provided by those layers. Import your own Toaster requires extra Python dependencies in order to run. install -r ./​poky/bitbake/toaster-requirements.txt $ pip install mysql $ pip install MySQL-​python  Welcome to the Yocto Project! The Yocto Project is an open-source collaboration project focused on embedded Linux developers. Among other things, the Yocto Project uses a build system based on the OpenEmbedded (OE) project, which uses the BitBake tool, to construct complete Linux images.

ypdd-2016.04-San_Diego - Google Slides, Additional Information About the Local Yocto Project Release; 3.10.3. Building a Specific See dependency relationships between recipes, packages, and tasks. See performance Toaster requires extra Python dependencies in order to run. A Toaster pip3 install --user -r bitbake/toaster-requirements.txt. The previous  This version of the Toaster User Manual is for the 2.5 release of the Yocto Project. To be sure you have the latest version of the manual for this release, go to the Yocto Project documentation page and select the manual from that site.

1.4. Obtaining the Yocto Project, However, you cannot use Toaster to customize image recipes, which still must either be Fundamentally, a local instance of Toaster is suited for a single user developing on a Toaster requires extra Python dependencies in order to run. poky/bitbake/toaster-requirements.txt $ pip install mysql $ pip install MySQL-​python  Managing Application Dependencies¶. The package installation tutorial covered the basics of getting set up to install and update Python packages.. However, running these commands interactively can get tedious even for your own personal projects, and things get even more difficult when trying to set up development environments automatically for projects with multiple contributors.

Comments
  • I'm not familiar with Yocto, but the syntax caught my eye, it's very similar to Gentoo's EAPI. In Gentoo's ebuilds, RDEPENDS declares runtime dependencies, so packages listed there won't be available at build time. Try adding python3 & python3-pip to DEPENDS list instead.
  • As for the fetcher failure, pip install -r requirements.txt will try to download all packages listed in the file and install them. If you add myproject to the list of packages, it will make no exception. You can instead instruct pip to install from the source dir directly: pip install path/to/dir, where path/to/dir is the one containing the project's setup.py script.
  • Although I also might be completely wrong with my assumptions :-)
  • @hoefling: no, myProject doesn't make part of the listed packages in requirements.txt. But requirements.txt makes part of myProject. All the packages in the list should really be downloaded.
  • @hoefling: I also tried declaring pip in DEPENDS, but it has the same effect. I assume it searches pip in the host system that way I used it? Am I wrong?
  • I know how to install pip, but I like to avoid installing pip in the host system but using this one that is packed on build time, if this is possible.