Prevent pip from installing some dependencies
We're developing an AWS Lambda function for Alexa skill in Python and using
pip to install the
ask-sdk package to our
pip install -t dist/ ask-sdk
The trouble is with the
-t dist/ because
pip wants to have all the dependencies there, even if they are installed system-wide.
ask-sdk has a dependency on
boto3 which pulls in a whole lot of other packages. However the AWS Lambda runtime environment provides
boto3 and there is no need to package that and its dependencies with our code. I do have
boto3 installed in the system and
import boto3 works, so I thought
pip should be happy, but because of
-t dist/ it always installs it.
Can I somehow install just
ask-sdk and its dependencies that don't exist in the system, e.g.
ask-sdk-core, but not those that are already installed?
One way is to list all the modules and use
pip --no-deps but that means a constantly keeping track of the dependencies manually, we would like to avoid that.
Somehow I would like to tell
pip: if the package is already installed, even if not in
-t dist/ don't put a copy in
Is that possible?
Although you can't tell pip to "install all dependencies except those required by boto3", you can generate the needed
requirements.txt by computing the difference between
pip freeze output (tested with Python 3.6.6):
# get boto3 requirements pip install boto3 -t py_lib.boto3 PYTHONPATH=py_lib.boto3 pip freeze > requirements-boto3.txt # get ask-sdk requirements pip install ask-sdk -t py_lib.ask-sdk PYTHONPATH=py_lib.ask-sdk pip freeze > requirements-ask-sdk.txt # compute their difference grep -v -x -f requirements-boto3.txt requirements-ask-sdk.txt > requirements-final.txt # patch to add one missing dep from boto3 # aws don't have this for some reason grep urllib3 requirements-boto3.txt >> requirements-final.txt
requirements-final.txt contains the following:
ask-sdk==1.5.0 ask-sdk-core==1.5.0 ask-sdk-dynamodb-persistence-adapter==1.5.0 ask-sdk-model==1.6.2 ask-sdk-runtime==1.5.0 certifi==2018.11.29 chardet==3.0.4 idna==2.8 requests==2.21.0 urllib3==1.24.1
To install the final set of dependencies to a folder:
pip install --no-deps -r requirements-final.txt -t py_lib
By skipping the
boto3 dependencies, you can save about 45M of data from your python dependencies. The
ask-sdk dependencies are only about 7.5M (2.1M compressed), allow you to use the build-in lambda code editor if you need to.
A way to blacklist 'provided' dependencies. · Issue #3090 · pypa/pip , pip install --log pip.log -r req.txt -target /mypath/pypkgs/ --exclude for installing plugins for an application that already provides some of the Running conda after pip has the potential to overwrite and potentially break packages installed via pip. Similarly, pip may upgrade or remove a package which a conda-installed package requires. In some cases these breakages are cosmetic, where a few files are present that should have been removed, but in other cases the environment may evolve into an unusable state.
You can try the option
To ignore all dependencies.
To exclude specific, you can put it in requirements file and pass it:
pip install --no-deps -r requirements.txt
pip install --user ignores system-installed dependencies, installs , pip install --user ignores system-installed dependencies, installs older When inside virtual environments or when running as root, keep the default distributions support an alternative install location that is specific to a user. Installing packages using pip and virtual environments¶ This guide discusses how to install packages using pip and a virtual environment manager: either venv for Python 3 or virtualenv for Python 2. These are the lowest-level tools for managing Python packages and are recommended if higher-level tools do not suit your needs.
This will work
$ pip install -t dist --no-deps ask-sdk
After the above command I checked out the dist directory content with tree and it had installed only ask-sdk without its dependencies
dist/ ├── ask_sdk │ ├── __init__.py │ ├── __init__.pyc │ ├── __version__.py │ ├── __version__.pyc │ ├── standard.py │ └── standard.pyc └── ask_sdk-0.1.3.dist-info ├── INSTALLER ├── METADATA ├── RECORD ├── WHEEL └── top_level.txt
pip install, As of v6.1.0, pip installs dependencies before their dependents, i.e. in to declare build dependencies, it may help certain projects install from sdist (that might While this cache attempts to minimize network activity, it does not prevent setup.py référence the dependencies (I use PBR to do it, i have not tested with find_packages, but in both cases I don’t see how the markers can be described in it) symptom is: pip install mypackage does install the conditioned dependency on python 2.7, pipenv install mypackage does not
The Nine Circles of Python Dependency Hell - Knerd, Dependency hell in Python often happens because pip does not have a dependency… If you just run pip install -r requirements.txt , pip will happily ignore any Some dependency management systems, like npm, avoid this problem by When installing from a version control repository, it is strongly recommended to pin the package by specifying a tag or commit, rather than branch. Sometimes your dependencies may themselves have unpinned dependencies. In this case, it can be worth explicitly pinning those too - you can easily pin all dependencies in a project automatically.
pipdeptree · PyPI, pipdeptree is a command line utility for displaying the installed python packages in form of a dependency tree. It works for packages installed globally on a Requirements files are used to force pip to properly resolve dependencies. As it is now, pip doesn’t have true dependency resolution, but instead simply uses the first specification it finds for a project. E.g. if pkg1 requires pkg3>=1.0 and pkg2 requires pkg3>=1.0,<=2.0, and if pkg1 is resolved first,
Managing Python packages the right way, When installing packages, pip will first resolve the dependencies, check if and some of these tools install into a non-user-writeable location. Packaging and distributing projects¶ This section covers the basics of how to configure, package and distribute your own Python projects. It assumes that you are already familiar with the contents of the Installing Packages page. The section does not aim to cover best practices for Python project development as a whole. For example, it does