How to customize a requirements.txt for multiple environments?

Question

I have two branches, Development and Production. Each has dependencies, some of which are different. Development points to dependencies that are themselves in development. Likewise for Production. I need to deploy to Heroku which expects each branch's dependencies in a single file called 'requirements.txt'.

What is the best way to organize?

What I've thought of:

• Maintain separate requirements files, one in each branch (must survive frequent merges!)
• Tell Heroku which requirements file I want to use (environment variable?)
• Write deploy scripts (create temp branch, modify requirements file, commit, deploy, delete temp branch)

Answer 1

You can cascade your requirements files and use the "-r" flag to tell pip to include the contents of one file inside another. You can break out your requirements into a modular folder hierarchy like this:

`-- django_project_root
|-- requirements
|   |-- common.txt
|   |-- dev.txt
|   `-- prod.txt
`-- requirements.txt

The files' contents would look like this:

common.txt:

# Contains requirements common to all environments
req1==1.0
req2==1.0
req3==1.0
...

dev.txt:

# Specifies only dev-specific requirements
# But imports the common ones too
-r common.txt
dev_req==1.0
...

prod.txt:

# Same for prod...
-r common.txt
prod_req==1.0
...

Outside of Heroku, you can now setup environments like this:

pip install -r requirements/dev.txt

or

pip install -r requirements/prod.txt

Since Heroku looks specifically for "requirements.txt" at the project root, it should just mirror prod, like this:

requirements.txt:

# Mirrors prod
-r requirements/prod.txt

Answer 2

A viable option today which didn't exist when the original question and answer was posted is to use pipenv instead of pip to manage dependencies.

With pipenv, manually managing two separate requirement files like with pip is no longer necessary, and instead pipenv manages the development and production packages itself via interactions on the command line.

To install a package for use in both production and development:

pipenv install <package>

To install a package for the development environment only:

pipenv install <package> --dev

Via those commands, pipenv stores and manages the environment configuration in two files (Pipfile and Pipfile.lock). Heroku's current Python buildpack natively supports pipenv and will configure itself from Pipfile.lock if it exists instead of requirements.txt.

See the pipenv link for full documentation of the tool.

Answer 3

If your requirement is to be able to switch between environments on the same machine, it may be necessary to create different virtualenv folders for each environment you need to switch to.

python3 -m venv venv_dev
source venv_dev/bin/activate
pip install -r pip/common.txt
pip install -r pip/dev.txt
exit
python3 -m venv venv_prod
source venv_prod/bin/activate
pip install -r pip/common.txt
exit
source venv_dev/bin/activate
# now we are in dev environment so your code editor and build systems will work.

# let's install a new dev package:
# pip install awesome
# pip freeze -r pip/temp.txt
# find that package, put it into pip/dev.txt
# rm pip/temp.txt

# pretty cumbersome, but it works. 

Answer 4

I ended up having requirements-all.txt, which I'll update with every install, and save it's diff to respective file.

E.g. I add pytest.

• pip freeze > requirements-all.txt
• git diff HEAD requirements-all.txt > temp-diff.txt saves what was installed for pytest to temp-diff.txt
• (manually) copy all +<lines> from temp-diff.txt to requirements-dev.txt
• commit both requirements-all.txt and requirements-dev.txt

Hope there is a better way