388

I've tried reading through questions about sibling imports and even the package documentation, but I've yet to find an answer.

With the following structure:

├── LICENSE.md
├── README.md
├── api
│   ├── __init__.py
│   ├── api.py
│   └── api_key.py
├── examples
│   ├── __init__.py
│   ├── example_one.py
│   └── example_two.py
└── tests
│   ├── __init__.py
│   └── test_one.py

How can the scripts in the examples and tests directories import from the api module and be run from the commandline?

Also, I'd like to avoid the ugly sys.path.insert hack for every file. Surely this can be done in Python, right?

4
  • 15
    I recommend skipping past all the sys.path hacks and reading the only actual solution that's been posted thus far (after 7 years!).
    – Aran-Fey
    Commented Nov 2, 2018 at 10:32
  • 2
    By the way, there's still room for another good solution: Separating executable code from library code; most of the time a script inside a package shouldn't be executable to begin with.
    – Aran-Fey
    Commented Nov 2, 2018 at 10:46
  • This is so helpful, both the question and the answers. I am just curious, how come "Accepted Answer" is not the same as the one awarded the bounty in this case?
    – Indominus
    Commented Dec 27, 2018 at 21:41
  • 1
    @Aran-Fey That's an underrated reminder in these relative import error Q&As. I've been looking for a hack this whole time, but deep down I knew there was a simple way to design my way out of the problem. Not to say that it's the solution for everyone here reading, but it's a good reminder as it could be for many.
    – colorlace
    Commented Apr 19, 2019 at 22:10

12 Answers 12

432
+300

Tired of sys.path hacks?

There are plenty of sys.path.append -hacks available, but I found an alternative way of solving the problem in hand.

Summary

  • Wrap the code into one folder (e.g. packaged_stuff)
  • Create pyproject.toml file to describe your package (see minimal pyproject.toml below)
  • Pip install the package in editable state with pip install -e <myproject_folder>
  • Import using from packaged_stuff.modulename import function_name

Setup

The starting point is the file structure you have provided, wrapped in a folder called myproject.

.
└── myproject
    ├── api
    │   ├── api_key.py
    │   ├── api.py
    │   └── __init__.py
    ├── examples
    │   ├── example_one.py
    │   ├── example_two.py
    │   └── __init__.py
    ├── LICENCE.md
    ├── README.md
    └── tests
        ├── __init__.py
        └── test_one.py

I will call the . the root folder, and in my example case it is located at C:\tmp\test_imports\.

api.py

As a test case, let's use the following ./api/api.py

def function_from_api():
    return 'I am the return value from api.api!'

test_one.py

from api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

Try to run test_one:

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\myproject\tests\test_one.py", line 1, in <module>
    from api.api import function_from_api
ModuleNotFoundError: No module named 'api'

Also trying relative imports wont work:

Using from ..api.api import function_from_api would result into

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\tests\test_one.py", line 1, in <module>
    from ..api.api import function_from_api
ValueError: attempted relative import beyond top-level package

Steps

1) Make a pyproject.toml file to the root level directory

(previously people used a setup.py file)

The contents for a minimal pyproject.toml would be*

[project]
name = "myproject"
version = "0.1.0"
description = "My small project"

[build-system]
build-backend = "flit_core.buildapi"
requires = ["flit_core >=3.2,<4"]

2) Use a virtual environment

If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)

  • Create virtual env
    • python -m venv venv
  • Activate virtual env
    • source ./venv/bin/activate (Linux, macOS) or ./venv/Scripts/activate (Win)

To learn more about this, just Google out "python virtual env tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.

Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis

PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>

and your folder tree should look like this**

.
├── myproject
│   ├── api
│   │   ├── api_key.py
│   │   ├── api.py
│   │   └── __init__.py
│   ├── examples
│   │   ├── example_one.py
│   │   ├── example_two.py
│   │   └── __init__.py
│   ├── LICENCE.md
│   ├── README.md
│   └── tests
│       ├── __init__.py
│       └── test_one.py
├── pyproject.toml
└── venv
    ├── Include
    ├── Lib
    ├── pyvenv.cfg
    └── Scripts [87 entries exceeds filelimit, not opening dir]

3) pip install your project in editable state

Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package. Using pyproject.toml and -e flag requires pip >= 21.3

In the root directory, run

pip install -e . (note the dot, it stands for "current directory")

You can also see that it is installed by using pip freeze

Obtaining file:///home/user/projects/myproject
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: myproj
  Building editable for myproj (pyproject.toml) ... done
  Created wheel for myproj: filename=myproj-0.1.0-py2.py3-none-any.whl size=903 sha256=f19858b080d4e770c2a172b9a73afcad5f33f4c43c86e8eb9bdacbe50a627064
  Stored in directory: /tmp/pip-ephem-wheel-cache-qohzx1u0/wheels/55/5f/e4/507fdeb40cdef333e3e0a8c50c740a430b8ce84cbe17ae5875
Successfully built myproject
Installing collected packages: myproject
Successfully installed myproject-0.1.0
(venv) PS C:\tmp\test_imports> pip freeze
myproject==0.1.0

4) Add myproject. into your imports

Note that you will have to add myproject. only into imports that would not work otherwise. Imports that worked without the pyproject.toml & pip install will work still work fine. See an example below.


Test the solution

Now, let's test the solution using api.py defined above, and test_one.py defined below.

test_one.py

from myproject.api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

running the test

(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
I am the return value from api.api!

* here using flit as build backend. Other alternatives exist.

** In reality, you could put your virtual environment anywhere on your hard disk.

25
  • 27
    Thanks for the detailed post. Here is my problem. If I do everything you said and I do a pip freeze, I get a line -e git+https://[email protected]/folder/myproject.git@f65466656XXXXX#egg=myproject Any Idea on how to resolve?
    – Si Mon
    Commented Dec 11, 2018 at 16:42
  • 8
    Why doesn't the relative import solution work? I believe you, but I'm trying to understand Python's convoluted system. Commented Feb 1, 2019 at 2:56
  • 35
    Has anyone has problems with regards to a ModuleNotFoundError? I've installed 'myproject' into a virtualenv following these steps, and when I enter an interpreted session and run import myproject I get ModuleNotFoundError: No module named 'myproject'? pip list installed | grep myproject shows that it is there, the directory is correct, and both the verison of pip and python are verified to be correct.
    – ThoseKind
    Commented Aug 28, 2019 at 14:40
  • 6
    Spent about 2 hours trying to figure out how to make relative imports work, and this answer was the one that finally actually did something sensible. 👍👍
    – Graham Lea
    Commented Nov 28, 2019 at 4:29
  • 42
    What I think is sickening is that I have to come to stackoverflow to find a real answer on how to do relative imports correctly. The documentation around this stuff really needs to be improved.
    – teuber789
    Commented Feb 20, 2020 at 17:21
119

Seven years after

Since I wrote the answer below, modifying sys.path is still a quick-and-dirty trick that works well for private scripts, but there has been several improvements

  • Installing the package (in a virtualenv or not) will give you what you want, though I would suggest using pip to do it rather than using setuptools directly (and using setup.cfg to store the metadata)
  • Using the -m flag and running as a package works too (but will turn out a bit awkward if you want to convert your working directory into an installable package).
  • For the tests, specifically, pytest is able to find the api package in this situation and takes care of the sys.path hacks for you

So it really depends on what you want to do. In your case, though, since it seems that your goal is to make a proper package at some point, installing through pip -e is probably your best bet, even if it is not perfect yet.

Old answer

As already stated elsewhere, the awful truth is that you have to do ugly hacks to allow imports from siblings modules or parents package from a __main__ module. The issue is detailed in PEP 366. PEP 3122 attempted to handle imports in a more rational way but Guido has rejected it one the account of

The only use case seems to be running scripts that happen to be living inside a module's directory, which I've always seen as an antipattern.

(here)

Though, I use this pattern on a regular basis with

# Ugly hack to allow absolute import from the root folder
# whatever its name is. Please forgive the heresy.
if __name__ == "__main__" and __package__ is None:
    from sys import path
    from os.path import dirname as dir

    path.append(dir(path[0]))
    __package__ = "examples"

import api

Here path[0] is your running script's parent folder and dir(path[0]) your top level folder.

I have still not been able to use relative imports with this, though, but it does allow absolute imports from the top level (in your example api's parent folder).

7
  • 4
    you don't have to if you run from a project directory using -m form or if you install the package (pip and virtualenv make it easy)
    – jfs
    Commented May 8, 2014 at 13:15
  • 4
    How does pytest find the api package for you? Amusingly, I found this thread because I'm running into this problem specifically with pytest and sibling package importing. Commented Jan 18, 2019 at 0:30
  • 1
    I have two questions, please. 1. Your pattern seems to work without __package__ = "examples" for me. Why do you use it? 2. In what situation is __name__ == "__main__" but __package__ is not None? Commented Nov 6, 2019 at 16:57
  • @actual_panda Setting __packages__ helps if you want absolute path such as examples.api to work iirc (but it has been a long time since I last did that) and checking that package is not None was mostly a failsafe for weird situations and futureproofing.
    – Evpok
    Commented Jan 30, 2020 at 10:48
  • 1
    Gosh, if only other languages would make the same process as easy as it is in Python. I see why everyone loves this language. Btw, documentation is also excellent. I love extracting return types from unstructured text, it's a nice change from Javadoc and phpdoc. ffs....
    – matt
    Commented Jul 8, 2020 at 16:04
51

Here is another alternative that I insert at top of the Python files in tests folder:

# Path hack.
import sys, os
sys.path.insert(0, os.path.abspath('..'))
6
  • 1
    +1 really simple and it worked perfectly. You need to add the parent class to the import (ex api.api, examples.example_two) but I prefer it that way. Commented Jun 1, 2012 at 21:35
  • 15
    I think it's worth mentioning to newbies (like myself) that .. here is relative to the directory you're executing from---not the directory containing that test/example file. I'm executing from the project directory, and I needed ./ instead. Hope this helps someone else.
    – user7851115
    Commented May 23, 2018 at 18:01
  • @JoshDetwiler, yes absoluely. I was not aware of that. Thanks.
    – doak
    Commented Jan 31, 2019 at 11:12
  • 6
    This is a poor answer. Hacking the path is not good practice; it's scandalous how much it's used in the python world. One of the main points of this question was to see how imports could be done while avoiding this kind of hack.
    – teuber789
    Commented Feb 20, 2020 at 17:20
  • 2
    sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) @JoshuaDetwiler
    – vldbnc
    Commented May 15, 2020 at 18:27
46

You don't need and shouldn't hack sys.path unless it is necessary and in this case it is not. Use:

import api.api_key # in tests, examples

Run from the project directory: python -m tests.test_one.

You should probably move tests (if they are api's unittests) inside api and run python -m api.test to run all tests (assuming there is __main__.py) or python -m api.test.test_one to run test_one instead.

You could also remove __init__.py from examples (it is not a Python package) and run the examples in a virtualenv where api is installed e.g., pip install -e . in a virtualenv would install inplace api package if you have proper setup.py.

5
  • @Alex the answer does not assume that tests are API tests except for the paragraph where it says explicitly "if they are api's unittests".
    – jfs
    Commented Nov 24, 2016 at 20:30
  • unfortunately then you are stuck with running from root dir and PyCharm still does not find the file for its nice functions
    – mhstnsc
    Commented Nov 3, 2017 at 17:28
  • 1
    @mhstnsc: it is not correct. You should be able to run python -m api.test.test_one from anywhere when the virtualenv is activated. If you can't configure PyCharm to run your tests, try to ask a new Stack Overflow question (if you can't find an existing question on this topic).
    – jfs
    Commented Nov 3, 2017 at 18:05
  • @jfs I missed the virtual env path but i don't want to use anything more than shebang line to run this stuff from any directory ever. Its not about running with PyCharm. Devs with PyCharm would know also they have completion and jump through functions which i could not make it work with any solution.
    – mhstnsc
    Commented Nov 3, 2017 at 20:48
  • 1
    @mhstnsc an appropriate shebang is enough in many cases (point it to the virtualenv python binary. Any decent Python IDE should support a virtualenv.
    – jfs
    Commented Sep 25, 2018 at 19:43
13

For readers in 2023: If you're not confident with pip install -e :

TL;DR: A script(usually an entry point) can only import anything the same or below its level.

Consider this hierarchy, as recommended by an answer from Relative imports in Python 3:

MyProject
├── src
│   ├── bot
│   │   ├── __init__.py
│   │   ├── main.py
│   │   └── sib1.py
│   └── mod
│       ├── __init__.py
│       └── module1.py
└── main.py

To run our program from the starting point with the simple command python main.py, we use absolute import (no leading dot(s)) in main.py here:

from src.bot import main


if __name__ == '__main__':
    main.magic_tricks()

The content of bot/main.py, which takes advantage of explicit relative imports to show what we're importing, looks like this:

from .sib1 import my_drink                # Both are explicit-relative-imports.
from ..mod.module1 import relative_magic

def magic_tricks():
    # Using sub-magic
    relative_magic(in=["newbie", "pain"], advice="cheer_up")
    
    my_drink()
    # Do your work
    ...

These are the reasonings:

  • We don't want to give "OK, so this is a module" a funk when we want to run our Python program, sincerely.
    • So we use absolute import for the entry point main.py, this way we can run our program by simply python main.py.
    • Behind the scene, Python will use sys.path to resolve packages for us, but this also means that the package we want to import can probably be superseded by any other package of the same name due to the ordering of paths in sys.path e.g. try import test.
  • To avoid those conflicts, we use explicit relative import.
    • The from ..mod syntax makes it very clear about "we're importing our own local package".
    • But the drawback is that you need to think about "OK, so this is a module" again when you want to run the module as a script.
  • Finally, the from ..mod part means that it will go up one level to MyProject/src.

Conclusion

  1. Put your main.py script next to the root of all your packages MyProject/src, and use absolute import in python main.py to import anything. No one will create a package named src.
  2. Those explicit relative imports will just work.
  3. To run a module, use python -m ....

Appendix: More about running any file under src/ as a script?

Then you should use the syntax python -m and take a look at my other post: ModuleNotFoundError: No module named 'sib1'

1
  • This actually worked for me. I think without this "src" directory/package, the subpackages didn't have common root package.
    – stoper
    Commented May 8, 2023 at 16:58
12

I don't yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling/relative import hack. Until that day, this is my solution. For examples or tests to import stuff from ..\api, it would look like:

import sys.path
import os.path
# Import from sibling directory ..\api
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/..")
import api.api
import api.api_key
1
  • This would still give you the api parent directory and you wouldn't need the "/.." concatenation sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(file)))) Commented Jul 23, 2014 at 3:46
6

For siblings package imports, you can use either the insert or the append method of the [sys.path][2] module:

if __name__ == '__main__' and if __package__ is None:
    import sys
    from os import path
    sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
    import api

This will work if you are launching your scripts as follows:

python examples/example_one.py
python tests/test_one.py

On the other hand, you can also use the relative import:

if __name__ == '__main__' and if __package__ is not None:
    import ..api.api

In this case you will have to launch your script with the '-m' argument (note that, in this case, you must not give the '.py' extension):

python -m packageName.examples.example_one
python -m packageName.tests.test_one

Of course, you can mix the two approaches, so that your script will work no matter how it is called:

if __name__ == '__main__':
    if __package__ is None:
        import sys
        from os import path
        sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
        import api
    else:
        import ..api.api
2
  • I was using the Click framework which doesn't have the __file__ global so I had to use the following: sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(sys.argv[0])))) But it works in any directory now
    – GammaGames
    Commented Mar 8, 2019 at 15:39
  • 1
    There are no good answers to this. And shows how the makers of python must live in some fantasy bubble where a relative import is just almost impossible.
    – C.J.
    Commented Apr 21, 2021 at 18:03
2

You need to look to see how the import statements are written in the related code. If examples/example_one.py uses the following import statement:

import api.api

...then it expects the root directory of the project to be in the system path.

The easiest way to support this without any hacks (as you put it) would be to run the examples from the top level directory, like this:

PYTHONPATH=$PYTHONPATH:. python examples/example_one.py 
2
  • With Python 2.7.1 I get the following: $ python examples/example.py Traceback (most recent call last): File "examples/example.py", line 3, in <module> from api.api import API ImportError: No module named api.api. I also get the same with import api.api.
    – zachwill
    Commented Jun 12, 2011 at 18:51
  • Updated my answer...you do have to add the current directory to the import path, no way around that.
    – AJ.
    Commented Jun 12, 2011 at 18:58
1

Just in case someone using Pydev on Eclipse end up here: you can add the sibling's parent path (and thus the calling module's parent) as an external library folder using Project->Properties and setting External Libraries under the left menu Pydev-PYTHONPATH. Then you can import from your sibling, e. g. from sibling import some_class.

1

I wanted to comment on the solution provided by np8 but I don't have enough reputation so I'll just mention that you can create a setup.py file exactly as they suggested, and then do pipenv install --dev -e . from the project root directory to turn it into an editable dependency. Then your absolute imports will work e.g. from api.api import foo and you don't have to mess around with system-wide installations.

Documentation

1

If you're using pytest then the pytest docs describe a method of how to reference source packages from a separate test package.

The suggested project directory structure is:

setup.py
src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    __init__.py
    foo/
        __init__.py
        test_view.py
    bar/
        __init__.py
        test_view.py

Contents of the setup.py file:

from setuptools import setup, find_packages

setup(name="PACKAGENAME", packages=find_packages())

Install the packages in editable mode:

pip install -e .

The pytest article references this blog post by Ionel Cristian Mărieș.

1

I made a sample project to demonstrate how I handled this, which is indeed another sys.path hack as indicated above. Python Sibling Import Example, which relies on:

if __name__ == '__main__': import os import sys sys.path.append(os.getcwd())

This seems to be pretty effective so long as your working directory remains at the root of the Python project.

1
  • 1
    This only works if you are running from the script's parent directory
    – Evpok
    Commented Dec 17, 2019 at 20:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.