Question:
Increasingly, there are questions about "broken" / incompatible / stopped working modules in Python and the errors that go with it:
-
ImportError: DLL load failed: ...
-
ImportError: Could not import ...
-
UnsatisfiableError: The following specifications were found to be in conflict: ...
and many others…
The problems described above most often arise due to:
- installing a new module
- Python updates
- module/module updates
- updates/changes in VirtualEnv (Python virtual environment)
- misuse of
conda
– for example, using the default virtual environment created bybase
(which is used byconda
to manage other virtual environments), instead of an explicitly created new virtual environment
Question:
how to fix these problems and how to avoid them in the future?
Answer:
After a long "walking on the rake" (both under UNIX * and under Windows), I settled on the following approach, which has never let me down so far.
The basic idea is to install the Anaconda package and create independent virtual environments using the conda
package manager.
Anaconda checks module version compatibility (including dependencies). This minimizes the chance of breaking Python by simply installing or updating some module(s).
The algorithm for installing Anaconda and creating a VirtualEnv
(an independent Python virtual environment):
-
Update the
conda
package manager ( NOTE: to avoid problems – always runconda
fromAnaconda Prompt
):conda update conda
-
Never "touch" the Python installed by default in the OS or installed by other software (for example, when installing Oracle Database, a separate Python is installed that will be used by Oracle). By
"не трогайте" Python
I mean making any changes that affect Python or its modules:
- installing new modules
- python update
- module update
-
Do not install modules in the
base
virtual environment created by default. Install modules only in those virtual environments that you have explicitly created (see next point –5
). Thebase
virtual environment is the technical environment created byconda
to manage the rest of the virtual environments. If you don't want to break all virtual environments at once, don't touchbase
. -
For each more or less independent Python project, create an independent virtual environment (VirtualEnv). You can additionally create one shared environment for common purposes. In this example, I will create a general environment called
ml
(Machine Learning) forPython 3.7
and a core set of modules for working on machine learning tasks (with Nvidia GPU support):conda create --name ml python=3.7 anaconda keras-gpu
-
In order to run
Python
/Jupyter
/iPython
/ etc. From the created VirtualEnv, you can use one of the following options:
-
run
Anaconda Prompt
–> activate the necessary Wirth in it. environment (conda activate <env_name>
) –> startipython
/Jupyter-Notebook
-
use cmd/shell script to run
ipython
from desired virt. environment:@echo off set conda_dir=%USERPROFILE%\Anaconda3 set env_name=%1 if "%env_name%"=="" set env_name=ml set env_dir=%conda_dir%\envs\%env_name% rem cd %env_dir% call %conda_dir%\Scripts\activate.bat %env_dir% %env_dir%\Scripts\ipython.exe Пример вызова: `c:\bin\ipy_env.cmd ml37`
-
use cmd/shell script to launch
Jupyter-Notebook
from desired virt. environment:@echo off set env_name=%1 if "%env_name%"=="" set env_name=ml set env_dir=%USERPROFILE%\Anaconda3\envs\%env_name% rem cd %env_dir% call %USERPROFILE%\Anaconda3\Scripts\activate.bat %env_dir% start cmd.exe /k "%USERPROFILE%\Anaconda3\envs\%env_name%\Scripts\jupyter-notebook.exe" Пример вызова: `C:\bin\jupyter_env.cmd torch`
-
for a project in PyCharm, you can specify an existing
Conda Environment
asProject Interpreter
- To install a new module always try to do it in the following sequence:
-
always try to find the required module in the default Anaconda repository first
conda search <module_name>
-
if the module is found, install it in our
VirtualEnv
(ml
in our example):conda install -n ml <module_name>
-
if the module is not found, then we try to find this module in the
conda-forge
repository (A community-led collection)conda search -c conda-forge <module_name>
-
if the module is found, install it in our
VirtualEnv
(ml
in our example):conda install -c conda-forge -n ml <module_name>
-
only if the required module is not found in either the original Anaconda repository or
conda-forge
– usepip install
:conda activate ml pip install <module_name>
-
to update a module use the
conda
package manager:conda update -n ml <module_name>
Useful links: