Python Virtual Environments

Overview

Python's virtual environments how to create and manage separate virtual environments for your Python projects and why is important in utilizing them.

My first "mystery" (it was wiked at that time) when I started to learn Python was virtual environments.

I understood the concepts, but struggled first to understand clearly their role and then to configure them on my PyCharm but once I got it, I loved it.

The goal of this writing would be to share my view  with you on below aspects:

• understand what is a virtual environment and how it can help
• create and activate a Python virtual environment
• customize your virtual environment
• deactivate and remove virtual environment
• understand additional tools for managing virtual environments

Python Virtual Environments

Python virtual environments are just that, pockets(folders) in your computer which segregate different version of software that you can use to develop and test at ease; it creates isolated contexts to keep dependencies required by different projects separate so they don’t mix with other projects or system wide packages. Simple put it, virtual environments is the best way to have different Python projects separated virtually.

When you are a beginner, you tend to have a single system python interpreter (usually in base environment) were you will dump all your packages and it is ok-ish until you learn the hard way, that it's not.

Having only one source (providers in term of packages) place for all your work is fine until one point when you want to start building greater things and you'll be working with many programs having different complex dependencies, and yes in that moment of time learning to jingle with virtual environments will be a tremendous step in the right direction; thus is better to start from beginning to know about virtual environments.

Quoting from Python documentation 28.3.1 bottom Note: "A virtual environment is a Python environment such that the Python interpreter, libraries and scripts installed into it are isolated from those installed in other virtual environments, and (by default) any libraries installed in a “system” Python, i.e., one which is installed as part of your operating system.A virtual environment is a directory tree which contains Python executable files and other files which indicate that it is a virtual environment."

This means that when you activate your project virtual environment, your project virtually becomes independent of the system Python and its modules/packages etc (your project becomes self-contained); your virtual environment has its own pip to handle libraries, its own folder structure (different tools to handle virtual environments), its own Python interpreter with the required version which will be used to interpret your current or future program.

In some cases virtual environments are a make, brake or take deal. Having this in place it's just a simpler way to write code or automated tests agaist different versions of Python, packages or a combination of both.

The importance of virtual environments becomes much apparent when you have various Python projects on the same machine that has dependencies on different versions of same package.

In my case I've discovered (of course after reading blogs and articles and having to play with them) that the best practice, for each project I build is to generate a dedicated environment and never install packages globally ( it’s a common and effective technique to be used in Python development and not only).

Let me give you an example.

Let's say you are working on 2 different projects which requires 2 different  Pandas packages 1.5 and 2.0. This would lead to compatibility issues because Python cannot simultaneously use multiple version of the same package in the same virtual environment.

Another example, would be, you are pushing code to a place where are used certain versions of packages and you have to mirror your development environment to match that certain requirements.

Python virtual environments has 2 main components:

• Python interpreter on which virtual environment runs on
• A folder which keeps all 3rd parties libraries installed in that virtual environment, which means dependencies from one particular program running on some particular versions of libraries are not going to be affected by newer libraries released, or newer/older version of Python. Thus we can create multiple virtual environments (folders) with different Python versions together with different libraries or the same libraries in different Python versions.

To provide a visual demonstration of the above alambicated phrase, please see below:

The above picture its a good representation of what you can have on you system, when you create a Python virtual environment which is a tree folder containing a specific python version together with the 3rd parties libraries. As far as I know there is no limitation on the number of virtual environment which you can make on your machine (in the end are folders with files or other folders).

Enough theory on this subject, lets deep dive into possible tools to aid/use virtual environments.

There are a couple of options which can be used to create this so called virtual environment and we will take a look at the most important one:

• venv - it’s a subset (lighter version) of virtualenv and it has been shipped since Python 3.3 (if not mistaken), this is covered in PEP 405 – Python Virtual Environments  " Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories, but shares the standard library with the base installed Python".

• virtualenv - 3rd party tool that has been developed and maintained separately from Python itself. It offers more advanced features and customization options for managing virtual environments and more often than not, this is desired for complex or specialized use cases.

• pipenv (pipenv) -  from Python 3.7 and above, lets not reinvent the weel, so here it goes their presentation "is a production-ready tool that aims to bring the best of all packaging worlds to the Python world. It harnesses Pipfile, pip, and virtualenv into one single command. It features very pretty terminal colors. Pipenv is primarily meant to provide users and developers of applications with an easy method to arrive at a consistent working project environment."

• conda - It is an awesome package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, Fortran, and more. Conda is commonly associated with Anaconda, which is a popular distribution of Python for data science and scientific computing (its huge in size, but its good).

• poetry -> Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution (awesome tool).

• docker (docker) - takes a broader approach by providing a platform for creating isolated containerized environments that can run entire applications and services, not limited to Python, can be used as a form of virtual environment for running applications and services (separate your applications from your infrastructure), but it's different from the above methods of just "simply" separating python environments. It's more focused on creating isolated infrastructure environments for running applications vs Poetry, Conda, and Pipenv etc. ( these tools are specifically designed for managing dependencies within Python projects. They offer mechanisms for specifying, installing, and locking dependencies).