1:mod:`venv` --- Creation of virtual environments 2================================================ 3 4.. module:: venv 5 :synopsis: Creation of virtual environments. 6 7.. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk> 8.. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk> 9 10.. versionadded:: 3.3 11 12**Source code:** :source:`Lib/venv/` 13 14.. index:: pair: Environments; virtual 15 16-------------- 17 18The :mod:`venv` module provides support for creating lightweight "virtual 19environments" with their own site directories, optionally isolated from system 20site directories. Each virtual environment has its own Python binary (which 21matches the version of the binary that was used to create this environment) and 22can have its own independent set of installed Python packages in its site 23directories. 24 25See :pep:`405` for more information about Python virtual environments. 26 27.. seealso:: 28 29 `Python Packaging User Guide: Creating and using virtual environments 30 <https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment>`__ 31 32 33Creating virtual environments 34----------------------------- 35 36.. include:: /using/venv-create.inc 37 38 39.. _venv-def: 40 41.. note:: A virtual environment is a Python environment such that the Python 42 interpreter, libraries and scripts installed into it are isolated from those 43 installed in other virtual environments, and (by default) any libraries 44 installed in a "system" Python, i.e., one which is installed as part of your 45 operating system. 46 47 A virtual environment is a directory tree which contains Python executable 48 files and other files which indicate that it is a virtual environment. 49 50 Common installation tools such as setuptools_ and pip_ work as 51 expected with virtual environments. In other words, when a virtual 52 environment is active, they install Python packages into the virtual 53 environment without needing to be told to do so explicitly. 54 55 When a virtual environment is active (i.e., the virtual environment's Python 56 interpreter is running), the attributes :attr:`sys.prefix` and 57 :attr:`sys.exec_prefix` point to the base directory of the virtual 58 environment, whereas :attr:`sys.base_prefix` and 59 :attr:`sys.base_exec_prefix` point to the non-virtual environment Python 60 installation which was used to create the virtual environment. If a virtual 61 environment is not active, then :attr:`sys.prefix` is the same as 62 :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as 63 :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment 64 Python installation). 65 66 When a virtual environment is active, any options that change the 67 installation path will be ignored from all :mod:`distutils` configuration 68 files to prevent projects being inadvertently installed outside of the 69 virtual environment. 70 71 When working in a command shell, users can make a virtual environment active 72 by running an ``activate`` script in the virtual environment's executables 73 directory (the precise filename and command to use the file is 74 shell-dependent), which prepends the virtual environment's directory for 75 executables to the ``PATH`` environment variable for the running shell. There 76 should be no need in other circumstances to activate a virtual 77 environment; scripts installed into virtual environments have a "shebang" 78 line which points to the virtual environment's Python interpreter. This means 79 that the script will run with that interpreter regardless of the value of 80 ``PATH``. On Windows, "shebang" line processing is supported if you have the 81 Python Launcher for Windows installed (this was added to Python in 3.3 - see 82 :pep:`397` for more details). Thus, double-clicking an installed script in a 83 Windows Explorer window should run the script with the correct interpreter 84 without there needing to be any reference to its virtual environment in 85 ``PATH``. 86 87 88.. _venv-api: 89 90API 91--- 92 93.. highlight:: python 94 95The high-level method described above makes use of a simple API which provides 96mechanisms for third-party virtual environment creators to customize environment 97creation according to their needs, the :class:`EnvBuilder` class. 98 99.. class:: EnvBuilder(system_site_packages=False, clear=False, \ 100 symlinks=False, upgrade=False, with_pip=False, \ 101 prompt=None, upgrade_deps=False) 102 103 The :class:`EnvBuilder` class accepts the following keyword arguments on 104 instantiation: 105 106 * ``system_site_packages`` -- a Boolean value indicating that the system Python 107 site-packages should be available to the environment (defaults to ``False``). 108 109 * ``clear`` -- a Boolean value which, if true, will delete the contents of 110 any existing target directory, before creating the environment. 111 112 * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the 113 Python binary rather than copying. 114 115 * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing 116 environment with the running Python - for use when that Python has been 117 upgraded in-place (defaults to ``False``). 118 119 * ``with_pip`` -- a Boolean value which, if true, ensures pip is 120 installed in the virtual environment. This uses :mod:`ensurepip` with 121 the ``--default-pip`` option. 122 123 * ``prompt`` -- a String to be used after virtual environment is activated 124 (defaults to ``None`` which means directory name of the environment would 125 be used). If the special string ``"."`` is provided, the basename of the 126 current directory is used as the prompt. 127 128 * ``upgrade_deps`` -- Update the base venv modules to the latest on PyPI 129 130 .. versionchanged:: 3.4 131 Added the ``with_pip`` parameter 132 133 .. versionadded:: 3.6 134 Added the ``prompt`` parameter 135 136 .. versionadded:: 3.9 137 Added the ``upgrade_deps`` parameter 138 139 Creators of third-party virtual environment tools will be free to use the 140 provided :class:`EnvBuilder` class as a base class. 141 142 The returned env-builder is an object which has a method, ``create``: 143 144 .. method:: create(env_dir) 145 146 Create a virtual environment by specifying the target directory 147 (absolute or relative to the current directory) which is to contain the 148 virtual environment. The ``create`` method will either create the 149 environment in the specified directory, or raise an appropriate 150 exception. 151 152 The ``create`` method of the :class:`EnvBuilder` class illustrates the 153 hooks available for subclass customization:: 154 155 def create(self, env_dir): 156 """ 157 Create a virtualized Python environment in a directory. 158 env_dir is the target directory to create an environment in. 159 """ 160 env_dir = os.path.abspath(env_dir) 161 context = self.ensure_directories(env_dir) 162 self.create_configuration(context) 163 self.setup_python(context) 164 self.setup_scripts(context) 165 self.post_setup(context) 166 167 Each of the methods :meth:`ensure_directories`, 168 :meth:`create_configuration`, :meth:`setup_python`, 169 :meth:`setup_scripts` and :meth:`post_setup` can be overridden. 170 171 .. method:: ensure_directories(env_dir) 172 173 Creates the environment directory and all necessary directories, and 174 returns a context object. This is just a holder for attributes (such as 175 paths), for use by the other methods. The directories are allowed to 176 exist already, as long as either ``clear`` or ``upgrade`` were 177 specified to allow operating on an existing environment directory. 178 179 .. method:: create_configuration(context) 180 181 Creates the ``pyvenv.cfg`` configuration file in the environment. 182 183 .. method:: setup_python(context) 184 185 Creates a copy or symlink to the Python executable in the environment. 186 On POSIX systems, if a specific executable ``python3.x`` was used, 187 symlinks to ``python`` and ``python3`` will be created pointing to that 188 executable, unless files with those names already exist. 189 190 .. method:: setup_scripts(context) 191 192 Installs activation scripts appropriate to the platform into the virtual 193 environment. 194 195 .. method:: upgrade_dependencies(context) 196 197 Upgrades the core venv dependency packages (currently ``pip`` and 198 ``setuptools``) in the environment. This is done by shelling out to the 199 ``pip`` executable in the environment. 200 201 .. versionadded:: 3.9 202 203 .. method:: post_setup(context) 204 205 A placeholder method which can be overridden in third party 206 implementations to pre-install packages in the virtual environment or 207 perform other post-creation steps. 208 209 .. versionchanged:: 3.7.2 210 Windows now uses redirector scripts for ``python[w].exe`` instead of 211 copying the actual binaries. In 3.7.2 only :meth:`setup_python` does 212 nothing unless running from a build in the source tree. 213 214 .. versionchanged:: 3.7.3 215 Windows copies the redirector scripts as part of :meth:`setup_python` 216 instead of :meth:`setup_scripts`. This was not the case in 3.7.2. 217 When using symlinks, the original executables will be linked. 218 219 In addition, :class:`EnvBuilder` provides this utility method that can be 220 called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to 221 assist in installing custom scripts into the virtual environment. 222 223 .. method:: install_scripts(context, path) 224 225 *path* is the path to a directory that should contain subdirectories 226 "common", "posix", "nt", each containing scripts destined for the bin 227 directory in the environment. The contents of "common" and the 228 directory corresponding to :data:`os.name` are copied after some text 229 replacement of placeholders: 230 231 * ``__VENV_DIR__`` is replaced with the absolute path of the environment 232 directory. 233 234 * ``__VENV_NAME__`` is replaced with the environment name (final path 235 segment of environment directory). 236 237 * ``__VENV_PROMPT__`` is replaced with the prompt (the environment 238 name surrounded by parentheses and with a following space) 239 240 * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory 241 (either ``bin`` or ``Scripts``). 242 243 * ``__VENV_PYTHON__`` is replaced with the absolute path of the 244 environment's executable. 245 246 The directories are allowed to exist (for when an existing environment 247 is being upgraded). 248 249There is also a module-level convenience function: 250 251.. function:: create(env_dir, system_site_packages=False, clear=False, \ 252 symlinks=False, with_pip=False, prompt=None, \ 253 upgrade_deps=False) 254 255 Create an :class:`EnvBuilder` with the given keyword arguments, and call its 256 :meth:`~EnvBuilder.create` method with the *env_dir* argument. 257 258 .. versionadded:: 3.3 259 260 .. versionchanged:: 3.4 261 Added the ``with_pip`` parameter 262 263 .. versionchanged:: 3.6 264 Added the ``prompt`` parameter 265 266 .. versionchanged:: 3.9 267 Added the ``upgrade_deps`` parameter 268 269An example of extending ``EnvBuilder`` 270-------------------------------------- 271 272The following script shows how to extend :class:`EnvBuilder` by implementing a 273subclass which installs setuptools and pip into a created virtual environment:: 274 275 import os 276 import os.path 277 from subprocess import Popen, PIPE 278 import sys 279 from threading import Thread 280 from urllib.parse import urlparse 281 from urllib.request import urlretrieve 282 import venv 283 284 class ExtendedEnvBuilder(venv.EnvBuilder): 285 """ 286 This builder installs setuptools and pip so that you can pip or 287 easy_install other packages into the created virtual environment. 288 289 :param nodist: If true, setuptools and pip are not installed into the 290 created virtual environment. 291 :param nopip: If true, pip is not installed into the created 292 virtual environment. 293 :param progress: If setuptools or pip are installed, the progress of the 294 installation can be monitored by passing a progress 295 callable. If specified, it is called with two 296 arguments: a string indicating some progress, and a 297 context indicating where the string is coming from. 298 The context argument can have one of three values: 299 'main', indicating that it is called from virtualize() 300 itself, and 'stdout' and 'stderr', which are obtained 301 by reading lines from the output streams of a subprocess 302 which is used to install the app. 303 304 If a callable is not specified, default progress 305 information is output to sys.stderr. 306 """ 307 308 def __init__(self, *args, **kwargs): 309 self.nodist = kwargs.pop('nodist', False) 310 self.nopip = kwargs.pop('nopip', False) 311 self.progress = kwargs.pop('progress', None) 312 self.verbose = kwargs.pop('verbose', False) 313 super().__init__(*args, **kwargs) 314 315 def post_setup(self, context): 316 """ 317 Set up any packages which need to be pre-installed into the 318 virtual environment being created. 319 320 :param context: The information for the virtual environment 321 creation request being processed. 322 """ 323 os.environ['VIRTUAL_ENV'] = context.env_dir 324 if not self.nodist: 325 self.install_setuptools(context) 326 # Can't install pip without setuptools 327 if not self.nopip and not self.nodist: 328 self.install_pip(context) 329 330 def reader(self, stream, context): 331 """ 332 Read lines from a subprocess' output stream and either pass to a progress 333 callable (if specified) or write progress information to sys.stderr. 334 """ 335 progress = self.progress 336 while True: 337 s = stream.readline() 338 if not s: 339 break 340 if progress is not None: 341 progress(s, context) 342 else: 343 if not self.verbose: 344 sys.stderr.write('.') 345 else: 346 sys.stderr.write(s.decode('utf-8')) 347 sys.stderr.flush() 348 stream.close() 349 350 def install_script(self, context, name, url): 351 _, _, path, _, _, _ = urlparse(url) 352 fn = os.path.split(path)[-1] 353 binpath = context.bin_path 354 distpath = os.path.join(binpath, fn) 355 # Download script into the virtual environment's binaries folder 356 urlretrieve(url, distpath) 357 progress = self.progress 358 if self.verbose: 359 term = '\n' 360 else: 361 term = '' 362 if progress is not None: 363 progress('Installing %s ...%s' % (name, term), 'main') 364 else: 365 sys.stderr.write('Installing %s ...%s' % (name, term)) 366 sys.stderr.flush() 367 # Install in the virtual environment 368 args = [context.env_exe, fn] 369 p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) 370 t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) 371 t1.start() 372 t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) 373 t2.start() 374 p.wait() 375 t1.join() 376 t2.join() 377 if progress is not None: 378 progress('done.', 'main') 379 else: 380 sys.stderr.write('done.\n') 381 # Clean up - no longer needed 382 os.unlink(distpath) 383 384 def install_setuptools(self, context): 385 """ 386 Install setuptools in the virtual environment. 387 388 :param context: The information for the virtual environment 389 creation request being processed. 390 """ 391 url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py' 392 self.install_script(context, 'setuptools', url) 393 # clear up the setuptools archive which gets downloaded 394 pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz') 395 files = filter(pred, os.listdir(context.bin_path)) 396 for f in files: 397 f = os.path.join(context.bin_path, f) 398 os.unlink(f) 399 400 def install_pip(self, context): 401 """ 402 Install pip in the virtual environment. 403 404 :param context: The information for the virtual environment 405 creation request being processed. 406 """ 407 url = 'https://bootstrap.pypa.io/get-pip.py' 408 self.install_script(context, 'pip', url) 409 410 def main(args=None): 411 compatible = True 412 if sys.version_info < (3, 3): 413 compatible = False 414 elif not hasattr(sys, 'base_prefix'): 415 compatible = False 416 if not compatible: 417 raise ValueError('This script is only for use with ' 418 'Python 3.3 or later') 419 else: 420 import argparse 421 422 parser = argparse.ArgumentParser(prog=__name__, 423 description='Creates virtual Python ' 424 'environments in one or ' 425 'more target ' 426 'directories.') 427 parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', 428 help='A directory in which to create the ' 429 'virtual environment.') 430 parser.add_argument('--no-setuptools', default=False, 431 action='store_true', dest='nodist', 432 help="Don't install setuptools or pip in the " 433 "virtual environment.") 434 parser.add_argument('--no-pip', default=False, 435 action='store_true', dest='nopip', 436 help="Don't install pip in the virtual " 437 "environment.") 438 parser.add_argument('--system-site-packages', default=False, 439 action='store_true', dest='system_site', 440 help='Give the virtual environment access to the ' 441 'system site-packages dir.') 442 if os.name == 'nt': 443 use_symlinks = False 444 else: 445 use_symlinks = True 446 parser.add_argument('--symlinks', default=use_symlinks, 447 action='store_true', dest='symlinks', 448 help='Try to use symlinks rather than copies, ' 449 'when symlinks are not the default for ' 450 'the platform.') 451 parser.add_argument('--clear', default=False, action='store_true', 452 dest='clear', help='Delete the contents of the ' 453 'virtual environment ' 454 'directory if it already ' 455 'exists, before virtual ' 456 'environment creation.') 457 parser.add_argument('--upgrade', default=False, action='store_true', 458 dest='upgrade', help='Upgrade the virtual ' 459 'environment directory to ' 460 'use this version of ' 461 'Python, assuming Python ' 462 'has been upgraded ' 463 'in-place.') 464 parser.add_argument('--verbose', default=False, action='store_true', 465 dest='verbose', help='Display the output ' 466 'from the scripts which ' 467 'install setuptools and pip.') 468 options = parser.parse_args(args) 469 if options.upgrade and options.clear: 470 raise ValueError('you cannot supply --upgrade and --clear together.') 471 builder = ExtendedEnvBuilder(system_site_packages=options.system_site, 472 clear=options.clear, 473 symlinks=options.symlinks, 474 upgrade=options.upgrade, 475 nodist=options.nodist, 476 nopip=options.nopip, 477 verbose=options.verbose) 478 for d in options.dirs: 479 builder.create(d) 480 481 if __name__ == '__main__': 482 rc = 1 483 try: 484 main() 485 rc = 0 486 except Exception as e: 487 print('Error: %s' % e, file=sys.stderr) 488 sys.exit(rc) 489 490 491This script is also available for download `online 492<https://gist.github.com/vsajip/4673395>`_. 493 494 495.. _setuptools: https://pypi.org/project/setuptools/ 496.. _pip: https://pypi.org/project/pip/ 497