python-packaging

https://www.reddit.com/r/Python/comments/pwkb0q/introducing_pants_27_python_tool_lockfiles_yapf/

Poetry and Pants are similar in that both do dependency management, virtualenv management, and creating distributions. They’re different in that Pants focuses intensely on scale, like supporting many projects in the same repository.
For example, Pants coordinates the tools you might be using like Pytest, MyPy, Black/Yapf/isort, Bandit/Flake8/Pylint, and codegen with Protobufs. You have a consistent interface, like ./pants lint :: to run all your linters, with benefits like running the tools in parallel and with caching.
The key differentiator for Pants to tools like Tox and Poetry (or fancy bash scripts) is that it understands your project and its dependencies at the file level, like that the file app.py depends on util.py, which transitively depends on another_util.py. That happens automatically through “dependency inference”, mapping your imports to the rest of your project. Understanding your project’s dependencies at the file-level means that Pants can safely do things like cache your test results at the file-level—if none of the transitive dependencies of a test changed, the cache can be used. You can also do things like ./pants dependees --transitive //:Django for you to find everything that transitively uses Django.
Feel free to DM or stop by on Slack if you want to talk through anything! I’d be happy to help https://www.pantsbuild.org/docs/getting-help

file:///usr/lib/python3.10 (standard library)
Imports site.py (Lib/site.py at source)

• cpython/site.py at main · python/cpython
  
• sys.prefix comes from:
  
  • PYTHONHOME
    
  • cpython/sysmodule.c at main · python/cpython
    
  • Algorithm
    Starting from the location of the Python binary, search upwards.
    At each step, look for lib/pythonX.X/os.py.
    #define LANDMARK “os.py” → it’s harcoded
    cpython/getpath.c at main · python/cpython
    If it exists, we’ve found sys.prefix.
    If we never find it, fallback on hardcoded configure –prefix from build
    
• Also uses file:///usr/lib/python3.10/lib-dynload (.so files) → sys.exec_prefix
  
• site.py depends itself on the stdlib (bootstrapping problem)
  

1. Just running ~/.virtualenvs/<name>/bin/python
   
2. Prepend $PATH with ~/.virtualenvs/<name>/bin
   
3. Set $VIRTUAL_ENV to ~/.virtualenvs/<name>
   

Como virtualenvwrapper pero sin hooks. Está casi igual de poco mantenida
Tiene un comando “add” que te añade dependencias globales que tienen que estar instaladas en todos los entornos virtuales

1. pyenv
   Why? I need to run multiple Python versions, isolated from the system Python. pyenv makes it easy to install, manage, and switch between those multiple Pythons.
   
2. pipx
   Why? pipx lets me install Python-based CLI stuff (youtube-dl, awscli, doc2dash, etc.) without those projects’ dependencies messing up my global Python.
   
3. Poetry
   Why? Poetry handles dependency- and virtual-environment-management in a way that’s very intuitive (to me), and fits perfectly with my desired workflow.
   

—

• https://github.com/ephes/awesome-python-devenv
  

«I currently do not use Poetry, instead I use zpy, which provides pipz as a drop-in replacement for pipx»

Everytime a package changes (like /usr/lib/libssl.so.1.1 becomes /usr/lib/libssl.so.3) you have to run again pyenv install <version>

read
pyenv versions | grep -v system | xargs -I _ pyenv install -f _

https://github.com/jazzband/pip-tools/issues/1558

Hay ciertos paquetes que sólo estan en Windows, o que tienen distinto hash

https://gabnotes.org/pip-tools-for-python-dependencies-management/

Creo que no lo tiene y sería una buena opción a tener. Igual que tienes de dónde viene cada dependencia, que te ponga el tamaño y puedas ordenar el requirements.txt por tamaño en vez de alfabéticamente
https://stackoverflow.com/questions/34266159/how-to-see-sizes-of-installed-pip-packages
No incluye paquetes que tienen un nombre distinto al importar vs al instalar como beautifulsoup4 ↔ bs4

LANG=C LC_ALL=C pip list \
  | tail -n +3 \
  | awk '{print $1}' \
  | xargs pip show \
  | grep -E 'Location:|Name:' \
  | cut -d ' ' -f 2 \
  | paste -d ' ' - - \
  | awk '{print $2 "/" tolower($1)}' \
  | xargs du -sh 2> /dev/null \
  | sort -hr

Posible mejorada:

LANG=C LC_ALL=C pip list \
  | tail -n +3 \
  | awk '{print $1}' \
  | xargs -I {} python -c 'import os; from importlib_metadata import packages_distributions; print(os.environ["VIRTUAL_ENV"]+"/lib/python3.10/site-packages/"+([k for k, v in packages_distributions().items() if "{}" in v and "__dummy__" != k][0]))' \
  | xargs du -sh 2> /dev/null \
  | sort -hr

• Si tenemos la misma versión de unos mismos paquetes, estaría bien que no tuviésemos ese paquete repetido muchas veces a lo largo de muchos venvs
  
• La solución sería(?) algo así como tener un directorio común donde tuviésemos guardadas las versiones comunes de un mismo paquete, pero cuando actualicemos, que se detecte de algún modo esta diferencia
  

  • Una buena opción es usar ~.local, que es donde se instalan los paquetes con pip install pandas como usuario normal
    

        ~/.local/lib/python3.8/site-packages/pandas/*
        ~/.local/lib/python3.8/site-packages/pandas-1.2.2.dist-info/*
    

• Así, para los paquetes comunes, hacemos un link
  
  • Podría ser un hard link a cada uno de los archivos, o un soft link a la carpeta. Si intentas actualizar un paquete y tienes un soft link no sé cómo se gestiona, pero si tienes un hard link y se remplaza uno por otro debería ser transparente
    

  • De hecho las instalaciones editables de pip hace links de site-packages a donde tengas el paquete, se podría hacer una instalación editable del paquete del entorno virtual común en el venv actual
    https://www.reddit.com/r/learnpython/comments/ayx7za/how_does_pip_install_e_work_is_there_a_specific/
    

    Rather than take all the code and put it into the site-packages directory where my python.exe (or just python) is located, make a link in the site-packages directory telling it to look here for the package.
    

    Creo que no se puede hacer porque en el site-packages no hay ningún setup.py
    

• Esto probablemente debería ir con pip (cuando actualices un paquete, tiene que darse cuenta de que en el resto de venvs hay otra versión y deshacer los links, y quizás que te avise de que se han actualizado cuando abras el resto de venvs afectados), o quizás en virtualenvwrapper
  
• Un buen caso de uso es una aplicación que sea un “doctor” que te diga qué dependencias están duplicadas
  
  • incluso qué paquetes de qué entorno puedes subir/bajar de versión para que se conviertan en dependencias comunes
    

Flatpak OSTree

https://superuser.com/questions/909731/how-to-replace-all-duplicate-files-with-hard-links
I know of 4 command-line solutions for linux. My preferred one is the last one listed here, rdfind, because of all the options available.

• fdupes
  
  • This appears to be the most recommended/most well-known one.
    
  • It’s the simplest to use, but its only action is to delete duplicates.
    
  • To ensure duplicates are actually duplicates (while not taking forever to run), comparisons between files are done first by file size, then md5 hash, then bye-by-byte comparison.
    
• hardlink
  
  • Designed to, as the name indicates, replace found files with hardlinks.
    
  • Has a --dry-run option.
    
  • Does not indicate how contents are compared, but unlike all other options, does take into account file mode, owner, and modified time.
    
• duff
  
  • Made to find files that the user then acts upon; has no actions available.
    
  • Comparisons are done by file size, then sha1 hash.
    
    • Hash can be changed to sha256, sha384, or sha512.
      
    • Hash can be disabled to do a byte-by-byte comparison
      
• rdfind
  
  • Options have an unusual syntax (meant to mimic find?).
    
  • Several options for actions to take on duplicate files (delete, make symlinks, make hardlinks).
    
  • Has a dry-run mode.
    
  • Comparisons are done by file size, then first-bytes, then last-bytes, then either md5 (default) or sha1.
    
  • Ranking of files found makes it predictable which file is considered the original.
    

• find-duplicate-dependencies - npm
  
• package-duplication-report - npm
  

https://github.com/python-poetry/poetry#what-about-pipenv
https://pypi.org/project/jaraco.clipboard/ porque convierte . en - y en _

• Three tools (pyenv, pipx, pipenv) make for smooth, isolated, reproducible Python developer and production environments.
  Ya se nos está empezando a ir la pinza como en javascript, 4000 herramientas distintas que tienen menos de un año cada una :/
  

• Utilizar Pathlib para que sea lo más portable posible
  https://realpython.com/python-pathlib/
  
• Utilizar tox para probar distintas versiones de Python
  
• Ponerlo con CI/CD para que no rompa el entorno virtual
  

• Hacer links independientes del sistema operativo
  https://docs.python.org/3/library/os.html#os.symlink
  

  On newer versions of Windows 10, unprivileged accounts can create symlinks if Developer Mode is enabled. When Developer Mode is not available/enabled, the SeCreateSymbolicLinkPrivilege privilege is required, or the process must be run as an administrator.
  

Opted-in centralized installation cache like pnpm.
Has a plugin for creating virtual environments associated to a project https://github.com/pdm-project/pdm-venv
https://discuss.python.org/t/pep-582-python-local-packages-directory/963/430

Exclude dependencies?
https://github.com/pdm-project/pdm/issues/1316

Direct → Only list packages you use
No version lock → ⇈ upgradability ⇊ reproducibility

Direct → Only list packages you use
Version lock → ⇊ upgradability ↑ reproducibility
Reproducibility is not the best since indirect dependencies are not specified

Full version lock, direct and indirect dependencies → ⇊ upgradability ⇈ reproducibility

1. You use the versioned direct file to generate the locked dependency file.
   List packages you use with a specific version
   
2. When creating an environment, you use the locked dependency file.
   List all the packages you use and its recursive dependencies with a specific version
   

This gives you the best of both worlds: most of the time you are just creating a new, reproducible environment from the locked dependency file. When you want to upgrade, you regenerate the locked file, and since you’re starting with a versioned direct dependency list, the hope is that the changes of dependencies-of-dependencies won’t be too bad.

The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
    apt install python3.9-venv
You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.
Failing command: ['/home/julian/code/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']