Package Managers

A software package for our purposes refers to multiple files of code and resources bundled together and a sold as a set.

A package manager is a collection of software tools that automates the process of installing, upgrading, configuring and removing software packages in a consistent manner. It is often tied to a particular operating system or language.

This section described various software package managers that are useful to know.

pip - the default package installer for Python that comes with Python
conda - the package manager for the Anaconda platform, which is widely used by data scientists (Python / R)
npm - the package manager for NodeJS
homebrew - a package manager for macOS

pip

The recommended tool for installing packages in Python is pip, which comes with Python. It supports installing from PYPI, version control, local projects, and directly from distribution files that have been downloaded previously.

Installing Packages

The most common way to use pip is from the command line within your virtual environment.

$ pip install SomePackage              # latest version
$ pip install SomePackage==1.0.4       # specific version
$ pip install 'SomePackage>=1.0.4'     # minimum version

If you are not within your virtual environment, you may need to precede the pip install with ‘python’ in order to run the command: python pip install SomePackage.

To uninstall a package, use the same format, but with the command ‘uninstall’.

$ pip uninstall SomePackage              # latest version
$ pip uninstall SomePackage==1.0.4       # specific version
$ pip uninstall 'SomePackage>=1.0.4'     # minimum version

Listing Packages

To list installed packages:

$ pip list

docutils (0.9.1)
Jinja2 (2.6)
Pygments (1.5)
Sphinx (1.1.2)

To list outdated packages, and show the latest version available:

$ pip list --outdated

Package            Version Latest      Type
------------------ ------- ----------- -----
Django             1.10.5  1.11.6      wheel
django-environ     0.4.3   0.4.4       wheel
Faker              0.7.12  0.8.6       wheel
pytz      2017.3  2018.3     wheel
Sphinx             1.1.2   1.1.3       wheel

To update outdated packages to the latest version available:

$ pip install --upgrade pytz

Collecting pytz
  Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
Installing collected packages: pytz
  Found existing installation: pytz 2017.3
    Uninstalling pytz-2017.3:
      Successfully uninstalled pytz-2017.3
Successfully installed pytz-2018.3

To show details about an installed package:

$ pip show sphinx

---
Name: Sphinx
Version: 1.1.3
Location: /my/env/lib/pythonx.x/site-packages
Requires: Pygments, Jinja2, docutils

For more information and examples, see the pip list and pip show reference pages.

Requirement Files

You can also install packages by using requirements files. Requirements files are files containing a list of packages to be installed. You also have the option to specify the exact version to use.

You can install all of the required packages conveniently like so:

pip install -r requirements.txt

Sample Requirements File:

## example-requirements.txt

## Requirements without Version Specifiers
## -------------------------------------------------------------------
nose
nose-cov
beautifulsoup4

## Requirements with Version Specifiers
## -------------------------------------------------------------------
## See https://www.python.org/dev/peps/pep-0440/#version-specifiers
docopt == 0.6.1       ## Version Matching. Must be version 0.6.1
keyring >= 4.1.1      ## Minimum version 4.1.1
coverage != 3.5       ## Version Exclusion. Anything except version 3.5
Mopidy-Dirble ~= 1.1  ## Compatible release. Same as >= 1.1, == 1.*

## Refer to other requirements files
## -------------------------------------------------------------------
-r other-requirements.txt

## A particular file
## -------------------------------------------------------------------
./downloads/numpy-1.9.2-cp34-none-win32.whl
http://wxpython.org/Phoenix/snapshot-builds/wxPython_Phoenix-3.0.3.dev1820+49a8884-cp34-none-win_amd64.whl

## Additional Requirements without Version Specifiers
## -------------------------------------------------------------------
## Same as 1st section, just here to show that you can put things in any order.
rejected
green

You can list the existing installed packages on the screen by using the freeze command:

$ pip freeze
docutils==0.11
Jinja2==2.7.2
MarkupSafe==0.19
Pygments==1.6
Sphinx==1.2.2

To create and save a requirements.txt file automatically:

$ pip freeze > requirements.txt

Details on the format of the files are here: Requirements File Format.

Conda

Conda is a package manager application that manages packages, dependencies and environments for multiple languages including Python, JavaScript, C/C++, etc. Conda quickly installs, runs, and updates packages and their dependencies.

It is downloaded with Anaconda or Miniconda. All instructions in this section refer to the full version using Python 3 (Anaconda3). Detailed documentation can be found at Using Conda. And a command reference can be found at https://docs.conda.io/projects/conda/en/latest/commands.html?highlight=commands or access from the command line with the --help flag.

The information in this section is from the Conda User Guide provided by Continuum Analytics .

Installing Anaconda

Windows:: Download the Windows installer and follow the prompts.

MacOS:: Download the macOS installer and follow the instructions.

Updating Anaconda

In the command prompt (with Administrator rights), navigate to the directory containing the Anaconda program and type conda update conda, then y to proceed.

c:\ProgramData\Anaconda3>conda update conda
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment C:\ProgramData\Anaconda3:

The following packages will be UPDATED:

    conda: 4.3.14-py36_1 --> 4.3.17-py36_0

Proceed ([y]/n)?

Managing Environments

Below are the main commands for managing environments. To find additional commands, see Conda Environments.

Create – Creates a new virtual environment with the specified version of Python and includes any packages that are listed.
```
conda create –-name <<envname>> python=3 django
```
Change (activate/deactivate) – Changes to the specified environment. Once an environment is activated, the environment name will show in the prompt in parenthesis.
```
activate <<envname>>
```
List – Shows a list of all existing virtual environments with an asterisk by the one that you are currently using
```
conda info –-envs
```

Clone – Copy an existing environment into a new one.

conda create --name <<new env name>> --clone <<old env name>>

Remove – Deletes the specified environment.
```
conda remove --name <<env name>> --all
```

Managing Packages

Conda provides ways to manage your Python packages. To find additional commands, see Managing Packages.

List all installed packages
- To see a list of packages in the active environment:
```
conda list
```
- To see a list of packages in a non-active environment:
```
conda list -n <<env name>>
```
Search for a package – searches for a specified package to install
Install a package – Installs a package in the specified environment. (There is a different procedure if packages are not available using conda install.)
Update a package – Update a package (or Python itself) with conda.
```
conda update <<packagename>>==<<version>>
```
Remove a package – Uninstalls a package from the specified environment.
- To remove a package from the active environment:
```
conda remove <<package name>>
```
- To remove a package from an inactive environment:
```
conda remove -n <<env name>> <<package name>>
```

NPM

Before explaining what NPM is, we need back up and explain what NodeJS is.

Traditionally, JavaScript was only used on the client-side in browsers to interact with the DOM. Node.js or Node is a server-side Javascript execution engine. It allows us to write and use JavaScript code in our development environment.

NPM (Node Package Manager) is the package manager for NodeJS.

There are two ways to install npm packages. You choose which kind of installation to use based on how you intend to use the package.

local install - (default) Install a package locally to only be used on the current project.
global install - Install a package so that it can be used as a command line tool.

Additionally, there are two types of dependencies used for packages:

dependencies – These packages are required by your application in production.
devDependencies – These packages are only needed for development and testing.

Note: To learn more about the install command's behavior, check out the CLI doc page.

NPM Commands

Node comes with NPM installed. However, NPM gets updated more frequently than Node does, so it is a good idea to frequently check that you have the latest version.

Check Node version – Run node -v to see the current version on your machine.

Check NPM version - Run npm -v to see the current version of NPM.

Update NPM – To update your current version of NPM, run the following command:

npm install npm@latest -g to verify.

INSTALLING PACKAGES

Initialize NPM – Initialize NPM and create the initial package.json for your project file by running npm init.

Install local packages – A package can be downloaded with the command npm install <package_name>. This will create the node_modules/ directory in your project folder (if one doesn't exist yet), and will download the package(s) to that directory.

To designate a package as either a project dependency or a development dependency only, use the --save and --save-dev install flags.

npm install <package_name> --save – Adds an entry to the package.json's dependencies section.
npm install <package_name> --save-dev – Adds an entry to the package.json's devDependencies section.

Uninstall local packages – You can remove a package from the node_modules/ directory using npm uninstall <package>. To remove it from the dependencies in package.json, you will need to use the --save or --save-dev flag:

## Remove dependency package from package.json
npm uninstall --save lodash

## Remove dev dependency package from package.json
npm uninstall --save-dev pixrem

Install global packages – To download packages globally, you simply use the command:

npm install -g <package>.

Uninstall global packages – Global packages can be uninstalled with:

npm uninstall -g <package>.

UPDATING PACKAGES

Check for Outdated Packages

You can easily see which packages are outdated by running the command npm outdated.

npm outdated – Checks for local packages that need updating. npm outdated -g – Checks for global packages that need updating.

You will see something like this:

$ npm outdated
Package      Current   Wanted   Latest  Location
glob          5.0.15   5.0.15    6.0.1  test-outdated-output
nothingness    0.0.3      git      git  test-outdated-output
npm            3.5.1    3.5.2    3.5.1  test-outdated-output
local-dev      0.0.3   linked   linked  test-outdated-output
once           1.3.2    1.3.3    1.3.3  test-outdated-output

Output notes:

wanted is the maximum version of the package that satisfies the semver range specified in package.json. If there's no available semver range (i.e. you're running npm outdated --global, or the package isn't included in package.json), then wanted shows the currently-installed version.
latest is the version of the package tagged as latest in the registry. Running npm publish with no special configuration will publish the package with a dist-tag of latest. This may or may not be the maximum version of the package, or the most-recently published version of the package, depending on how the package's developer manages the latest dist-tag.
location is where in the dependency tree the package is located. Note that npm outdated defaults to a depth of 0, so unless you override that with --depth=x, you will only see top-level dependencies that are outdated.
package type (when using --long / -l) tells you whether this package is a dependency or a devDependency. Packages not included in package.json are always marked dependencies.

Important: This command will only update to the top version indicated in the package.json file. For example, if the latest version is 8.1.0, but the package.json file requires ^7.1.2, the highest version that npm will update to is 7.9.9.

Update Packages

Every so often, you should update the packages that you depend on so you can get any security or update patches that have been made to the code upstream.

Update local packages – To update local packages, run npm update in the same directory as your package.json file. To confirm everything updated, run npm outdated. There should not be any results.
Update global packages – To update global packages, you can use npm update -g <package>. To find out which packages need to be updated, you can use npm outdated -g --depth=0.

Finding Packages in the Tree

To find all versions of the packages installed, as well as their dependencies, use the list command. This will list installed npm packages in a tree format.

npm ls <<packagename>>@<<version>>

You can also drill down into the tree levels using the --depth= parameter where 0 is the top level.

Level 0:

$ npm ls --depth=0
kruizetech-repo@1.0.0 C:\Users\blain\Documents\myWebSites\kruizetech-repo
+-- autoprefixer@7.2.1
+-- babel-core@6.26.0
+-- babel-loader@7.1.2
+-- babel-preset-es2015@6.24.1
+-- browser-sync@2.18.13
+-- browser-sync-client@2.5.1
  ...

Level 1:

$ npm ls --depth=1
kruizetech-repo@1.0.0 C:\Users\blain\Documents\myWebSites\kruizetech-repo
+-- autoprefixer@7.2.1
| +-- browserslist@2.9.1
| +-- caniuse-lite@1.0.30000778
| +-- normalize-range@0.1.2
| +-- num2fraction@1.2.2
| +-- postcss@6.0.14
| `-- postcss-value-parser@3.3.0
  ...

Using package.json

NPM uses a file called package.json to manage installed packages. It tracks what packages (with versions) the application depends on to run (both in production and in the development environment) and allows you to easily share with others.

Create the package.json – Run the command npm init which will ask a series of questions about the project and include that information within the package.json file that is created. It will look something like this when complete:

{
  "name": "my_package",
  "description": "",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/username/my_package.git"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/username/my_package/issues"
  },
  "homepage": "https://github.com/username/my_package"
}

Package.json contents:

name: the current directory name
version: always 1.0.0
description: info from the readme, else an empty string ""
main: always index.js
scripts: by default creates a empty test script
keywords: empty
author: empty
license: ISC
bugs: info from the current directory, if present
homepage: info from the current directory, if present

To install packages from an existing package.json file, use the following commands:

npm install – Installs only production dependencies from the package.json file
npm install --only=dev – Installs only development dependencies from the package.json file.

Package Versions

When you add packages as dependencies using the --save and --save-dev commands, the latest version of that package is installed and saved in the package.json file with a ^ notation.

{
  "name": "my_package",
  "version": "1.0.0",
  "dependencies": {
    "my_dep": "^1.0.0"
  },
  "devDependencies": {
    "my_test_framework": "^3.1.0"
  }
}

This notation denotes that your application uses a version of the package compatible with the version specified. This can make a significant difference in what gets installed for your application in the future and can cause issues if future versions don’t work the same.

Here is a list of the different ways you can express the dependencies needed for your application that you should be mindful of when creating your package.json.

Format	Meaning
`version`	Must match version exactly
`>version`	Must be greater than listed version
`>=version`	Must be greater than or equal to listed version
`<version`	Must be less than listed version
`<=version`	Must be less than or equal to listed version
`~version`	Approximately equivalent to listed version
`^version`	Compatible with listed version
`1.2.x`	1.2.0, 1.2.1, etc., but not 1.3.0
`http://...`	Specifies a tarball URL in place of a version range that will download and install
`*`	Matches any version
`“” (empty string)`	Matches any version (same as ‘*’)
`version1 – version2`	Must be greater than or equal to version1 and less than or equal to version2
`range1 \|\| range2`	Passes if either range1 or range2 are satisfied
`git...`	Refers to a git repository
`user/repo`	Refers to a Github repository
`tag`	A specific version tagged and published as tag
`path/path/path`	Points to a local directory that contains a package. (Only use for offline development.)

For more detailed information, see the semver section on npmjs.com.

NPM Configuration

NPM gets its configuration settings from various sources including the command line, environment variables, npmrc files and sometimes, the package.json file. You can update the configuration settings by editing the contents of the user and global npmrc files. All NPM config files contain a list of key = value parameters.

NPM gets its configuration values from the following sources (in this order):

1) Command Line Flags

The format for command line flags is as follows:

<<command>> --<<configuration parameter>> <<setting>>

Using a --flag without specifying a setting value sets that configuration parameter to true.

Example commands:

npm config set <key> <value> [-g|--global]
npm config get <key>
npm config delete <key>
npm config list [-l] [--json]
npm config edit
npm get <key>
npm set <key> <value> [-g|--global]

aliases: c

2) Environment Variables

Environment variables beginning with npm_config_ will be interpreted as a configuration parameter.

Notice that you need to use underscores instead of dashes, so --allow-same-version would become npm_config_allow_same_version=true.

3) NPMRC Files

The four relevant files are:

Per-project configuration file (/path/to/my/project/.npmrc)
Per-user configuration file (defaults to $HOME/.npmrc; configurable via CLI option --userconfig or environment variable $NPM_CONFIG_USERCONFIG)
Global configuration file (defaults to $PREFIX/etc/npmrc; configurable via CLI option --globalconfig or environment variable $NPM_CONFIG_GLOBALCONFIG)
NPM's built-in configuration file (/path/to/npm/npmrc)

Lines in .npmrc files are interpreted as comments when they begin with a ; or # character. .npmrc files are parsed by npm/ini, which specifies this comment syntax.

4) Default Configs

The default configuration can be viewed by running the command npm config ls -l.

To see a list of configuration parameters, see default-configs.

Homebrew

Homebrew is a package manager for Mac OS or Linux. It installs packages and applications directly from the terminal without downloading and dragging the app icon into the Applications window.

Useful Links

Essential Commands

brew --version

Format:

brew command [--verbose|-v] [options] [formula] …

With --verbose or -v, many commands print extra debugging information. Note that these flags should only appear after a command.

(Formula refers to a package.)

brew install [options] formula - Installs a new package or formula (Homebrew's name for a package installer). See https://docs.brew.sh/Manpage#install-options-formula for all of the options available.

brew uninstall formula - Uninstalls an existing package.

brew list [options] - Lists all installed formulae.

--full-name: Print formulae with fully-qualified names. If --full-name is not passed, other options (i.e. -1, -l, -t and -r) are passed to ls which produces the actual output.
--unbrewed: List all files in the Homebrew prefix not installed by Homebrew.
--versions: Show the version number for installed formulae, or only the specified formulae if formula are given.
--multiple: Only show formulae with multiple versions installed.
--pinned: Show the versions of pinned formulae, or only the specified (pinned) formulae if formula are given. See also pin, unpin.
-1: Force output to be one entry per line. This is the default when output is not to a terminal.
-l: List in long format. If the output is to a terminal, a total sum for all the file sizes is output on a line before the long listing.
-r: Reverse the order of the sort to get the oldest entries first.
-t: Sort by time modified (most recently modified first).

brew search (text|/text/) - Perform a substring search of cask tokens and formula names for text. If text is surrounded with slashes, then it is interpreted as a regular expression. The search for text is extended online to homebrew/core and homebrew/cask. If no search term is given, all locally available formulae are listed.

brew outdated [options] - Show formulae that have an updated version available.

brew update [options] - Fetch the newest version of Homebrew and all formulae from GitHub using git(1) and perform any necessary migrations.

--merge: git merge is used to include updates (rather than git rebase).

For the full command list, see the COMMANDS section on the Homebrew website.

Python

Homebrew provides one formula for Python 3.x (python) and another for Python 2.7.x (python@2).

The executables are organized as follows so that Python 2 and Python 3 can both be installed without conflict:

python3 points to Homebrew’s Python 3.x (if installed)
python2 points to Homebrew’s Python 2.7.x (if installed)
python points to Homebrew’s Python 2.7.x (if installed) otherwise the macOS system Python. Check out brew info python if you wish to add Homebrew’s 3.x python to your PATH.
pip3 points to Homebrew’s Python 3.x’s pip (if installed)
pip and pip2 point to Homebrew’s Python 2.7.x’s pip (if installed)

Site-packages and the PYTHONPATH

The site-packages folder is a directory that contains Python modules (especially bindings installed by other formulae). Homebrew creates it here:

$(brew --prefix)/lib/pythonX.Y/site-packages

So, for Python 3.y.z, you’ll find it at /usr/local/lib/python3.y/site-packages.

Python 3.y also searches for modules in:

/Library/Python/3.y/site-packages
~/Library/Python/3.y/lib/python/site-packages

Homebrew’s site-packages directory is first created if (1) any Homebrew formula with Python bindings are installed, or (2) upon running brew install python.

Terminology

Term	Description	Example
Formula	The package definition	`/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/foo.rb`
Keg	The installation prefix of a Formula	`/usr/local/Cellar/foo/0.1`
opt prefix	A symlink to the active version of a Keg	`/usr/local/opt/foo`
Cellar	All Kegs are installed here	`/usr/local/Cellar`
Tap	A Git repository of Formulae and/or commands	`/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core`
Bottle	Pre-built Keg used instead of building from source	`qt-4.8.4.mavericks.bottle.tar.gz`
Cask	An extension of Homebrew to install macOS native apps	`/Applications/MacDown.app/Contents/SharedSupport/bin/macdown`
Brew Bundle	An extension of Homebrew to describe dependencies	`brew 'myservice', restart_service: true`

.	.
Back to top	Forward to Other Tools ==>

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search