python-packaging

• Creating Python libraries for distribution
• Building command-line tools with entry points
• Publishing packages to PyPI or private repositories
• Setting up Python project structure
• Creating installable packages with dependencies
• Building wheels and source distributions
• Versioning and releasing Python packages
• Creating namespace packages
• Implementing package metadata and classifiers

1. Package Structure

• Source layout: src/package_name/ (recommended)
• Flat layout: package_name/ (simpler but less flexible)
• Package metadata: pyproject.toml, setup.py, or setup.cfg
• Distribution formats: wheel (.whl) and source distribution (.tar.gz)

2. Modern Packaging Standards

• PEP 517/518: Build system requirements
• PEP 621: Metadata in pyproject.toml
• PEP 660: Editable installs
• pyproject.toml: Single source of configuration

3. Build Backends

• setuptools: Traditional, widely used
• hatchling: Modern, opinionated
• flit: Lightweight, for pure Python
• poetry: Dependency management + packaging

4. Distribution

• PyPI: Python Package Index (public)
• TestPyPI: Testing before production
• Private repositories: JFrog, AWS CodeArtifact, etc.

Minimal Package Structure

```

my-package/

├── pyproject.toml

├── README.md

├── LICENSE

├── src/

│ └── my_package/

│ ├── __init__.py

│ └── module.py

└── tests/

└── test_module.py

```

Minimal pyproject.toml

```toml

[build-system]

requires = ["setuptools>=61.0"]

build-backend = "setuptools.build_meta"

[project]

name = "my-package"

version = "0.1.0"

description = "A short description"

authors = [{name = "Your Name", email = "you@example.com"}]

readme = "README.md"

requires-python = ">=3.8"

dependencies = [

"requests>=2.28.0",

]

[project.optional-dependencies]

dev = [

"pytest>=7.0",

"black>=22.0",

]

```

Pattern 1: Source Layout (Recommended)

```

my-package/

├── pyproject.toml

├── README.md

├── LICENSE

├── .gitignore

├── src/

│ └── my_package/

│ ├── __init__.py

│ ├── core.py

│ ├── utils.py

│ └── py.typed # For type hints

├── tests/

│ ├── __init__.py

│ ├── test_core.py

│ └── test_utils.py

└── docs/

└── index.md

```

Advantages:

• Prevents accidentally importing from source
• Cleaner test imports
• Better isolation

pyproject.toml for source layout:

```toml

[tool.setuptools.packages.find]

where = ["src"]

```

Pattern 2: Flat Layout

```

my-package/

├── pyproject.toml

├── README.md

├── my_package/

│ ├── __init__.py

│ └── module.py

└── tests/

└── test_module.py

```

Simpler but:

• Can import package without installing
• Less professional for libraries

Pattern 3: Multi-Package Project

```

project/

├── pyproject.toml

├── packages/

│ ├── package-a/

│ │ └── src/

│ │ └── package_a/

│ └── package-b/

│ └── src/

│ └── package_b/

└── tests/

```

Pattern 4: Full-Featured pyproject.toml

```toml

[build-system]

requires = ["setuptools>=61.0", "wheel"]

build-backend = "setuptools.build_meta"

[project]

name = "my-awesome-package"

version = "1.0.0"

description = "An awesome Python package"

readme = "README.md"

requires-python = ">=3.8"

license = {text = "MIT"}

authors = [

{name = "Your Name", email = "you@example.com"},

]

maintainers = [

{name = "Maintainer Name", email = "maintainer@example.com"},

]

keywords = ["example", "package", "awesome"]

classifiers = [

"Development Status :: 4 - Beta",

"Intended Audience :: Developers",

"License :: OSI Approved :: MIT License",

"Programming Language :: Python :: 3",

"Programming Language :: Python :: 3.8",

"Programming Language :: Python :: 3.9",

"Programming Language :: Python :: 3.10",

"Programming Language :: Python :: 3.11",

"Programming Language :: Python :: 3.12",

]

dependencies = [

"requests>=2.28.0,<3.0.0",

"click>=8.0.0",

"pydantic>=2.0.0",

]

[project.optional-dependencies]

dev = [

"pytest>=7.0.0",

"pytest-cov>=4.0.0",

"black>=23.0.0",

"ruff>=0.1.0",

"mypy>=1.0.0",

]

docs = [

"sphinx>=5.0.0",

"sphinx-rtd-theme>=1.0.0",

]

all = [

"my-awesome-package[dev,docs]",

]

[project.urls]

Homepage = "https://github.com/username/my-awesome-package"

Documentation = "https://my-awesome-package.readthedocs.io"

Repository = "https://github.com/username/my-awesome-package"

"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"

Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"

[project.scripts]

my-cli = "my_package.cli:main"

awesome-tool = "my_package.tools:run"

[project.entry-points."my_package.plugins"]

plugin1 = "my_package.plugins:plugin1"

[tool.setuptools]

package-dir = {"" = "src"}

zip-safe = false

[tool.setuptools.packages.find]

where = ["src"]

include = ["my_package*"]

exclude = ["tests*"]

[tool.setuptools.package-data]

my_package = ["py.typed", ".pyi", "data/.json"]

# Black configuration

[tool.black]

line-length = 100

target-version = ["py38", "py39", "py310", "py311"]

include = '\.pyi?$'

# Ruff configuration

[tool.ruff]

line-length = 100

target-version = "py38"

[tool.ruff.lint]

select = ["E", "F", "I", "N", "W", "UP"]

# MyPy configuration

[tool.mypy]

python_version = "3.8"

warn_return_any = true

warn_unused_configs = true

disallow_untyped_defs = true

# Pytest configuration

[tool.pytest.ini_options]

testpaths = ["tests"]

python_files = ["test_*.py"]

addopts = "-v --cov=my_package --cov-report=term-missing"

# Coverage configuration

[tool.coverage.run]

source = ["src"]

omit = ["/tests/"]

[tool.coverage.report]

exclude_lines = [

"pragma: no cover",

"def __repr__",

"raise AssertionError",

"raise NotImplementedError",

]

```

Pattern 5: Dynamic Versioning

```toml

[build-system]

requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]

build-backend = "setuptools.build_meta"

[project]

name = "my-package"

dynamic = ["version"]

description = "Package with dynamic version"

[tool.setuptools.dynamic]

version = {attr = "my_package.__version__"}

# Or use setuptools-scm for git-based versioning

[tool.setuptools_scm]

write_to = "src/my_package/_version.py"

```

In __init__.py:

```python

# src/my_package/__init__.py

__version__ = "1.0.0"

# Or with setuptools-scm

from importlib.metadata import version

__version__ = version("my-package")

```

Pattern 6: CLI with Click

```python

# src/my_package/cli.py

import click

@click.group()

@click.version_option()

def cli():

"""My awesome CLI tool."""

pass

@cli.command()

@click.argument("name")

@click.option("--greeting", default="Hello", help="Greeting to use")

def greet(name: str, greeting: str):

"""Greet someone."""

click.echo(f"{greeting}, {name}!")

@cli.command()

@click.option("--count", default=1, help="Number of times to repeat")

def repeat(count: int):

"""Repeat a message."""

for i in range(count):

click.echo(f"Message {i + 1}")

def main():

"""Entry point for CLI."""

cli()

if __name__ == "__main__":

main()

```

Register in pyproject.toml:

```toml

[project.scripts]

my-tool = "my_package.cli:main"

```

Usage:

```bash

pip install -e .

my-tool greet World

my-tool greet Alice --greeting="Hi"

my-tool repeat --count=3

```

Pattern 7: CLI with argparse

```python

# src/my_package/cli.py

import argparse

import sys

def main():

"""Main CLI entry point."""

parser = argparse.ArgumentParser(

description="My awesome tool",

prog="my-tool"

)

parser.add_argument(

"--version",

action="version",

version="%(prog)s 1.0.0"

)

subparsers = parser.add_subparsers(dest="command", help="Commands")

# Add subcommand

process_parser = subparsers.add_parser("process", help="Process data")

process_parser.add_argument("input_file", help="Input file path")

process_parser.add_argument(

"--output", "-o",

default="output.txt",

help="Output file path"

)

args = parser.parse_args()

if args.command == "process":

process_data(args.input_file, args.output)

else:

parser.print_help()

sys.exit(1)

def process_data(input_file: str, output_file: str):

"""Process data from input to output."""

print(f"Processing {input_file} -> {output_file}")

if __name__ == "__main__":

main()

```

Pattern 8: Build Package Locally

```bash

# Install build tools

pip install build twine

# Build distribution

python -m build

# This creates:

# dist/

# my-package-1.0.0.tar.gz (source distribution)

# my_package-1.0.0-py3-none-any.whl (wheel)

# Check the distribution

twine check dist/*

```

Pattern 9: Publishing to PyPI

```bash

# Install publishing tools

pip install twine

# Test on TestPyPI first

twine upload --repository testpypi dist/*

# Install from TestPyPI to test

pip install --index-url https://test.pypi.org/simple/ my-package

# If all good, publish to PyPI

twine upload dist/*

```

Using API tokens (recommended):

```bash

# Create ~/.pypirc

[distutils]

index-servers =

pypi

testpypi

[pypi]

username = __token__

password = pypi-...your-token...

[testpypi]

username = __token__

password = pypi-...your-test-token...

```

Pattern 10: Automated Publishing with GitHub Actions

```yaml

# .github/workflows/publish.yml

name: Publish to PyPI

on:

release:

types: [created]

jobs:

publish:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v3

- name: Set up Python

uses: actions/setup-python@v4

with:

python-version: "3.11"

- name: Install dependencies

run: |

pip install build twine

- name: Build package

run: python -m build

- name: Check package

run: twine check dist/*

- name: Publish to PyPI

env:

TWINE_USERNAME: __token__

TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

run: twine upload dist/*

```

Pattern 11: Including Data Files

```toml

[tool.setuptools.package-data]

my_package = [

"data/*.json",

"templates/*.html",

"static/css/*.css",

"py.typed",

]

```

Accessing data files:

```python

# src/my_package/loader.py

from importlib.resources import files

import json

def load_config():

"""Load configuration from package data."""

config_file = files("my_package").joinpath("data/config.json")

with config_file.open() as f:

return json.load(f)

# Python 3.9+

from importlib.resources import files

data = files("my_package").joinpath("data/file.txt").read_text()

```

Pattern 12: Namespace Packages

For large projects split across multiple repositories:

```

# Package 1: company-core

company/

└── core/

├── __init__.py

└── models.py

# Package 2: company-api

company/

└── api/

├── __init__.py

└── routes.py

```

Do NOT include __init__.py in the namespace directory (company/):

```toml

# company-core/pyproject.toml

[project]

name = "company-core"

[tool.setuptools.packages.find]

where = ["."]

include = ["company.core*"]

# company-api/pyproject.toml

[project]

name = "company-api"

[tool.setuptools.packages.find]

where = ["."]

include = ["company.api*"]

```

Usage:

```python

# Both packages can be imported under same namespace

from company.core import models

from company.api import routes

```

Pattern 13: C Extensions

```toml

[build-system]

requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"]

build-backend = "setuptools.build_meta"

[tool.setuptools]

ext-modules = [

{name = "my_package.fast_module", sources = ["src/fast_module.c"]},

]

```

Or with setup.py:

```python

# setup.py

from setuptools import setup, Extension

setup(

ext_modules=[

Extension(

"my_package.fast_module",

sources=["src/fast_module.c"],

include_dirs=["src/include"],

)

]

)

```

Pattern 14: Semantic Versioning

```python

# src/my_package/__init__.py

__version__ = "1.2.3"

# Semantic versioning: MAJOR.MINOR.PATCH

# MAJOR: Breaking changes

# MINOR: New features (backward compatible)

# PATCH: Bug fixes

```

Version constraints in dependencies:

```toml

dependencies = [

"requests>=2.28.0,<3.0.0", # Compatible range

"click~=8.1.0", # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0)

"pydantic>=2.0", # Minimum version

"numpy==1.24.3", # Exact version (avoid if possible)

]

```

Pattern 15: Git-Based Versioning

```toml

[build-system]

requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]

build-backend = "setuptools.build_meta"

[project]

name = "my-package"

dynamic = ["version"]

[tool.setuptools_scm]

write_to = "src/my_package/_version.py"

version_scheme = "post-release"

local_scheme = "dirty-tag"

```

Creates versions like:

• 1.0.0 (from git tag)
• 1.0.1.dev3+g1234567 (3 commits after tag)

Pattern 16: Editable Install

```bash

# Install in development mode

pip install -e .

# With optional dependencies

pip install -e ".[dev]"

pip install -e ".[dev,docs]"

# Now changes to source code are immediately reflected

```

Pattern 17: Testing in Isolated Environment

```bash

# Create virtual environment

python -m venv test-env

source test-env/bin/activate # Linux/Mac

# test-env\Scripts\activate # Windows

# Install package

pip install dist/my_package-1.0.0-py3-none-any.whl

# Test it works

python -c "import my_package; print(my_package.__version__)"

# Test CLI

my-tool --help

# Cleanup

deactivate

rm -rf test-env

```

Pattern 18: README.md Template

```markdown

# My Package

[](https://pypi.org/project/my-package/)

[](https://pypi.org/project/my-package/)

[](https://github.com/username/my-package/actions)

Brief description of your package.

```bash

pip install my-package

```

```python

from my_package import something

result = something.do_stuff()

```

• Feature 1
• Feature 2
• Feature 3

Full documentation: https://my-package.readthedocs.io

```bash

git clone https://github.com/username/my-package.git

cd my-package

pip install -e ".[dev]"

pytest

```

MIT

```

Pattern 19: Multi-Architecture Wheels

```yaml

# .github/workflows/wheels.yml

name: Build wheels

on: [push, pull_request]

jobs:

build_wheels:

name: Build wheels on ${{ matrix.os }}

runs-on: ${{ matrix.os }}

strategy:

matrix:

os: [ubuntu-latest, windows-latest, macos-latest]

steps:

- uses: actions/checkout@v3

- name: Build wheels

uses: pypa/cibuildwheel@v2.16.2

- uses: actions/upload-artifact@v3

with:

path: ./wheelhouse/*.whl

```

Pattern 20: Private Package Index

```bash

# Install from private index

pip install my-package --index-url https://private.pypi.org/simple/

# Or add to pip.conf

[global]

index-url = https://private.pypi.org/simple/

extra-index-url = https://pypi.org/simple/

# Upload to private index

twine upload --repository-url https://private.pypi.org/ dist/*

```

.gitignore for Python Packages

```gitignore

# Build artifacts

build/

dist/

*.egg-info/

*.egg

.eggs/

# Python

__pycache__/

*.py[cod]

*$py.class

*.so

# Virtual environments

venv/

env/

ENV/

# IDE

.vscode/

.idea/

*.swp

# Testing

.pytest_cache/

.coverage

htmlcov/

# Distribution

*.whl

*.tar.gz

```

MANIFEST.in

```

# MANIFEST.in

include README.md

include LICENSE

include pyproject.toml

recursive-include src/my_package/data *.json

recursive-include src/my_package/templates *.html

recursive-exclude * __pycache__

recursive-exclude .py[co]

```

• [ ] Code is tested (pytest passing)
• [ ] Documentation is complete (README, docstrings)
• [ ] Version number updated
• [ ] CHANGELOG.md updated
• [ ] License file included
• [ ] pyproject.toml is complete
• [ ] Package builds without errors
• [ ] Installation tested in clean environment
• [ ] CLI tools work (if applicable)
• [ ] PyPI metadata is correct (classifiers, keywords)
• [ ] GitHub repository linked
• [ ] Tested on TestPyPI first
• [ ] Git tag created for release

• Python Packaging Guide: https://packaging.python.org/
• PyPI: https://pypi.org/
• TestPyPI: https://test.pypi.org/
• setuptools documentation: https://setuptools.pypa.io/
• build: https://pypa-build.readthedocs.io/
• twine: https://twine.readthedocs.io/

1. Use src/ layout for cleaner package structure
2. Use pyproject.toml for modern packaging
3. Pin build dependencies in build-system.requires
4. Version appropriately with semantic versioning
5. Include all metadata (classifiers, URLs, etc.)
6. Test installation in clean environments
7. Use TestPyPI before publishing to PyPI
8. Document thoroughly with README and docstrings
9. Include LICENSE file
10. Automate publishing with CI/CD