python-performance-optimization

• Code runs too slowly for production requirements
• High CPU usage or memory consumption issues
• Need to reduce API response times or batch processing duration
• Application fails to scale under load
• Optimizing data processing pipelines or scientific computing
• Reducing cloud infrastructure costs through efficiency gains
• Profile-guided optimization after measuring performance bottlenecks

The Golden Rule: Never optimize without profiling first. 80% of execution time is spent in 20% of code.

Optimization Hierarchy (in priority order):

1. Algorithm complexity - O(n²) → O(n log n) provides exponential gains
2. Data structure choice - List → Set for lookups (10,000x faster)
3. Language features - Comprehensions, built-ins, generators
4. Caching - Memoization for repeated calculations
5. Compiled extensions - NumPy, Numba, Cython for hot paths
6. Parallelism - Multiprocessing for CPU-bound work

Key Principle: Algorithmic improvements beat micro-optimizations every time.

Load detailed guides for specific optimization areas:

| Task | Load reference |

| --- | --- |

| Profile code and find bottlenecks | skills/python-performance-optimization/references/profiling.md |

| Algorithm and data structure optimization | skills/python-performance-optimization/references/algorithms.md |

| Memory optimization and generators | skills/python-performance-optimization/references/memory.md |

| String concatenation and file I/O | skills/python-performance-optimization/references/string-io.md |

| NumPy, Numba, Cython, multiprocessing | skills/python-performance-optimization/references/acceleration.md |

Phase 1: Measure

1. Profile with cProfile - Identify slow functions
2. Line profile hot paths - Find exact slow lines
3. Memory profile - Check for memory bottlenecks
4. Benchmark baseline - Record current performance

Phase 2: Analyze

1. Check algorithm complexity - Is it O(n²) or worse?
2. Evaluate data structures - Are you using lists for lookups?
3. Identify repeated work - Can results be cached?
4. Find I/O bottlenecks - Database queries, file operations

Phase 3: Optimize

1. Improve algorithms first - Biggest impact
2. Use appropriate data structures - Set/dict for O(1) lookups
3. Apply caching - @lru_cache for expensive functions
4. Use generators - For large datasets
5. Leverage NumPy/Numba - For numerical code
6. Parallelize - Multiprocessing for CPU-bound tasks

Phase 4: Validate

1. Re-profile - Verify improvements
2. Benchmark - Measure speedup quantitatively
3. Test correctness - Ensure optimizations didn't break functionality
4. Document - Explain why optimization was needed

Pattern 1: Replace List with Set for Lookups

```python

# Slow: O(n) lookup

if item in large_list: # Bad

# Fast: O(1) lookup

if item in large_set: # Good

```

Pattern 2: Use Comprehensions

```python

# Slower

result = []

for i in range(n):

result.append(i * 2)

# Faster (35% speedup)

result = [i * 2 for i in range(n)]

```

Pattern 3: Cache Expensive Calculations

```python

from functools import lru_cache

@lru_cache(maxsize=None)

def expensive_function(n):

# Result cached automatically

return complex_calculation(n)

```

Pattern 4: Use Generators for Large Data

```python

# Memory inefficient

def read_file(path):

return [line for line in open(path)] # Loads entire file

# Memory efficient

def read_file(path):

for line in open(path): # Streams line by line

yield line.strip()

```

Pattern 5: Vectorize with NumPy

```python

# Pure Python: ~500ms

result = sum(i**2 for i in range(1000000))

# NumPy: ~5ms (100x faster)

import numpy as np

result = np.sum(np.arange(1000000)**2)

```

1. Optimizing before profiling - You'll optimize the wrong code
2. Using lists for membership tests - Use sets/dicts instead
3. String concatenation in loops - Use "".join() or StringIO
4. Loading entire files into memory - Use generators
5. N+1 database queries - Use JOINs or batch queries
6. Ignoring built-in functions - They're C-optimized and fast
7. Premature optimization - Focus on algorithmic improvements first
8. Not benchmarking - Always measure improvements quantitatively

Start here: Profile with cProfile to find bottlenecks

Hot path is algorithm?

• Yes → Check complexity, improve algorithm, use better data structures
• No → Continue

Hot path is computation?

• Numerical loops → NumPy or Numba
• CPU-bound → Multiprocessing
• Already fast enough → Done

Hot path is memory?

• Large data → Generators, streaming
• Many objects → __slots__, object pooling
• Caching needed → @lru_cache or custom cache

Hot path is I/O?

• Database → Batch queries, indexes, connection pooling
• Files → Buffering, streaming
• Network → Async I/O, request batching

1. Profile before optimizing - Measure to find real bottlenecks
2. Optimize algorithms first - O(n²) → O(n) beats micro-optimizations
3. Use appropriate data structures - Set/dict for lookups, not lists
4. Leverage built-ins - C-implemented built-ins are faster than pure Python
5. Avoid premature optimization - Optimize hot paths identified by profiling
6. Use generators for large data - Reduce memory usage with lazy evaluation
7. Batch operations - Minimize overhead from syscalls and network requests
8. Cache expensive computations - Use @lru_cache or custom caching
9. Consider NumPy/Numba - Vectorization and JIT for numerical code
10. Parallelize CPU-bound work - Use multiprocessing to utilize all cores

• Python Performance: https://wiki.python.org/moin/PythonSpeed
• cProfile: https://docs.python.org/3/library/profile.html
• NumPy: https://numpy.org/doc/stable/user/absolute_beginners.html
• Numba: https://numba.pydata.org/
• Cython: https://cython.readthedocs.io/
• High Performance Python (Book by Gorelick & Ozsvald)