rusty-page-indexer

Performance

• Parallel Indexing: Uses Rayon for parallel file parsing (238 files in ~0.04s)
• Rust-Native Parsing: pdf-extract and pulldown-cmark for fast document processing
• Incremental Updates: Hash-based caching skips unchanged files

Multi-Repository Support

• Index multiple repos: Each indexed folder is tracked separately
• Query across all: Search spans all indexed repositories by default
• Manage indices: List, filter, and clean up indices easily

Unified Tree Architecture

• Folder → File → Section hierarchy preserves document structure
• Single tree per repo: Efficient storage and navigation
• Smart search: Auto-unwraps folder roots for better LLM context

---

| Feature | Original PageIndex | RustyPageIndex |

|---------|-------------------|----------------|

| Language | Python | Rust |

| Indexing | Per-file indices | Unified folder tree |

| Multi-repo | Not supported | Full support with list/clean |

| Parallelism | Sequential | Rayon parallel processing |

| Storage | Cloud-based (MCP) | Local filesystem |

| Tree Structure | Flat sections | Folder → File → Section hierarchy |

| Headerless Markdown | Empty tree | Auto-creates "Document" node |

---

Installation

One-liner Install (Unix/macOS):

```bash

curl -fsSL https://raw.githubusercontent.com/Algiras/rusty-pageindex/main/install.sh | bash

```

One-liner Install (Windows PowerShell):

```powershell

irm https://raw.githubusercontent.com/Algiras/rusty-pageindex/main/install.ps1 | iex

```

Via Cargo:

```bash

cargo install rusty-page-indexer

```

🧙 Use as an Agent Skill

```bash

npx skills add https://github.com/Algiras/rusty-pageindex --skill rusty-page-indexer

```

🔑 Authentication

```bash

# For OpenAI

rusty-page-indexer auth --api-key "your-key-here"

# For Ollama (local LLM)

rusty-page-indexer auth --api-key "ollama" --api-base "http://localhost:11434/v1" --model "llama3.2"

```

---

Indexing Documents

```bash

# Index a repository

rusty-page-indexer index ./my-project

# Index with LLM-generated summaries

rusty-page-indexer index ./my-project --enrich

# Force re-index (ignores cache)

rusty-page-indexer index ./my-project --force

# Preview what would be indexed

rusty-page-indexer index ./my-project --dry-run

```

Managing Multiple Repositories

```bash

# Index multiple repos

rusty-page-indexer index ./repo-a

rusty-page-indexer index ./repo-b

# List all indexed repositories

rusty-page-indexer list

# Example output:

# 📋 Indexed Repositories

# ────────────────────────────────────────────────────────────

# 📁 repo-a (125.3 KB)

# /Users/you/projects/repo-a

# 📁 repo-b (89.7 KB)

# /Users/you/projects/repo-b

# ────────────────────────────────────────────────────────────

# Total: 2 indices

```

Querying

```bash

# Search across ALL indexed repositories

rusty-page-indexer query "how does authentication work"

# Search within a specific repository

rusty-page-indexer query "kafka messaging" --path repo-a

```

Cleanup

```bash

# Remove a specific index

rusty-page-indexer clean repo-a

# Remove all indices

rusty-page-indexer clean --all

```

Status Information

```bash

rusty-page-indexer info

```

---

OpenAI Models (Remote)

| Model | Cost | Speed | Notes |

|-------|------|-------|-------|

| gpt-4o | $$$ | Fast | Best accuracy, recommended for complex queries |

| `gpt