llava

Use when:

• Building vision-language chatbots
• Visual question answering (VQA)
• Image description and captioning
• Multi-turn image conversations
• Visual instruction following
• Document understanding with images

Metrics:

• 23,000+ GitHub stars
• GPT-4V level capabilities (targeted)
• Apache 2.0 License
• Multiple model sizes (7B-34B params)

Use alternatives instead:

• GPT-4V: Highest quality, API-based
• CLIP: Simple zero-shot classification
• BLIP-2: Better for captioning only
• Flamingo: Research, not open-source

Installation

```bash

# Clone repository

git clone https://github.com/haotian-liu/LLaVA

cd LLaVA

# Install

pip install -e .

```

Basic usage

```python

from llava.model.builder import load_pretrained_model

from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token

from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN

from llava.conversation import conv_templates

from PIL import Image

import torch

# Load model

model_path = "liuhaotian/llava-v1.5-7b"

tokenizer, model, image_processor, context_len = load_pretrained_model(

model_path=model_path,

model_base=None,

model_name=get_model_name_from_path(model_path)

)

# Load image

image = Image.open("image.jpg")

image_tensor = process_images([image], image_processor, model.config)

image_tensor = image_tensor.to(model.device, dtype=torch.float16)

# Create conversation

conv = conv_templates["llava_v1"].copy()

conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")

conv.append_message(conv.roles[1], None)

prompt = conv.get_prompt()

# Generate response

input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)

with torch.inference_mode():

output_ids = model.generate(

input_ids,

images=image_tensor,

do_sample=True,

temperature=0.2,

max_new_tokens=512

)

response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()

print(response)

```

| Model | Parameters | VRAM | Quality |

|-------|------------|------|---------|

| LLaVA-v1.5-7B | 7B | ~14 GB | Good |

| LLaVA-v1.5-13B | 13B | ~28 GB | Better |

| LLaVA-v1.6-34B | 34B | ~70 GB | Best |

```python

# Load different models

model_7b = "liuhaotian/llava-v1.5-7b"

model_13b = "liuhaotian/llava-v1.5-13b"

model_34b = "liuhaotian/llava-v1.6-34b"

# 4-bit quantization for lower VRAM

load_4bit = True # Reduces VRAM by ~4×

```

```bash

# Single image query

python -m llava.serve.cli \

--model-path liuhaotian/llava-v1.5-7b \

--image-file image.jpg \

--query "What is in this image?"

# Multi-turn conversation

python -m llava.serve.cli \

--model-path liuhaotian/llava-v1.5-7b \

--image-file image.jpg

# Then type questions interactively

```

```bash

# Launch Gradio interface

python -m llava.serve.gradio_web_server \

--model-path liuhaotian/llava-v1.5-7b \

--load-4bit # Optional: reduce VRAM

# Access at http://localhost:7860

```

```python

# Initialize conversation

conv = conv_templates["llava_v1"].copy()

# Turn 1

conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")

conv.append_message(conv.roles[1], None)

response1 = generate(conv, model, image) # "A dog playing in a park"

# Turn 2

conv.messages[-1][1] = response1 # Add previous response

conv.append_message(conv.roles[0], "What breed is the dog?")

conv.append_message(conv.roles[1], None)

response2 = generate(conv, model, image) # "Golden Retriever"

# Turn 3

conv.messages[-1][1] = response2

conv.append_message(conv.roles[0], "What time of day is it?")

conv.append_message(conv.roles[1], None)

response3 = generate(conv, model, image)

```

Image captioning

```python

question = "Describe this image in detail."

response = ask(model, image, question)

```

Visual question answering

```python

question = "How many people are in the image?"

response = ask(model, image, question)

```

Object detection (textual)

```python

question = "List all the objects you can see in this image."

response = ask(model, image, question)

```

Scene understanding

```python

question = "What is happening in this scene?"

response = ask(model, image, question)

```

Document understanding

```python

question = "What is the main topic of this document?"

response = ask(model, document_image, question)

```

```bash

# Stage 1: Feature alignment (558K image-caption pairs)

bash scripts/v1_5/pretrain.sh

# Stage 2: Visual instruction tuning (150K instruction data)

bash scripts/v1_5/finetune.sh

```

```python

# 4-bit quantization

tokenizer, model, image_processor, context_len = load_pretrained_model(

model_path="liuhaotian/llava-v1.5-13b",

model_base=None,

model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),

load_4bit=True # Reduces VRAM ~4×

)

# 8-bit quantization

load_8bit=True # Reduces VRAM ~2×

```

1. Start with 7B model - Good quality, manageable VRAM
2. Use 4-bit quantization - Reduces VRAM significantly
3. GPU required - CPU inference extremely slow
4. Clear prompts - Specific questions get better answers
5. Multi-turn conversations - Maintain conversation context
6. Temperature 0.2-0.7 - Balance creativity/consistency
7. max_new_tokens 512-1024 - For detailed responses
8. Batch processing - Process multiple images sequentially

| Model | VRAM (FP16) | VRAM (4-bit) | Speed (tokens/s) |

|-------|-------------|--------------|------------------|

| 7B | ~14 GB | ~4 GB | ~20 |

| 13B | ~28 GB | ~8 GB | ~12 |

| 34B | ~70 GB | ~18 GB | ~5 |

On A100 GPU

LLaVA achieves competitive scores on:

• VQAv2: 78.5%
• GQA: 62.0%
• MM-Vet: 35.4%
• MMBench: 64.3%

1. Hallucinations - May describe things not in image
2. Spatial reasoning - Struggles with precise locations
3. Small text - Difficulty reading fine print
4. Object counting - Imprecise for many objects
5. VRAM requirements - Need powerful GPU
6. Inference speed - Slower than CLIP

LangChain

```python

from langchain.llms.base import LLM

class LLaVALLM(LLM):

def _call(self, prompt, stop=None):

# Custom LLaVA inference

return response

llm = LLaVALLM()

```

Gradio App

```python

import gradio as gr

def chat(image, text, history):

response = ask_llava(model, image, text)

return response

demo = gr.ChatInterface(

chat,

additional_inputs=[gr.Image(type="pil")],

title="LLaVA Chat"

)

demo.launch()

```

• GitHub: https://github.com/haotian-liu/LLaVA ⭐ 23,000+
• Paper: https://arxiv.org/abs/2304.08485
• Demo: https://llava.hliu.cc
• Models: https://huggingface.co/liuhaotian
• License: Apache 2.0