Create a multimodal RAG system that processes both images and text for document understanding with table extraction.
You are a multimodal AI engineer. Build a RAG system that handles documents with text, images, tables, and charts. ## Configuration - **Document Types**: [DOC_TYPES e.g. PDF reports, slide decks, scanned documents] - **Vision Model**: [VISION e.g. GPT-5-Vision / Claude 4 Vision / Gemini 2.5 Pro] - **Text Model**: [TEXT_LLM e.g. Claude 4 Opus / GPT-5] - **Vector Store**: [VECTOR_STORE e.g. Qdrant / ChromaDB / Weaviate] - **OCR Engine**: [OCR e.g. Tesseract / Azure Document Intelligence / AWS Textract] ## System Components ### 1. Document Processing (`processing/`) - PDF to image conversion with configurable DPI - Layout analysis: detect text blocks, images, tables, charts - Text extraction with [OCR] for scanned documents - Table extraction to structured format (pandas DataFrame) - Chart/graph interpretation using [VISION] - Image captioning for visual elements - Metadata extraction: title, author, date, page numbers ### 2. Multimodal Chunking (`chunking/`) - Text chunks with semantic boundary detection - Image chunks with captions and surrounding context - Table chunks with column headers and summary - Chart chunks with data interpretation - Cross-modal linking: associate images with nearby text - Hierarchical chunks: page → section → element ### 3. Multimodal Embedding (`embedding/`) - Text embeddings: standard text encoder - Image embeddings: CLIP or SigLIP - Unified embedding space with projection layers - Table embeddings: linearize and embed - Composite embeddings for multi-element chunks ### 4. Retrieval & Generation (`retrieval/`) - Query classification: text-only, image-seeking, table-seeking - Cross-modal retrieval with [VECTOR_STORE] - Context assembly: combine text + image + table chunks - Multimodal generation using [VISION] with retrieved context - Source attribution with page and element references - Answer grounding verification ### 5. API & UI (`interface/`) - Upload and process documents endpoint - Natural language query endpoint - Visual answer display with highlighted sources - Conversation history with document context Provide complete implementation with sample documents and evaluation metrics.
Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.5 Pro.
Prepare sample documents of each type. Configure vision and text models with API keys. Process documents through the pipeline and test with queries that span text, table, and image content.
Initial release
Sign in and download this prompt to leave a review.