CocoIndex

Overview

CocoIndex is an ultra-performant real-time data transformation framework for AI with incremental processing. This skill enables building indexing flows that extract data from sources, apply transformations (chunking, embedding, LLM extraction), and export to targets (vector databases, graph databases, relational databases).

Core capabilities:

1. Write indexing flows - Define ETL pipelines using Python
2. Create custom functions - Build reusable transformation logic
3. Operate flows - Run and manage flows using CLI or Python API

Key features:

• Incremental processing (only processes changed data)
• Live updates (continuously sync source changes to targets)
• Built-in functions (text chunking, embeddings, LLM extraction)
• Multiple data sources (local files, S3, Azure Blob, Google Drive, Postgres)
• Multiple targets (Postgres+pgvector, Qdrant, LanceDB, Neo4j, Kuzu)

For detailed documentation: https://cocoindex.io/docs/ Search documentation: https://cocoindex.io/docs/search?q=url%20encoded%20keyword

When to Use This Skill

Use when users request:

• "Build a vector search index for my documents"
• "Create an embedding pipeline for code/PDFs/images"
• "Extract structured information using LLMs"
• "Build a knowledge graph from documents"
• "Set up live document indexing"
• "Create custom transformation functions"
• "Run/update my CocoIndex flow"

Flow Writing Workflow

Step 1: Understand Requirements

Ask clarifying questions to understand:

Data source:

• Where is the data? (local files, S3, database, etc.)
• What file types? (text, PDF, JSON, images, code, etc.)
• How often does it change? (one-time, periodic, continuous)

Transformations:

• What processing is needed? (chunking, embedding, extraction, etc.)
• Which embedding model? (SentenceTransformer, OpenAI, custom)
• Any custom logic? (filtering, parsing, enrichment)

Target:

• Where should results go? (Postgres, Qdrant, Ne...