Platform Case Study · Content Intelligence
Content Intelligence Platform
From Admin Commit to Search and RAG Projections
This system turns one authoritative content mutation into independently rebuildable search and retrieval projections. The synchronous write path ends at PostgreSQL; Kafka carries post-commit work to OpenSearch and pgvector without coupling authoring latency to model or index availability.
Repository Wiki
Explore the architecture and implementation with source-linked repository documentation.
AI-generated repository documentation; it may lag behind the code. Devin sign-in and workspace access may be required.
1. Problem and Design Thesis
A portfolio article must be editable and immediately durable, but search enrichment, semantic chunking and embeddings are slower and less reliable than a database transaction. Calling those dependencies inline would make publishing fragile and create a dual-write problem.
Commit the source of truth once, then derive every search and AI projection asynchronously.
2. Architecture
3. Write and Publication Contract
| Stage | Responsibility | Guarantee |
|---|---|---|
| Validate | JWT allow-list, schema and optimistic version | Unauthorized or stale writes are rejected |
| Commit | Source row, content version, audit and outbox | One atomic PostgreSQL transaction |
| Publish | Post-commit outbox dispatcher | At-least-once Kafka delivery |
| Project | Search and RAG consumers | Idempotent upsert by source ID and version |
4. Search Projection
The search consumer fetches the committed source document, applies deterministic field mapping and optionally generates query expansions. It writes a versioned document to OpenSearch. Retrieval combines BM25 fields, query-time synonyms and bounded expansion terms; failed enrichment degrades to the original document instead of blocking indexing.
5. RAG Projection
The RAG consumer normalizes HTML, splits content into stable chunks and creates embeddings. New chunks become ACTIVE only after the new version is complete. Previous chunks can then be retired, preventing a partially indexed article from becoming visible to the agent.
- Chunk identity is derived from source ID, version and ordinal.
- Embedding calls are retryable and isolated from the admin request.
- The projection can be rebuilt from source content without data loss.
- Retrieval permission filters run before candidate ranking.
6. Failure and Replay Model
Consumers acknowledge only after a durable projection write. Poison events move to a dead-letter path with the source ID, version and failure category. Admin operators can retry a single job or replay from source data after an index mapping change.
7. Observability
Every content version shares correlation fields across the outbox, Kafka headers, indexing jobs and target documents. Dashboards expose queue age, indexing latency, retry count, zero-result rate and projection version drift.
8. Trade-offs
The design accepts eventual consistency: a published article can be durable before it becomes searchable. In return, authoring remains available during model or OpenSearch outages, and every derived store remains disposable and rebuildable.
9. Engineering Outcome
The platform unifies admin CRUD, versioning, OpenSearch and RAG behind one event-driven content lifecycle while preserving clear ownership: PostgreSQL owns truth; consumers own projections; Kafka owns delivery between them.