1. Executive Summary

This project is not a traditional static portfolio website. It is a production-grade distributed platform built around event-driven microservices, AI-assisted user interaction, real-time analytics, search indexing, notification delivery, and reliable asynchronous processing.

The portfolio website is only the user-facing surface. Behind it is a backend platform designed around the same engineering concerns found in real production systems:

• Reliable content propagation through Kafka using the Transactional Outbox Pattern.
• Search indexing through OpenSearch BM25, index-time query expansion, and query-time synonym expansion.
• AI orchestration through an Agent Service, RAG, and MCP-based tool governance.
• Analytics processing through an idempotent event pipeline with deduplication and privacy-aware aggregation.
• Email notification delivery through a state-machine-based dispatcher with retry, backoff, and failure isolation.

The architecture prioritizes reliability, loose coupling, graceful degradation, observability, cost efficiency, and clear service ownership.

Core Principle: Keep the source of truth simple and reliable, then use events to build scalable, independent projections around it.

2. Product Context

The system supports five major user-facing capabilities:

3. Architecture at a Glance

4. High-Level Architecture

Architecture Principle

Each major capability is isolated behind an independently deployable service. Services communicate asynchronously through Kafka when eventual consistency is acceptable, and synchronously through APIs only when a user-facing request requires an immediate response.

This prevents one slow or failing downstream system from breaking the primary publishing workflow.

5. Service Ownership and Boundaries

6. Core Design Goals

Functional Requirements

• Publish projects, blogs, and portfolio content.
• Index content for full-text and semantic-friendly search.
• Generate RAG embeddings for AI chat.
• Send content update notifications to subscribers.
• Track page views and visitor activity.
• Support AI assistant workflows through controlled backend tools.
• Rebuild derived projections when search, RAG, analytics, or notification views become stale.

Non-Functional Requirements

Deep Dive 1 — Content Publishing Pipeline

7. Problem

When an admin publishes a project or blog, multiple downstream systems need to react:

• Search must update the OpenSearch document.
• RAG must regenerate embeddings.
• Notification Service may email subscribers.
• Analytics or audit systems may record the publishing activity.

A naive implementation would call these downstream systems synchronously inside the publish request:

Admin Service
  -> Save content
  -> Call Search Service
  -> Call RAG Service
  -> Call Notification Service

This design is fragile:

• If Search is unavailable, content publishing may fail.
• If Notification is slow, the admin request becomes slow.
• If the database commit succeeds but Kafka publish fails, downstream systems miss the event.
• If Kafka publishes but the database transaction rolls back, consumers may process a phantom event.

The system needs reliable event propagation without distributed transactions.

8. Solution — Transactional Outbox Pattern

The Admin Service writes the content row and the outbox event inside the same database transaction.

The Admin Service does not need a distributed transaction across PostgreSQL and Kafka. PostgreSQL is the durable source of truth for the event until the publisher successfully sends it to Kafka.

9. Outbox Table Design

CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(64) NOT NULL,
    aggregate_id UUID NOT NULL,
    event_type VARCHAR(128) NOT NULL,
    event_version INT NOT NULL,
    payload JSONB NOT NULL,
    status VARCHAR(32) NOT NULL,
    retry_count INT NOT NULL DEFAULT 0,
    next_retry_at TIMESTAMP,
    created_at TIMESTAMP NOT NULL,
    published_at TIMESTAMP
);

Example event:

{
  "eventId": "018f9f5a-7d52-7f43-9b7e-91b21a0cc921",
  "eventType": "CONTENT_PUBLISHED",
  "eventVersion": 1,
  "contentId": "project_123",
  "contentType": "PROJECT",
  "title": "Portfolio Platform",
  "slug": "portfolio-platform",
  "publishedAt": "2026-06-30T18:30:00Z"
}

10. Kafka Topic Design

content.search.index.v1
content.rag.index.v1
content.notification.feature_updates.v1
content.audit.activity.v1

The platform uses topic-level fan-out instead of one generic content topic because each downstream pipeline has different scaling, retention, replay, and failure-handling requirements.

11. Why This Design Works

The Transactional Outbox pattern gives the system strong local consistency without distributed transactions.

• If the database transaction rolls back, no outbox event exists.
• If the database transaction commits, the event is durable.
• If Kafka is temporarily unavailable, the outbox publisher retries.
• If a consumer receives the same event more than once, the consumer deduplicates by event ID.

Deep Dive 2 — Search Architecture

12. Problem

Portfolio search needs to handle more than exact keyword matching.

A user may search:

k8s
spring async event
vector search
slc kayak
system design outbox

The relevant documents may contain:

Kubernetes
event-driven architecture
pgvector
Salt Lake City
transactional outbox pattern

A basic SQL ILIKE search would miss many of these relationships. It also does not provide ranking, stemming, synonyms, or relevance tuning.

13. Solution — BM25 + Index-Time Expansion + Query-Time Synonyms

14. Three-Layer Recall Strategy

Layer 1 — BM25 Full-Text Ranking

OpenSearch provides standard full-text search and ranking through BM25.

The query uses weighted fields:

title^3
summary^2
search_terms^1.5
content^1

Title matches rank highest. Summary matches are also strong. Generated search terms improve recall. Body content provides broad coverage.

Layer 2 — Index-Time Semantic Expansion

When content is published, the Search Indexer consumes the Kafka event and builds an OpenSearch document.

{
  "id": "project_123",
  "type": "PROJECT",
  "title": "Portfolio Platform",
  "summary": "Production-grade event-driven microservice architecture",
  "content": "...",
  "search_terms": [
    "event driven architecture",
    "kafka outbox",
    "microservice portfolio",
    "spring boot kafka",
    "opensearch indexing",
    "rag search",
    "system design"
  ],
  "published_at": "2026-06-30T18:30:00Z"
}

The search_terms field is generated at index time using Gemini. This follows a Doc2Query-style approach: predict likely user search phrases when the document is indexed.

The benefit is cost control. The platform pays the LLM cost once during indexing instead of paying for every user query.

Layer 3 — Query-Time Synonym Expansion

OpenSearch applies a synonym graph filter at query time.

k8s => kubernetes
js => javascript
slc => salt lake city
rag => retrieval augmented generation

So a query like:

k8s deployment

can match content containing:

Kubernetes deployment

without calling an LLM during search.

15. Search Failure Handling

Search enrichment is designed to fail open.

• If Gemini is unavailable, the indexer still writes the document to OpenSearch without generated search terms.
• Publishing does not fail.
• Basic search still works.
• Search quality may temporarily degrade.
• The event can be replayed later to regenerate enriched search terms.

Deep Dive 3 — AI Agent and MCP Tool Governance

16. Problem

The chat assistant should answer user questions and interact with backend capabilities. However, allowing an LLM to call backend APIs directly is unsafe.

Potential risks include:

• The model calls the wrong endpoint.
• The model retries a mutation and creates duplicate side effects.
• The model attempts a high-risk action without confirmation.
• The model bypasses authorization.
• Internal APIs become exposed through prompt injection or tool misuse.

The AI system needs to treat the LLM as a reasoning layer, not as a trusted execution layer.

17. Solution — Agent Service + MCP Gateway

The Agent Service handles reasoning and orchestration. The MCP Gateway handles execution governance.

The LLM never directly calls internal services.

18. Tool Governance Model

Every tool is registered with metadata:

{
  "toolName": "publishContent",
  "riskLevel": "HIGH",
  "requiredRole": "ADMIN",
  "idempotencyRequired": true,
  "schema": {
    "contentId": "string",
    "publish": "boolean"
  }
}

19. Why This Design Works

The MCP Gateway provides:

• Schema validation.
• Role-based access control.
• Tool allowlisting.
• Idempotency enforcement.
• Risk-level gating.
• Audit logging.
• Duplicate call protection.

Deep Dive 4 — Real-Time Analytics Pipeline

20. Problem

The platform tracks visitor activity to support:

• Page view counts.
• Content popularity.
• Visitor location visualization.
• 3D globe activity.
• Time-based traffic trends.
• Admin analytics dashboards.

However, analytics must be reliable and privacy-aware. The system should avoid storing raw IP addresses or overly precise user location data.

21. Analytics Data Flow

22. Event Schema

{
  "eventId": "018fa20b-2d61-7c84-9d30-144fd8f67a71",
  "eventType": "PAGE_VIEW",
  "path": "/projects/portfolio-platform",
  "referrer": "https://google.com",
  "userAgent": "...",
  "ipHash": "hmac_sha256(...)",
  "timestamp": "2026-06-30T18:45:00Z"
}

23. Deduplication Strategy

Kafka provides at-least-once delivery. That means the same event can be delivered more than once.

The aggregator uses Valkey SETNX to deduplicate:

SETNX analytics:event:{eventId} 1 EX 86400

If the key is created, the event is processed. If the key already exists, the event is skipped.

24. Privacy Design

25. Rollup Table Design

The system stores aggregated data at multiple granularities:

5-minute rollup
1-day rollup
content-level rollup
geo-level rollup

Example table:

CREATE TABLE analytics_geo_time_rollups (
    bucket_start TIMESTAMP NOT NULL,
    granularity VARCHAR(16) NOT NULL,
    geo_cell VARCHAR(64) NOT NULL,
    page_views BIGINT NOT NULL,
    unique_visitors BIGINT NOT NULL,
    PRIMARY KEY (bucket_start, granularity, geo_cell)
);

26. Graceful Degradation

If Kafka is unavailable, the tracking endpoint can fall back to a direct database insert.

• Preferred path: Kafka ingestion.
• Fallback path: Direct database insert.
• Worst case: Analytics quality degrades, but portfolio browsing remains unaffected.

Deep Dive 5 — Notification Service

27. Problem

When a new project or blog is published, subscribers may need to receive an email notification.

The system must handle:

• Many subscribers.
• Provider failures.
• Duplicate Kafka events.
• Per-recipient retry.
• Exponential backoff.
• Open/read tracking.
• Failure isolation.

A single Kafka event should not directly send all emails synchronously. That would make the publish flow slow and fragile.

28. Notification Fan-Out

The Kafka consumer expands one content event into many recipient rows.

CONTENT_PUBLISHED event
        |
        v
Find matching subscribers
        |
        v
Insert notification_recipients rows
        |
        v
Scheduler claims due rows
        |
        v
Workers send emails

29. Email State Machine

30. Dispatch Table

CREATE TABLE notification_recipients (
    id UUID PRIMARY KEY,
    notification_id UUID NOT NULL,
    subscriber_email VARCHAR(255) NOT NULL,
    status VARCHAR(32) NOT NULL,
    retry_count INT NOT NULL DEFAULT 0,
    next_retry_at TIMESTAMP,
    provider_message_id VARCHAR(255),
    last_error TEXT,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

31. Batch Claiming

The scheduler claims rows using a database-level lock:

SELECT *
FROM notification_recipients
WHERE status IN ('PENDING', 'RETRY')
  AND next_retry_at <= NOW()
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

This allows multiple workers to run safely in parallel without sending the same email twice.

32. Backoff Policy

1st retry: 60 seconds
2nd retry: 120 seconds
3rd retry: 240 seconds
4th retry: 480 seconds
5th retry: 960 seconds
Then: FAILED

33. Why This Design Works

• Subscriber fan-out is separated from provider delivery.
• Each recipient has an independent state machine.
• Provider failures do not block content publishing.
• Retries are controlled by the database state, not in-memory worker state.
• Multiple workers can safely claim work with FOR UPDATE SKIP LOCKED.
• Failed rows remain inspectable and replayable.

Deep Dive 6 — Reliability and Failure Handling

34. End-to-End Delivery Semantics

The system intentionally uses at-least-once messaging with idempotent side effects.

This is more practical than trying to enforce global exactly-once behavior across databases, Kafka, APIs, and third-party providers.

35. Failure Scenarios

36. Dead Letter Queue Strategy

Each major Kafka consumer has a DLQ topic:

content.search.index.dlq.v1
content.rag.index.dlq.v1
content.notification.dlq.v1
analytics.raw.events.dlq.v1

A DLQ message includes the original topic, event ID, consumer name, failure reason, retry count, and payload.

{
  "originalTopic": "content.search.index.v1",
  "eventId": "018fa20b...",
  "consumer": "search-indexer",
  "failureReason": "OpenSearch mapping conflict",
  "retryCount": 5,
  "payload": {}
}

The DLQ allows failed events to be inspected, fixed, and replayed without blocking the entire pipeline.

Deep Dive 7 — Data Consistency Model

37. Source of Truth vs Derived Projections

The Admin Service database is the source of truth for portfolio content.

OpenSearch, pgvector, notification rows, and analytics rollups are derived projections.

This means they can be rebuilt from source data and Kafka events if needed.

38. Consistency Trade-Off

The platform accepts eventual consistency for derived systems.

• A blog may be published immediately.
• Search may update seconds later.
• RAG embeddings may update after indexing.
• Notifications may send asynchronously.

The source of truth remains consistent immediately. Derived projections become consistent asynchronously.

Deep Dive 8 — Deployment Architecture

39. Runtime Deployment

40. CI/CD and Database Migration

• Docker containers for backend services.
• GitHub Actions for CI/CD.
• Google Cloud Run for service deployment.
• Flyway for version-controlled database migrations.
• Environment-specific configuration for development and production.

Database migrations are treated as part of the application release process rather than manual production changes.

41. Technology Stack

42. Production Reliability Patterns

43. Engineering Outcomes

This platform demonstrates several production-level engineering capabilities:

• Designing event-driven systems with Kafka.
• Applying the Transactional Outbox pattern correctly.
• Building idempotent consumers and retryable pipelines.
• Separating source-of-truth data from derived projections.
• Implementing practical search using BM25, synonyms, and index-time expansion.
• Designing safer AI agents with MCP governance.
• Building real-time analytics with privacy-aware aggregation.
• Designing reliable email dispatch with state machines and backoff.
• Deploying independently scalable services using Cloud Run and managed infrastructure.

The result is a portfolio platform that functions as both a personal website and a living system design showcase.

It is intentionally built to reflect how production systems are designed: loosely coupled, observable, resilient to partial failure, and optimized around clear ownership boundaries.

44. Final Architecture Summary

At a high level, the system follows one core principle:

Keep the source of truth simple and reliable, then use events to build scalable, independent projections around it.

That principle makes the architecture easier to scale, easier to debug, and safer to evolve over time.