Multi-Tenant AI Assistant Platform (Dify RAG / LLM Chatbot)

Challenge

• Multiple departments wanted to use AI assistants, but a shared platform could not co-mingle each department's confidential information.
• The existing Dify implementation assumed single-tenant use and lacked sufficient permission and log isolation.
• An abstraction layer was needed to flexibly switch between LLM providers (Claude / OpenAI / Gemini).

Solution

• Embedded tenant IDs in JWTs and validated them at Dify API calls via both URL and header.
• Isolated the Pinecone vector space per tenant and physically prohibited cross-tenant search at the API layer.
• Implemented an LLM provider abstraction in FastAPI; controlled model switching via environment variables and feature flags.
• Distributed GKE Deployments per tenant group and applied resource limits at the tenant level.

Outcomes