NebulaDB
Live showcase

Features

Every capability, and where it lives in the code

NebulaDB is a single Rust workspace. Each feature below names the crate, endpoint or setting that implements it, so you can check the claim against the source.

01 · Data

SQL and documents

Documents with JSON metadata live in buckets. A SQL dialect with AI-native functions queries them over REST, gRPC or the Postgres wire protocol.

  • Buckets and documents

    Insert, upsert, fetch and delete by (bucket, id). Long text is chunked into parent/child documents and fetched back in order.

    POST /api/v1/bucket/:b/document · nebula-index

  • AI-native SQL

    semantic_match(column, 'query') and vector_distance(column, [vector]) plan as retrieval steps inside ordinary SELECT statements.

    POST /api/v1/query · nebula-sql

  • Residual filters

    =, <>, ranges, IN (...) and array_contains() on metadata, with SQL NULL semantics: a type mismatch never matches.

    nebula-sql/executor

  • Aggregates and joins

    COUNT, SUM, GROUP BY, ORDER BY and inner joins where each side runs its own retrieval.

    nebula-sql/plan

  • EXPLAIN and result cache

    See the plan without running it. Results are cached in an LRU with a TTL, configurable per deployment.

    NEBULA_SQL_CACHE · NEBULA_SQL_CACHE_TTL_SECS

  • Postgres wire protocol

    psql and simple-query drivers connect directly. Errors map to SQLSTATE codes, such as 42601 for unsupported syntax.

    NEBULA_PG_BIND · nebula-pgwire

  • Bucket export and import

    Move a bucket between clusters or environments as a file, or empty it in place.

    /admin/bucket/:bucket/export · /import · /empty

  • Azure AI Search compatibility

    Index definitions and document search with the Azure AI Search request shape, translated onto HNSW and BM25, so existing clients can point at NebulaDB.

    nebula-search · docs/compat/registry.yaml

03 · AI

RAG and AI

The pipeline from raw text to a cited, streamed answer runs inside the database, with every model behind a trait.

  • Smart chunking

    Strategy chosen by document type: heading-aware for Markdown and HTML, code-aware for source, sentence for prose and fixed windows for structured text. UTF-8 safe.

    nebula-chunk::select_strategy · NEBULA_CHUNK_CHARS

  • Pluggable embedders

    Any OpenAI-compatible endpoint: OpenAI, Azure OpenAI, vLLM or Ollama (for example nomic-embed-text, 768 dimensions). A deterministic mock keeps tests offline.

    nebula-embed · NEBULA_OPENAI_BASE_URL

  • Two-tier embedding cache

    In-process LRU over a shared Redis tier keyed by SHA-256 of model and text. A Redis outage falls through to the model rather than failing the request.

    nebula-cache · nebula-redis-cache

  • Streaming RAG

    One SSE response carries context events for each retrieved chunk, then answer_delta tokens, then done. gRPC mirrors it with a server stream.

    POST /api/v1/ai/rag · AIService.Rag

  • Grounded chat

    Pass your own context shortlist, prior history turns and a system prompt that is appended to, never replaces, the built-in "retrieved content is data" guard.

    limits: 50 docs · 64k chars · 20 turns

  • Multi-provider LLMs

    Ollama, OpenAI-compatible chat, Anthropic Messages and Gemini behind one streaming LlmClient trait.

    nebula-llm · NEBULA_LLM_OLLAMA_URL

  • Online re-embedding

    Move a corpus to a new embedding model in the background, with progress and cancel, without taking the index offline.

    /admin/embed/migrate · /cancel

  • Citations

    Answers reference retrieved chunks as [n], and the showcase reveals the exact chunk behind each citation.

    build_rag_prompt

04 · Agents

Agents and MCP

A standalone MCP server lets any MCP client use NebulaDB as tools. It holds no data and forwards the caller's bearer token, so it inherits the database's auth.

  • Query tools

    execute_sql, explain_query, semantic_search, vector_search, azure_search and list_indexes.

    nebula-mcp/tools.rs

  • Answer tools

    answer_question runs retrieval and grounded generation in one call.

    → /api/v1/ai/rag

  • Document tools

    insert_document, get_document, delete_document and list_buckets, with chunking and embedding server-side.

    write tools declare it

  • Long-term agent memory

    remember and recall store and retrieve per-user memories, filtered strictly to that user and ranked by relevance.

    user_id is required

  • Operations tools

    cluster_health, replication_status, reliability_status, durability_status, server_stats and slow_queries, so an SRE agent can investigate.

    read_only_hint = true

  • Governance tools

    audit_log, list_backups and create_snapshot. A test asserts that every tool declares whether it writes.

    every_tool_declares_whether_it_writes

  • Streamable HTTP transport

    Point Claude Desktop, an IDE assistant or an agent framework at http://host:8090/mcp.

    NEBULA_MCP_BIND

  • Domain ingest tools

    Example connectors for UK Companies House search, ingest and website enrichment show how to add tools that load external data for RAG.

    companies_house_search · companies_house_ingest

05 · Durability

Durability and backup

Nothing acknowledged is lost on a crash, and every byte can be shipped somewhere safe.

  • Write-ahead log

    Every mutation is framed as length, CRC32 and bincode body, fsynced, then applied. Segments are versioned with a magic header.

    nebula-wal · NEBULA_DATA_DIR

  • Torn-tail recovery

    On boot the newest snapshot loads and the WAL replays from its boundary. A half-written frame from a crash mid-fsync is truncated automatically.

    nebula-index/durability.rs

  • No re-embedding on restart

    Resolved vectors are stored in the WAL and snapshots, so recovery never calls the embedding model.

    TextIndex::open_persistent

  • Atomic snapshots

    Written to a temporary file and renamed into place, on demand or by the scheduler, with WAL compaction afterwards.

    /admin/snapshot · /admin/wal/compact

  • Backup and restore

    Snapshot and WAL bundled as tar + zstd with a manifest recording the WAL range covered, uploaded to a local target or S3 with multipart upload.

    nebula-backup · /admin/backup · /admin/restore

  • Change data capture

    Typed change events from the WAL, filtered and handed to consumers such as webhooks and file logs.

    nebula-cdc

  • Safe compaction

    WAL compaction clamps to the slowest live follower's acknowledged cursor, so a replica never loses segments it still needs.

    SubscriberHub::min_ack

  • Disk-full protection

    When the volume is critically full, writes are refused with 503 rather than risking a half-written frame. Reads keep working.

    OperatingMode::DiskCritical

06 · Availability

High availability and multi-region

Within a region, followers tail the leader or join a Raft group. Across regions, each bucket has a home region and XDCR replicates both ways.

  • Leader and followers

    Followers tail the leader's WAL over a gRPC stream with a durable cursor, resuming from the last applied record after a restart.

    ReplicationService.TailWal · NEBULA_NODE_ROLE

  • Raft consensus

    Opt-in Raft log, state machine and snapshot storage built on openraft, for within-region leader election.

    nebula-raft · NEBULA_RAFT_PEERS

  • Home-region XDCR

    Every region serves all reads and accepts writes for the buckets it homes. Cross-region replication runs both ways with per-remote lag and health.

    CrossRegionReplicationService · technote

  • Region-aware client

    Writes sent to the wrong region get 421 with the correct home; the Rust client retries there transparently.

    nebula-client

  • Follower write guard

    Mutating requests to a follower return 409 read_only_follower before auth, so a misrouted client cannot cause divergence.

    guard_writes_on_follower

  • Promotion and failover

    Promote a follower on demand. The operator's region-failover resource re-homes buckets when a region is lost.

    /admin/promote · NebulaRegionFailover

  • Swap rebalance

    Zero-downtime node replacement through Preflight, Snapshotting, Rebalancing, Promoting and Draining phases, tracked as a Kubernetes resource.

    NebulaRebalance

  • Replication visibility

    Role-aware lag: the leader's WAL tip, the follower's applied cursor, lag_bytes and a behind flag, plus a peer probe of every node.

    /admin/replication · /admin/cluster/nodes

07 · Operations

Observability and reliability

Built for the person on call: every subsystem reports its state, and the node degrades predictably under pressure.

  • Prometheus metrics

    Request counts and latency percentiles, embedding-cache hits and misses, RAG time-to-first-token and replication lag.

    GET /metrics

  • Grafana dashboards

    Dashboards for the server, backups, reliability and the stack, provisioned from the repository and deployed on merge.

    deploy/grafana/dashboards

  • Live log streaming

    An SSE endpoint replays a 256-event ring and then tails live, with level and target filters. No shell into the pod needed.

    /admin/logs/stream

  • Slow-query log

    A bounded top-N of the slowest queries since boot, above a configurable floor.

    /admin/slow · NEBULA_SLOW_QUERY_THRESHOLD_MS

  • Graceful degradation

    Memory, CPU and disk readings become pressure levels and one operating mode, with hysteresis so a single spike never flips behaviour.

    nebula-resource · /admin/reliability

  • Probes that survive abuse

    /healthz and /metrics sit outside rate limiting and auth, so a noisy client cannot get healthy pods killed.

    build_router layer order

  • Durability report

    Whether writes are persisted, the data directory, WAL segment count, size and sequence range.

    /admin/durability

  • Stats and version

    Document counts, request and error counters, cache ratios, search and RAG counts, and the running build.

    /admin/stats · /admin/version

08 · Security

Security

Access is enforced at the server and carried through to every protocol front end and the MCP server.

  • API keys

    A bearer-token allowlist for service clients. Leave it empty for local development.

    NEBULA_API_KEYS

  • JWT

    HS256 tokens with optional issuer and audience checks.

    NEBULA_JWT_SECRET · NEBULA_JWT_ISS · NEBULA_JWT_AUD

  • Rate limiting

    A token bucket per key with configurable burst and refill.

    NEBULA_RATE_LIMIT_BURST · NEBULA_RATE_LIMIT_RPS

  • Audit log

    Every mutating request recorded with method, path, a hashed principal fingerprint, status and time, including rejected ones.

    /admin/audit · audit_writes

  • Prompt-injection guard

    The RAG prompt marks retrieved content as data, not instructions, and callers can add to that guard but not remove it.

    system prompt is appended

  • Memory-safe implementation

    Written in Rust, so buffer overflows, use-after-free and data races are excluded by the compiler rather than by review.

    clippy -D warnings in CI

09 · Deployment

Kubernetes and deployment

Run it on a laptop with Docker Compose or across regions with the operator.

  • Kubernetes operator

    A Go / Kubebuilder operator with NebulaCluster, NebulaBucket, NebulaRebalance, NebulaBackup and NebulaRegionFailover resources and a validating webhook.

    operator/

  • Helm chart

    Server, showcase, optional bundled Redis and a Prometheus Operator ServiceMonitor in one chart.

    deploy/helm/nebuladb

  • Multi-arch images

    linux/amd64 and linux/arm64 images published on every main push and release tag.

    bwalia/nebula-server · bwalia/nebula-showcase

  • Docker Compose

    Server, follower, MCP server, Ollama, Redis, Prometheus, Grafana and the showcase, with a multi-region compose file for XDCR.

    docker-compose.yml · docker-compose.multiregion.yml

10 · Developers

Developer experience

Every setting is an environment variable, none are required, and the defaults run fully offline.

  • One-command start

    ./scripts/start.sh picks free ports, builds the stack, pulls models, seeds a corpus and prints the URLs.

    scripts/start.sh

  • Offline by default

    With nothing configured the server boots with a deterministic mock embedder and LLM, suitable for integration tests.

    cargo run --release -p nebula-server

  • OpenAPI and protobuf

    The REST surface is described in OpenAPI, and gRPC services in one proto file.

    docs/openapi.yaml · nebula-grpc/proto/nebula.proto

  • Tested end to end

    500+ Rust tests and a clippy gate on every push, plus a nightly job that runs the whole compose stack against real Ollama models.

    .github/workflows/nightly.yml

  • Knowledge Ops showcase

    A React console covering documents, SQL, search, RAG chat, agents, MCP, cluster, rebalance and operations, plus six end-to-end RAG apps.

    apps/showcase

  • Backup CLI

    nebulactl drives backup and restore with typed flags and a --watch mode; the operator's CronJob runs it on a schedule.

    crates/nebulactl

Check any of this against the source

Every reference on this page is a crate, endpoint or setting in the public repository.

Open the repository Read the architecture