Skip to content

feat(monitoring): switch VLogs to cluster mode for secure tenant log isolation #2000

Description

Problem

PR #1971 proposes adding NetworkPolicy egress to VLogs pods so that nested cluster Fluent Bit instances can forward logs to the parent tenant's VLogs. However, VLogs in single-node mode exposes both write (/insert/*) and read (/select/*) endpoints on the same port (9428). If we open NetworkPolicy egress to VLogs, a child tenant would be able to query /select/* and read ALL logs stored in the parent tenant's VLogs — including logs from other tenants.

This is why PR #1971 cannot be merged as-is (see kvaps' comment about the security concern).

Note on metrics: vminsert is architecturally write-only — it has no query endpoints. The existing NetworkPolicy egress to vminsert is secure. This issue is VLogs-specific.

Current State

  • VLogs is deployed via the deprecated VLogs CRD (operator.victoriametrics.com/v1beta1) in single-node mode
  • victoria-metrics-operator is at v0.55.0
  • VictoriaLogs image is v1.17.0
  • The VLogs CRD will become read-only after operator v0.61.0

Proposed Solution

Switch VLogs from single-node to cluster mode using the VLCluster CRD (introduced in operator v0.59.0):

  • vlinsert — write-only component, accepts logs from tenants
  • vlselect — read-only component, handles log queries
  • vlstorage — storage backend

With this architecture, tenant NetworkPolicy egress would target only vlinsert pods (write-only by design, with -select.disable flag). Tenants can ingest logs but cannot read them. This mirrors the existing vminsert/vmselect separation for metrics.

Prerequisites

  1. Upgrade victoria-metrics-operator from v0.55.0 to v0.59.0+ (current stable: v0.67.0)
  2. Migrate from deprecated VLogs CRD (v1beta1) to VLCluster CRD (v1)
  3. Update VictoriaLogs image to a cluster-capable version

This also addresses the upcoming VLogs CRD deprecation.

Alternative

Deploy vmauth (VictoriaMetrics auth proxy) in front of single-node VLogs, routing only /insert/* paths to tenants. Simpler but adds an extra component without solving the CRD deprecation.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stalesecuritySecurity-related issues and featurestriage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions