Skip to content

[cozystack-controller] Implement cache for CozystackResourceDefinitions - #1427

Merged
Andrei Kvapil (kvaps) merged 1 commit into
mainfrom
crd-cache
Sep 17, 2025
Merged

[cozystack-controller] Implement cache for CozystackResourceDefinitions#1427
Andrei Kvapil (kvaps) merged 1 commit into
mainfrom
crd-cache

Conversation

@kvaps

@kvaps Andrei Kvapil (kvaps) commented Sep 17, 2025

Copy link
Copy Markdown
Member

Signed-off-by: Andrei Kvapil kvapss@gmail.com

What this PR does

This PR introduces shared cache for CozystackResourceDefinitions and warbs it up before making decidion on restart cozystack-api server.

Reastart logic was also updated to trigger restart only if consistent hash from the configuration has been changed.

Release note

[cozystack-controller] Implement cache for CozystackResourceDefinitions

Summary by CodeRabbit

  • New Features

    • Smarter, hash-based restarts for the API component, triggered only when configuration truly changes.
    • Debounced restart behavior to avoid rapid, repeated restarts during bursts of updates.
  • Performance

    • Introduces an internal in-memory configuration cache to speed up evaluations and reduce API calls.
    • Cache is primed at startup for faster, more responsive operations.
  • Bug Fixes

    • Eliminates unnecessary restarts when there are no effective config changes, improving stability and reducing disruption.

@coderabbitai

coderabbitai Bot commented Sep 17, 2025

Copy link
Copy Markdown
Contributor

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Introduces an in-memory cache for CozystackResourceDefinition objects and integrates it into the reconciler. Reconciliation now maintains the cache and triggers a debounced, hash-based Deployment patch when configuration changes are detected. Controller setup primes the cache and enqueues reconciles. Adds deterministic config-hash computation replacing restartedAt-based restart signaling.

Changes

Cohort / File(s) Summary
Reconciler updates and hash-based restart
internal/controller/cozystackresource_controller.go
Integrates crdmem.Memory for CRD upsert/delete; adds debouncedRestart using a stable config SHA-256 from a sorted snapshot; replaces restartedAt restarts with cozystack.io/config-hash annotation patch; updates controller setup to prime memory and enqueue reconcile for cozy-system/cozystack-api; adds helpers (crdHashView, computeConfigHash).
In-memory CRD cache
internal/shared/crdmem/memory.go
Adds thread-safe Memory cache (singleton + constructor) with Upsert, Delete, Snapshot, IsPrimed, EnsurePrimingWithManager, and ListFromCacheOrAPI. Implements manager-integrated priming runnable that syncs from API and marks cache as primed.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant K8s as Kubernetes API
    participant Mgr as Controller Manager
    participant Ctrl as CozystackResource Reconciler
    participant Mem as crdmem.Memory
    participant Dep as Deployment cozystack-api

    rect rgba(230,240,255,0.5)
    note over Mgr,Mem: Startup & Priming
    Mgr->>Mem: EnsurePrimingWithManager()
    Mgr-->>Mem: Start runnable after cache sync
    Mem->>K8s: List CozystackResourceDefinition
    Mem->>Mem: Upsert each CRD (mark primed)
    end

    rect rgba(240,255,240,0.5)
    note over Ctrl: Reconcile on CRD create/update/delete
    K8s-->>Ctrl: Event (CRD)
    alt Create/Update
      Ctrl->>Mem: Upsert(CRD)
    else NotFound
      Ctrl->>Mem: Delete(name)
    end
    end

    rect rgba(255,245,230,0.5)
    note over Ctrl,Dep: Debounced config-hash restart
    Ctrl->>Mem: Snapshot()
    Ctrl->>Ctrl: computeConfigHash(snapshot)
    Ctrl->>K8s: Get Deployment cozystack-api
    alt Hash changed
      Ctrl->>Dep: Patch annotation cozystack.io/config-hash=<new>
      Dep-->>Ctrl: Patched
    else No change
      Ctrl->>Ctrl: Mark lastHandled
    end
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • lllamnyp

Poem

A rabbit caches CRDs with care,
Hashing specs into autumn air.
No ticking clocks, no rushed restart—
Just quiet diffs and ordered art.
With primed memory and gentle pace,
Patch the Deployment, leave no trace.
Hop, hash, reconcile—ace! 🐇💾

✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch crd-cache

Tip

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

  • Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
  • Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Pre-merge checks

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title Check ✅ Passed The title "[cozystack-controller] Implement cache for CozystackResourceDefinitions" is a concise, single-sentence summary that correctly describes the primary change in this PR—adding an in-memory cache for CozystackResourceDefinition objects and related controller updates (see internal/shared/crdmem/memory.go and internal/controller/cozystackresource_controller.go), and it is specific enough for a reviewer scanning history to understand the main purpose.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello Andrei Kvapil (@kvaps), I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the cozystack-controller's efficiency and stability by introducing an intelligent caching and debouncing mechanism for CozystackResourceDefinition objects. The controller now maintains an in-memory snapshot of these definitions, using a computed hash to determine if a meaningful change has occurred before triggering a rollout of the cozystack-api deployment. This approach minimizes unnecessary restarts and optimizes resource utilization. Additionally, the LineageControllerWebhook has been decoupled from the main controller's initialization, and a new flag allows for configurable resource watching.

Highlights

  • CRD Caching Implementation: An in-memory cache (crdmem.Memory) has been introduced to efficiently store and manage CozystackResourceDefinition objects, reducing direct API server calls.
  • Optimized Deployment Rollouts: The CozystackResourceDefinitionReconciler now computes a SHA256 hash of the cached CRD configurations. The cozystack-api deployment is only restarted if this cozystack.io/config-hash annotation changes, preventing unnecessary rollouts.
  • Debounced Reconciliation: The reconciliation logic for CozystackResourceDefinition changes is now debounced, ensuring that multiple rapid updates to CRDs trigger a single, delayed restart of the cozystack-api deployment.
  • Lineage Controller Webhook Removal: The LineageControllerWebhook import and its setup logic have been removed from the main controller entry point, streamlining the controller's initialization.
  • Configurable Resource Watching: A new command-line flag --watch-resources has been added to the cozystack-controller, allowing operators to specify which Kubernetes resources the controller should monitor.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an intelligent caching mechanism for CozystackResourceDefinitions to prevent unnecessary restarts of the cozystack-api deployment. It achieves this by computing a configuration hash and only triggering a rollout when this hash changes. The implementation is solid and introduces a new crdmem package for managing the in-memory cache. Overall, this is a great improvement for system stability and efficiency. I have a few suggestions to enhance the robustness and maintainability of the new caching logic.

Comment on lines +52 to +54
for _, v := range m.data {
out = append(out, v)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The Snapshot function returns a slice of CozystackResourceDefinition structs by performing a shallow copy. Since the struct contains reference types like maps (e.g., Labels), consumers of the snapshot could inadvertently modify the cached data. This could lead to subtle concurrency issues. To ensure cache integrity and prevent such data corruption bugs, the function should return deep copies of the objects.

Suggested change
for _, v := range m.data {
out = append(out, v)
}
for _, v := range m.data {
out = append(out, *v.DeepCopy())
}

Comment on lines +53 to +55
r.mu.Lock()
r.lastEvent = time.Now()
r.mu.Unlock()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The lastEvent timestamp is updated here, but it's also updated in the Watches handler within SetupWithManager (lines 86-88). Since the Watches handler is responsible for enqueuing the request that leads to the debounced restart, it's the more appropriate place to manage this timestamp. The current duplication is redundant and could be confusing. I recommend removing this update to have a single, clear source of truth for the event time.

Comment on lines +76 to +84
if err := mgr.GetClient().List(ctx, &list); err == nil {
for i := range list.Items {
m.Upsert(&list.Items[i])
}
m.mu.Lock()
m.primed = true
m.mu.Unlock()
}
return nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In the EnsurePrimingWithManager runnable, a failure in mgr.GetClient().List() is silently ignored. This means cache priming could fail without any logs, making debugging difficult. The system would continue with an unprimed cache, which might eventually become consistent but could cause unexpected behavior initially. It's crucial to log this error to provide visibility into priming failures. Additionally, logging a success message upon completion would be beneficial for monitoring.

Here's a suggested implementation:

			var list cozyv1alpha1.CozystackResourceDefinitionList
			if err := mgr.GetClient().List(ctx, &list); err != nil {
				ctrl.Log.WithName("crdmem").Error(err, "failed to prime cache")
				return nil
			}

			for i := range list.Items {
				m.Upsert(&list.Items[i])
			}
			m.mu.Lock()
			m.primed = true
			m.mu.Unlock()
			ctrl.Log.WithName("crdmem").Info("cache primed successfully", "count", len(list.Items))
			return nil

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (4)
internal/shared/crdmem/memory.go (2)

68-88: Priming errors are silently ignored; add logs and success notice.

Lack of logging hides failures in cache priming and cache-sync issues.

Apply this diff:

 func (m *Memory) EnsurePrimingWithManager(mgr ctrl.Manager) error {
   var errOut error
   m.primeOnce.Do(func() {
     errOut = mgr.Add(runnable(func(ctx context.Context) error {
-      if ok := mgr.GetCache().WaitForCacheSync(ctx); !ok {
-        return nil
-      }
+      logger := ctrl.Log.WithName("crdmem")
+      if ok := mgr.GetCache().WaitForCacheSync(ctx); !ok {
+        logger.Error(nil, "cache sync failed; skipping priming")
+        return nil
+      }
       var list cozyv1alpha1.CozystackResourceDefinitionList
-      if err := mgr.GetClient().List(ctx, &list); err == nil {
-        for i := range list.Items {
-          m.Upsert(&list.Items[i])
-        }
-        m.mu.Lock()
-        m.primed = true
-        m.mu.Unlock()
-      }
+      if err := mgr.GetClient().List(ctx, &list); err != nil {
+        logger.Error(err, "failed to prime cache")
+        return nil
+      }
+      for i := range list.Items {
+        m.Upsert(&list.Items[i])
+      }
+      m.mu.Lock()
+      m.primed = true
+      m.mu.Unlock()
+      logger.Info("cache primed successfully", "count", len(list.Items))
       return nil
     }))
   })
   return errOut
 }

48-56: Snapshot must return deep copies to avoid cache corruption.

Shallow-copying structs that contain maps/slices shares backing storage with the cache; callers can mutate cached data.

Apply this diff:

 func (m *Memory) Snapshot() []cozyv1alpha1.CozystackResourceDefinition {
   m.mu.RLock()
   defer m.mu.RUnlock()
   out := make([]cozyv1alpha1.CozystackResourceDefinition, 0, len(m.data))
-  for _, v := range m.data {
-    out = append(out, v)
-  }
+  for _, v := range m.data {
+    out = append(out, *v.DeepCopy())
+  }
   return out
 }
internal/controller/cozystackresource_controller.go (2)

53-55: Duplicate lastEvent updates; keep it in the watch map-func only.

One source of truth avoids confusion.

Apply the deletions already shown in the previous diff (lines 53-55).

Also applies to: 86-88


46-67: Reconcile mixes CRD handling with Deployment-trigger requests; reorder to avoid false deletes.

When the mapped request for cozy-system/cozystack-api arrives, the code tries to Get a CRD named "cozystack-api" and may Delete it from memory on NotFound before calling debouncedRestart. Handle the special request first.

Apply this diff:

 func (r *CozystackResourceDefinitionReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
   logger := log.FromContext(ctx)
 
+  // Handle the synthetic request that triggers the debounced restart first.
+  if req.Namespace == "cozy-system" && req.Name == "cozystack-api" {
+    return r.debouncedRestart(ctx, logger)
+  }
+
   crd := &cozyv1alpha1.CozystackResourceDefinition{}
   err := r.Get(ctx, types.NamespacedName{Name: req.Name}, crd)
   if err == nil {
     if r.mem != nil {
       r.mem.Upsert(crd)
     }
-    r.mu.Lock()
-    r.lastEvent = time.Now()
-    r.mu.Unlock()
     return ctrl.Result{}, nil
   }
-  if err != nil && !apierrors.IsNotFound(err) {
-    return ctrl.Result{}, err
-  }
-  if apierrors.IsNotFound(err) && r.mem != nil {
-    r.mem.Delete(req.Name)
-  }
-  if req.Namespace == "cozy-system" && req.Name == "cozystack-api" {
-    return r.debouncedRestart(ctx, logger)
-  }
-  return ctrl.Result{}, nil
+  if apierrors.IsNotFound(err) {
+    if r.mem != nil {
+      r.mem.Delete(req.Name)
+    }
+    return ctrl.Result{}, nil
+  }
+  return ctrl.Result{}, err
 }
🧹 Nitpick comments (5)
internal/shared/crdmem/memory.go (1)

90-99: Optional: deep-copy API results for consistency.

ListFromCacheOrAPI returns live API objects by reference semantics; consider deep-copying for parity with Snapshot.

internal/controller/cozystackresource_controller.go (4)

126-151: Gate restarts until cache is primed to avoid hashing partial state.

Prevents early no-op patches and flapping before priming completes.

Apply this diff:

 func (r *CozystackResourceDefinitionReconciler) debouncedRestart(ctx context.Context, logger logr.Logger) (ctrl.Result, error) {
   r.mu.Lock()
   le := r.lastEvent
   lh := r.lastHandled
   debounce := r.Debounce
   r.mu.Unlock()
 
   if debounce <= 0 {
     debounce = 5 * time.Second
   }
   if le.IsZero() {
     return ctrl.Result{}, nil
   }
   if d := time.Since(le); d < debounce {
     return ctrl.Result{RequeueAfter: debounce - d}, nil
   }
   if !lh.Before(le) {
     return ctrl.Result{}, nil
   }
 
+  if r.mem == nil || !r.mem.IsPrimed() {
+    return ctrl.Result{RequeueAfter: 2 * time.Second}, nil
+  }
+
   newHash, err := r.computeConfigHash()
   if err != nil {
     return ctrl.Result{}, err
   }

161-167: Skip patch when hashes match, even if empty, to avoid useless API calls.

This avoids no-op patches when both sides are empty.

Apply this diff:

-  if oldHash == newHash && oldHash != "" {
+  if oldHash == newHash {
     r.mu.Lock()
     r.lastHandled = le
     r.mu.Unlock()
     logger.Info("No changes in CRD config; skipping restart", "hash", newHash)
     return ctrl.Result{}, nil
   }

82-83: Remove no-op WithPredicates() call.

Empty predicate list is redundant.

Apply this diff:

-    .For(&cozyv1alpha1.CozystackResourceDefinition{}, builder.WithPredicates()).
+    .For(&cozyv1alpha1.CozystackResourceDefinition{}).

156-172: Optional: extract annotation key into a constant.

Reduces typos and eases reuse.

Suggested snippet:

const configHashAnnotation = "cozystack.io/config-hash"

Then use configHashAnnotation at reads/writes.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 37720b9 and cdf53e8.

📒 Files selected for processing (2)
  • internal/controller/cozystackresource_controller.go (3 hunks)
  • internal/shared/crdmem/memory.go (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
internal/shared/crdmem/memory.go (1)
api/v1alpha1/cozystackresourcedefinitions_types.go (2)
  • CozystackResourceDefinition (26-31)
  • CozystackResourceDefinitionList (36-40)
internal/controller/cozystackresource_controller.go (2)
internal/shared/crdmem/memory.go (2)
  • Memory (12-17)
  • Global (28-31)
api/v1alpha1/cozystackresourcedefinitions_types.go (2)
  • CozystackResourceDefinition (26-31)
  • CozystackResourceDefinitionSpec (46-51)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Build
🔇 Additional comments (1)
internal/shared/crdmem/memory.go (1)

33-40: Good: defensive Upsert with DeepCopy.

Storing a deep copy on write protects the cache from external mutation. LGTM.

Comment on lines +5 to 9
"crypto/sha256"
"encoding/hex"
"encoding/json"
"sort"
"sync"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Use k8s DeepHashObject for stable, order-insensitive hashing; JSON map order is non-deterministic.

JSON encoding of maps can reorder keys, causing spurious rollouts. Prefer DeepHashObject with sha256.

Apply this diff:

 import (
   "context"
-  "crypto/sha256"
-  "encoding/hex"
-  "encoding/json"
+  "crypto/sha256"
+  "encoding/hex"
   "sort"
   "sync"
   "time"
@@
   appsv1 "k8s.io/api/apps/v1"
   apierrors "k8s.io/apimachinery/pkg/api/errors"
   "k8s.io/apimachinery/pkg/runtime"
   "k8s.io/apimachinery/pkg/types"
+  k8shash "k8s.io/apimachinery/pkg/util/hash"
@@
 func (r *CozystackResourceDefinitionReconciler) computeConfigHash() (string, error) {
   if r.mem == nil {
     return "", nil
   }
   snapshot := r.mem.Snapshot()
   sort.Slice(snapshot, func(i, j int) bool { return snapshot[i].Name < snapshot[j].Name })
   views := make([]crdHashView, 0, len(snapshot))
   for i := range snapshot {
     views = append(views, crdHashView{
       Name: snapshot[i].Name,
       Spec: snapshot[i].Spec,
     })
   }
-  b, err := json.Marshal(views)
-  if err != nil {
-    return "", err
-  }
-  sum := sha256.Sum256(b)
-  return hex.EncodeToString(sum[:]), nil
+  h := sha256.New()
+  k8shash.DeepHashObject(h, views)
+  return hex.EncodeToString(h.Sum(nil)), nil
 }

Also applies to: 18-21, 105-124

🤖 Prompt for AI Agents
internal/controller/cozystackresource_controller.go lines 5-9 (and similarly
18-21, 105-124): current code uses json.Marshal (and sorting) to hash objects
which is order-sensitive for maps; replace that logic to use
k8s.io/apimachinery/pkg/util/hash.DeepHashObject written into a sha256 hasher
and then hex-encode the sum. Update imports to remove encoding/json and sort
(and any unused packages) and add "k8s.io/apimachinery/pkg/util/hash". For each
hashing helper: create a sha256.New(), call hash.DeepHashObject(hasher, obj),
compute hasher.Sum(nil), and hex.EncodeToString(...) for the stable,
order-insensitive hash; apply this change in all specified ranges.

@kvaps
Andrei Kvapil (kvaps) merged commit d10b363 into main Sep 17, 2025
20 checks passed
@kvaps
Andrei Kvapil (kvaps) deleted the crd-cache branch September 17, 2025 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant