[cozystack-controller] Implement cache for CozystackResourceDefinitions - #1427
Conversation
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughIntroduces an in-memory cache for CozystackResourceDefinition objects and integrates it into the reconciler. Reconciliation now maintains the cache and triggers a debounced, hash-based Deployment patch when configuration changes are detected. Controller setup primes the cache and enqueues reconciles. Adds deterministic config-hash computation replacing restartedAt-based restart signaling. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant K8s as Kubernetes API
participant Mgr as Controller Manager
participant Ctrl as CozystackResource Reconciler
participant Mem as crdmem.Memory
participant Dep as Deployment cozystack-api
rect rgba(230,240,255,0.5)
note over Mgr,Mem: Startup & Priming
Mgr->>Mem: EnsurePrimingWithManager()
Mgr-->>Mem: Start runnable after cache sync
Mem->>K8s: List CozystackResourceDefinition
Mem->>Mem: Upsert each CRD (mark primed)
end
rect rgba(240,255,240,0.5)
note over Ctrl: Reconcile on CRD create/update/delete
K8s-->>Ctrl: Event (CRD)
alt Create/Update
Ctrl->>Mem: Upsert(CRD)
else NotFound
Ctrl->>Mem: Delete(name)
end
end
rect rgba(255,245,230,0.5)
note over Ctrl,Dep: Debounced config-hash restart
Ctrl->>Mem: Snapshot()
Ctrl->>Ctrl: computeConfigHash(snapshot)
Ctrl->>K8s: Get Deployment cozystack-api
alt Hash changed
Ctrl->>Dep: Patch annotation cozystack.io/config-hash=<new>
Dep-->>Ctrl: Patched
else No change
Ctrl->>Ctrl: Mark lastHandled
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing touches
🧪 Generate unit tests
Tip 👮 Agentic pre-merge checks are now available in preview!Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.
Please see the documentation for more information. Example: reviews:
pre_merge_checks:
custom_checks:
- name: "Undocumented Breaking Changes"
mode: "warning"
instructions: |
Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).Please share your feedback with us on this Discord post. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Pre-merge checks❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
|
There was a problem hiding this comment.
Summary of Changes
Hello Andrei Kvapil (@kvaps), I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the cozystack-controller's efficiency and stability by introducing an intelligent caching and debouncing mechanism for CozystackResourceDefinition objects. The controller now maintains an in-memory snapshot of these definitions, using a computed hash to determine if a meaningful change has occurred before triggering a rollout of the cozystack-api deployment. This approach minimizes unnecessary restarts and optimizes resource utilization. Additionally, the LineageControllerWebhook has been decoupled from the main controller's initialization, and a new flag allows for configurable resource watching.
Highlights
- CRD Caching Implementation: An in-memory cache (
crdmem.Memory) has been introduced to efficiently store and manageCozystackResourceDefinitionobjects, reducing direct API server calls. - Optimized Deployment Rollouts: The
CozystackResourceDefinitionReconcilernow computes a SHA256 hash of the cached CRD configurations. Thecozystack-apideployment is only restarted if thiscozystack.io/config-hashannotation changes, preventing unnecessary rollouts. - Debounced Reconciliation: The reconciliation logic for
CozystackResourceDefinitionchanges is now debounced, ensuring that multiple rapid updates to CRDs trigger a single, delayed restart of thecozystack-apideployment. - Lineage Controller Webhook Removal: The
LineageControllerWebhookimport and its setup logic have been removed from the main controller entry point, streamlining the controller's initialization. - Configurable Resource Watching: A new command-line flag
--watch-resourceshas been added to thecozystack-controller, allowing operators to specify which Kubernetes resources the controller should monitor.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | Gemini (@gemini-code-assist) | Responds in comments when explicitly tagged, both in pull request comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
f85137b to
56bfc83
Compare
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
56bfc83 to
cdf53e8
Compare
There was a problem hiding this comment.
Code Review
This pull request introduces an intelligent caching mechanism for CozystackResourceDefinitions to prevent unnecessary restarts of the cozystack-api deployment. It achieves this by computing a configuration hash and only triggering a rollout when this hash changes. The implementation is solid and introduces a new crdmem package for managing the in-memory cache. Overall, this is a great improvement for system stability and efficiency. I have a few suggestions to enhance the robustness and maintainability of the new caching logic.
| for _, v := range m.data { | ||
| out = append(out, v) | ||
| } |
There was a problem hiding this comment.
The Snapshot function returns a slice of CozystackResourceDefinition structs by performing a shallow copy. Since the struct contains reference types like maps (e.g., Labels), consumers of the snapshot could inadvertently modify the cached data. This could lead to subtle concurrency issues. To ensure cache integrity and prevent such data corruption bugs, the function should return deep copies of the objects.
| for _, v := range m.data { | |
| out = append(out, v) | |
| } | |
| for _, v := range m.data { | |
| out = append(out, *v.DeepCopy()) | |
| } |
| r.mu.Lock() | ||
| r.lastEvent = time.Now() | ||
| r.mu.Unlock() |
There was a problem hiding this comment.
The lastEvent timestamp is updated here, but it's also updated in the Watches handler within SetupWithManager (lines 86-88). Since the Watches handler is responsible for enqueuing the request that leads to the debounced restart, it's the more appropriate place to manage this timestamp. The current duplication is redundant and could be confusing. I recommend removing this update to have a single, clear source of truth for the event time.
| if err := mgr.GetClient().List(ctx, &list); err == nil { | ||
| for i := range list.Items { | ||
| m.Upsert(&list.Items[i]) | ||
| } | ||
| m.mu.Lock() | ||
| m.primed = true | ||
| m.mu.Unlock() | ||
| } | ||
| return nil |
There was a problem hiding this comment.
In the EnsurePrimingWithManager runnable, a failure in mgr.GetClient().List() is silently ignored. This means cache priming could fail without any logs, making debugging difficult. The system would continue with an unprimed cache, which might eventually become consistent but could cause unexpected behavior initially. It's crucial to log this error to provide visibility into priming failures. Additionally, logging a success message upon completion would be beneficial for monitoring.
Here's a suggested implementation:
var list cozyv1alpha1.CozystackResourceDefinitionList
if err := mgr.GetClient().List(ctx, &list); err != nil {
ctrl.Log.WithName("crdmem").Error(err, "failed to prime cache")
return nil
}
for i := range list.Items {
m.Upsert(&list.Items[i])
}
m.mu.Lock()
m.primed = true
m.mu.Unlock()
ctrl.Log.WithName("crdmem").Info("cache primed successfully", "count", len(list.Items))
return nilThere was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
internal/shared/crdmem/memory.go (2)
68-88: Priming errors are silently ignored; add logs and success notice.Lack of logging hides failures in cache priming and cache-sync issues.
Apply this diff:
func (m *Memory) EnsurePrimingWithManager(mgr ctrl.Manager) error { var errOut error m.primeOnce.Do(func() { errOut = mgr.Add(runnable(func(ctx context.Context) error { - if ok := mgr.GetCache().WaitForCacheSync(ctx); !ok { - return nil - } + logger := ctrl.Log.WithName("crdmem") + if ok := mgr.GetCache().WaitForCacheSync(ctx); !ok { + logger.Error(nil, "cache sync failed; skipping priming") + return nil + } var list cozyv1alpha1.CozystackResourceDefinitionList - if err := mgr.GetClient().List(ctx, &list); err == nil { - for i := range list.Items { - m.Upsert(&list.Items[i]) - } - m.mu.Lock() - m.primed = true - m.mu.Unlock() - } + if err := mgr.GetClient().List(ctx, &list); err != nil { + logger.Error(err, "failed to prime cache") + return nil + } + for i := range list.Items { + m.Upsert(&list.Items[i]) + } + m.mu.Lock() + m.primed = true + m.mu.Unlock() + logger.Info("cache primed successfully", "count", len(list.Items)) return nil })) }) return errOut }
48-56: Snapshot must return deep copies to avoid cache corruption.Shallow-copying structs that contain maps/slices shares backing storage with the cache; callers can mutate cached data.
Apply this diff:
func (m *Memory) Snapshot() []cozyv1alpha1.CozystackResourceDefinition { m.mu.RLock() defer m.mu.RUnlock() out := make([]cozyv1alpha1.CozystackResourceDefinition, 0, len(m.data)) - for _, v := range m.data { - out = append(out, v) - } + for _, v := range m.data { + out = append(out, *v.DeepCopy()) + } return out }internal/controller/cozystackresource_controller.go (2)
53-55: Duplicate lastEvent updates; keep it in the watch map-func only.One source of truth avoids confusion.
Apply the deletions already shown in the previous diff (lines 53-55).
Also applies to: 86-88
46-67: Reconcile mixes CRD handling with Deployment-trigger requests; reorder to avoid false deletes.When the mapped request for cozy-system/cozystack-api arrives, the code tries to Get a CRD named "cozystack-api" and may Delete it from memory on NotFound before calling debouncedRestart. Handle the special request first.
Apply this diff:
func (r *CozystackResourceDefinitionReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { logger := log.FromContext(ctx) + // Handle the synthetic request that triggers the debounced restart first. + if req.Namespace == "cozy-system" && req.Name == "cozystack-api" { + return r.debouncedRestart(ctx, logger) + } + crd := &cozyv1alpha1.CozystackResourceDefinition{} err := r.Get(ctx, types.NamespacedName{Name: req.Name}, crd) if err == nil { if r.mem != nil { r.mem.Upsert(crd) } - r.mu.Lock() - r.lastEvent = time.Now() - r.mu.Unlock() return ctrl.Result{}, nil } - if err != nil && !apierrors.IsNotFound(err) { - return ctrl.Result{}, err - } - if apierrors.IsNotFound(err) && r.mem != nil { - r.mem.Delete(req.Name) - } - if req.Namespace == "cozy-system" && req.Name == "cozystack-api" { - return r.debouncedRestart(ctx, logger) - } - return ctrl.Result{}, nil + if apierrors.IsNotFound(err) { + if r.mem != nil { + r.mem.Delete(req.Name) + } + return ctrl.Result{}, nil + } + return ctrl.Result{}, err }
🧹 Nitpick comments (5)
internal/shared/crdmem/memory.go (1)
90-99: Optional: deep-copy API results for consistency.ListFromCacheOrAPI returns live API objects by reference semantics; consider deep-copying for parity with Snapshot.
internal/controller/cozystackresource_controller.go (4)
126-151: Gate restarts until cache is primed to avoid hashing partial state.Prevents early no-op patches and flapping before priming completes.
Apply this diff:
func (r *CozystackResourceDefinitionReconciler) debouncedRestart(ctx context.Context, logger logr.Logger) (ctrl.Result, error) { r.mu.Lock() le := r.lastEvent lh := r.lastHandled debounce := r.Debounce r.mu.Unlock() if debounce <= 0 { debounce = 5 * time.Second } if le.IsZero() { return ctrl.Result{}, nil } if d := time.Since(le); d < debounce { return ctrl.Result{RequeueAfter: debounce - d}, nil } if !lh.Before(le) { return ctrl.Result{}, nil } + if r.mem == nil || !r.mem.IsPrimed() { + return ctrl.Result{RequeueAfter: 2 * time.Second}, nil + } + newHash, err := r.computeConfigHash() if err != nil { return ctrl.Result{}, err }
161-167: Skip patch when hashes match, even if empty, to avoid useless API calls.This avoids no-op patches when both sides are empty.
Apply this diff:
- if oldHash == newHash && oldHash != "" { + if oldHash == newHash { r.mu.Lock() r.lastHandled = le r.mu.Unlock() logger.Info("No changes in CRD config; skipping restart", "hash", newHash) return ctrl.Result{}, nil }
82-83: Remove no-op WithPredicates() call.Empty predicate list is redundant.
Apply this diff:
- .For(&cozyv1alpha1.CozystackResourceDefinition{}, builder.WithPredicates()). + .For(&cozyv1alpha1.CozystackResourceDefinition{}).
156-172: Optional: extract annotation key into a constant.Reduces typos and eases reuse.
Suggested snippet:
const configHashAnnotation = "cozystack.io/config-hash"Then use configHashAnnotation at reads/writes.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
internal/controller/cozystackresource_controller.go(3 hunks)internal/shared/crdmem/memory.go(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
internal/shared/crdmem/memory.go (1)
api/v1alpha1/cozystackresourcedefinitions_types.go (2)
CozystackResourceDefinition(26-31)CozystackResourceDefinitionList(36-40)
internal/controller/cozystackresource_controller.go (2)
internal/shared/crdmem/memory.go (2)
Memory(12-17)Global(28-31)api/v1alpha1/cozystackresourcedefinitions_types.go (2)
CozystackResourceDefinition(26-31)CozystackResourceDefinitionSpec(46-51)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Build
🔇 Additional comments (1)
internal/shared/crdmem/memory.go (1)
33-40: Good: defensive Upsert with DeepCopy.Storing a deep copy on write protects the cache from external mutation. LGTM.
| "crypto/sha256" | ||
| "encoding/hex" | ||
| "encoding/json" | ||
| "sort" | ||
| "sync" |
There was a problem hiding this comment.
Use k8s DeepHashObject for stable, order-insensitive hashing; JSON map order is non-deterministic.
JSON encoding of maps can reorder keys, causing spurious rollouts. Prefer DeepHashObject with sha256.
Apply this diff:
import (
"context"
- "crypto/sha256"
- "encoding/hex"
- "encoding/json"
+ "crypto/sha256"
+ "encoding/hex"
"sort"
"sync"
"time"
@@
appsv1 "k8s.io/api/apps/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
+ k8shash "k8s.io/apimachinery/pkg/util/hash"
@@
func (r *CozystackResourceDefinitionReconciler) computeConfigHash() (string, error) {
if r.mem == nil {
return "", nil
}
snapshot := r.mem.Snapshot()
sort.Slice(snapshot, func(i, j int) bool { return snapshot[i].Name < snapshot[j].Name })
views := make([]crdHashView, 0, len(snapshot))
for i := range snapshot {
views = append(views, crdHashView{
Name: snapshot[i].Name,
Spec: snapshot[i].Spec,
})
}
- b, err := json.Marshal(views)
- if err != nil {
- return "", err
- }
- sum := sha256.Sum256(b)
- return hex.EncodeToString(sum[:]), nil
+ h := sha256.New()
+ k8shash.DeepHashObject(h, views)
+ return hex.EncodeToString(h.Sum(nil)), nil
}Also applies to: 18-21, 105-124
🤖 Prompt for AI Agents
internal/controller/cozystackresource_controller.go lines 5-9 (and similarly
18-21, 105-124): current code uses json.Marshal (and sorting) to hash objects
which is order-sensitive for maps; replace that logic to use
k8s.io/apimachinery/pkg/util/hash.DeepHashObject written into a sha256 hasher
and then hex-encode the sum. Update imports to remove encoding/json and sort
(and any unused packages) and add "k8s.io/apimachinery/pkg/util/hash". For each
hashing helper: create a sha256.New(), call hash.DeepHashObject(hasher, obj),
compute hasher.Sum(nil), and hex.EncodeToString(...) for the stable,
order-insensitive hash; apply this change in all specified ranges.
Signed-off-by: Andrei Kvapil kvapss@gmail.com
What this PR does
This PR introduces shared cache for CozystackResourceDefinitions and warbs it up before making decidion on restart cozystack-api server.
Reastart logic was also updated to trigger restart only if consistent hash from the configuration has been changed.
Release note
Summary by CodeRabbit
New Features
Performance
Bug Fixes