Migrate from fluxcd-community charts to Flux-Operator by kingdonb · Pull Request #166 · cozystack/cozystack

Kingdon Barrett (kingdonb) · 2024-06-13T17:31:06Z

This is close enough to what we discussed at the Cozystack meeting today, to push as a draft

The flux-operator chart is much simpler

~~I will test upgrade locally on my home lab cozystack~~ ☑️

I think we can just use kubectl apply twice, and avoid creating a second "flux-instances" package in core for now (edit: it worked!)

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

we need to conform to the current version in use, 2.2.x, but otherwise we don't need to specify all of this, and as long as we don't specify any of this, then user can merge their own configs in with kubectl apply I hope the FluxInstance registry is set to default to ghcr, so we don't block anyone from using an enterprise or paid distribution of flux with this config! Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

spec.distribution.registry: Required value (drat!) Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) · 2024-06-13T18:05:28Z

At this point I have tested it out locally enough to know that we need at least spec.distribution.registry in the FluxInstance - which is a shame, we'll have to figure out another way that users can add their own settings for registry here, in case they bought Flux Enterprise ;)

Otherwise, users should be able to simply edit the FluxInstance in-place and have their edits merged in, I'll begin testing that. The migration was a success, the first thing that I expected happened once I corrected the error in the FluxInstance mentioned above, was the two image controllers got torn down. But that didn't happen - This is a default configuration in Flux, they are "extra controllers"

I can try using this for a while to see if there are any issues, if it enables all of the customizability that I was hoping, give it a few install-tests, but unless we uncover something I haven't already seen, this is going to be a very easy transition!

This reverts commit 4989e01.

Kingdon Barrett (kingdonb) · 2024-06-13T18:14:35Z

+  name: flux
+spec:
+  distribution:
+    version: 2.2.x


I pinned the version 2.2.x here because I imagine we will want to pin some version, and I know the 2.2.x version is the one that's currently used. We may not need any migration for the 2.3 upgrade because Kubernetes API does the upgrade transparently, from v2beta2 to v2, without intervention.

But these templates will need to be changed at least:

% egrep -ir helm.toolkit.fluxcd.io/v2beta2 * packages/core/platform/templates/helmreleases.yaml:apiVersion: helm.toolkit.fluxcd.io/v2beta2 packages/core/platform/templates/apps.yaml:{{- if .Capabilities.APIVersions.Has "helm.toolkit.fluxcd.io/v2beta2" }} packages/core/platform/templates/apps.yaml:{{- $tenantRoot = lookup "helm.toolkit.fluxcd.io/v2beta2" "HelmRelease" "tenant-root" "tenant-root" }} packages/core/platform/templates/apps.yaml:apiVersion: helm.toolkit.fluxcd.io/v2beta2

I left that stuff alone and opted for the version pin instead, we can upgrade Flux later, or ideally before the next release so we are caught up.

It should be easy enough to test the upgrade now, I guess I can build that out in a separate PR (and try to confirm if we don't really need any migrations!)

Makes sense, thank you

the chart installs by default all 6 components, so make sure we are taking them all over this way Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) · 2024-06-13T18:21:41Z

(I added them to the components list, that worked)

Andrei Kvapil (kvaps)

Now we have chicken-and-egg problem, Fluxcd can't be installed during the initial installation as CNIs are not installed yet, but CNI can't be installed because this construction expects that HelmRelease CR will be existing in a cluster

https://github.com/aenix-io/cozystack/blob/838bee5d25bbe65a48f01e1de610298a4beb91cf/scripts/installer.sh#L25-L32

https://github.com/aenix-io/cozystack/blob/838bee5d25bbe65a48f01e1de610298a4beb91cf/scripts/package-system.mk#L10-L11

here is part of my cozystack log:

make: Leaving directory '/cozystack/packages/core/platform'
make: Entering directory '/cozystack/packages/core/fluxcd'
helm template -n cozy-fluxcd fluxcd . --no-hooks --dry-run=server -a admissionregistration.k8s.io/v1 -a apiextensions.k8s.io/v1 -a apiregistration.k8s.io/v1 -a apps/v1 -a authentication.k8s.io/v1 -a authorization.k8s.io/v1 -a autoscaling/v1 -a autoscaling/v2 -a batch/v1 -a certificates.k8s.io/v1 -a coordination.k8s.io/v1 -a discovery.k8s.io/v1 -a events.k8s.io/v1 -a flowcontrol.apiserver.k8s.io/v1 -a flowcontrol.apiserver.k8s.io/v1beta3 -a fluxcd.controlplane.io/v1 -a networking.k8s.io/v1 -a node.k8s.io/v1 -a policy/v1 -a rbac.authorization.k8s.io/v1 -a scheduling.k8s.io/v1 -a storage.k8s.io/v1 -a v1 | kubectl apply -n cozy-fluxcd -f-
serviceaccount/fluxcd-flux-operator unchanged
customresourcedefinition.apiextensions.k8s.io/fluxinstances.fluxcd.controlplane.io unchanged
clusterrolebinding.rbac.authorization.k8s.io/fluxcd-flux-operator unchanged
service/fluxcd-flux-operator unchanged
deployment.apps/fluxcd-flux-operator configured
kubectl apply -n cozy-fluxcd -f flux-instance.yaml
fluxinstance.fluxcd.controlplane.io/flux unchanged
make: Leaving directory '/cozystack/packages/core/fluxcd'
error: the server doesn't have a resource type "helmrepositories"
time="2024-06-15T15:17:59Z" level=fatal msg="failed to run" err="exit status 1"

Andrei Kvapil (kvaps) · 2024-06-15T15:39:20Z

I was wrong, it seems happening right on the next step, here

https://github.com/aenix-io/cozystack/blob/838bee5d25bbe65a48f01e1de610298a4beb91cf/scripts/installer.sh#L46

Kingdon Barrett (kingdonb) · 2024-06-15T16:41:52Z

Perhaps we can add another wait step - the Flux-Operator FluxInstance has a ready status, let me take a look...

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) · 2024-06-15T16:51:13Z

We could add the wait in the package itself, or in the installer script. I'll give it a local test, I built a new installer image here and I have a little bit of time I can try wiping and reloading the cluster, and follow the logs 👍

Thanks for the review!

Kingdon Barrett (kingdonb) · 2024-06-15T17:29:32Z

Waiting where I have placed it makes this fail... I'll work it out and let you know when it's worked out here

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

IDK Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Andrei Kvapil (kvaps) · 2024-06-15T21:20:05Z

+  name: flux
+spec:
+  cluster:
+    domain: cozy.local


Is it possible to not hardcode the domain? Users might use the different one

We can make our own local values in the parent chart, and keep them for the FluxInstance - sure, I think so.

Everything that happens in the chart now is either a parameter or an override. 🎸 🚀 🪨

Maybe we can figure out how to pass-through to accept the domain as a parameter from the cozystack ConfigMap, before this merges - this will help me know how to add the distribution.registry later in a future build 👍

Stefan Prodan (@stefanprodan) provided an elegant solution for removing cluster domain dependency from fluxcd:
controlplaneio-fluxcd/flux-operator#34 (comment)

I'd like to go this way right now as we won't break the exisiting user flow.

But it seems we still have to add cluster-domain and kubernetes-endpoint options into cozystack configmap.
So if specified, it must be used by fluxcd and other applications, because in some cases this might be required.

I'll prepare an issue to keep this in mind.

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

it's tested now, it works, we know where the failures are (and they aren't a problem for now) Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) · 2024-06-16T02:37:37Z

I'm going to move ~~6d71499~~ (now e51f17b) to #167 and rebase that one on this one.

This is tested now, no errors and it works in a fresh install. It could use some upgrade testing, I think, but should also work. The upgrade was the first thing I tested.

Kingdon Barrett (kingdonb) · 2024-06-16T02:59:56Z

I think that #167 is mergeable now, or this one.

But be aware that flux-operator chart is out of sync with upstream, as these PRs haven't merged yet:

They will take another release of flux-operator, which won't happen before Monday anyway, so I suggest we can wait for it then we can run make update so the version numbers are in sync (and we are not making a fork later, in case something over there doesn't merge as-is.)

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Stefan Prodan (stefanprodan) · 2024-06-16T07:33:55Z

You may want to wait for controlplaneio-fluxcd/flux-operator#33 to get automated updates going forward. I will do a release with this next week, then you can add distribution.artifact to your FluxInstance template.

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

This cumulative PR includes the following changes: - Migrate from fluxcd-community charts to Flux-Operator #166 - Upgrade to Flux 2.3.x #167 - Refactor Flux 2.3 update #172 - Update flux plugin for dashboard #171 - Flux Operator 0.6 #178

Kingdon Barrett (kingdonb) added 5 commits June 12, 2024 20:59

delete flux2 chart

620ca4f

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

add flux-operator chart

6026836

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

redefine fluxcd chart source as flux-operator

ba5c7ca

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

default FluxInstance

97e1bda

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) force-pushed the flux-operator branch from feffc12 to b6bf053 Compare June 13, 2024 18:00

Kingdon Barrett (kingdonb) added 2 commits June 13, 2024 18:01

fix versions map (and tag)

4989e01

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

The FluxInstance "flux" is invalid:

0cd7ffc

spec.distribution.registry: Required value (drat!) Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) force-pushed the flux-operator branch from b6bf053 to 0cd7ffc Compare June 13, 2024 18:01

Kingdon Barrett (kingdonb) marked this pull request as ready for review June 13, 2024 18:05

Kingdon Barrett (kingdonb) changed the title ~~WIP: Flux operator~~ Migrate from fluxcd-community charts to Flux-Operator Jun 13, 2024

Revert "fix versions map (and tag)"

bbd30a3

This reverts commit 4989e01.

Kingdon Barrett (kingdonb) commented Jun 13, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

take over all components

c955975

the chart installs by default all 6 components, so make sure we are taking them all over this way Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) mentioned this pull request Jun 13, 2024

Upgrade to Flux 2.3.x #167

Merged

Andrei Kvapil (kvaps) requested changes Jun 15, 2024

View reviewed changes

kubectl wait for fluxinstance/flux

890ea9a

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) force-pushed the flux-operator branch from 14d28ed to 890ea9a Compare June 15, 2024 16:49

Kingdon Barrett (kingdonb) added 5 commits June 15, 2024 13:40

try a different wait order

8e31c29

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

now I see what this is doing

cb7c4ea

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

try to understand how to fix this

b7dedd4

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

test hostNetwork in chart

6670eb8

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

add hostNetwork: true value

9659702

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) added 6 commits June 15, 2024 15:53

test kubeprism with flux-operator

b7cb20e

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

is this a good validation

955f630

IDK Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

better logic order

758b069

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

set domain in cluster correctly for flux config

3f53a5e

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

set cozy-fluxcd namespace to privileged

3860385

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

magical operators

0e9bdd8

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Andrei Kvapil (kvaps) reviewed Jun 15, 2024

View reviewed changes

Kingdon Barrett (kingdonb) added 3 commits June 15, 2024 21:18

test cluster.domain

2f75341

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

parameterize everything

d5af8d6

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

remove unsure statements from installer.sh

4e6e1d7

it's tested now, it works, we know where the failures are (and they aren't a problem for now) Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Kingdon Barrett (kingdonb) force-pushed the flux-operator branch from 6d71499 to 4e6e1d7 Compare June 16, 2024 02:45

FluxInstance CRD

7790c7c

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Andrei Kvapil (kvaps) mentioned this pull request Jun 16, 2024

Configuration options for cluster-domain and kubernetes-endpoint in cozystack configmap. #168

Closed

Andrei Kvapil (kvaps) changed the base branch from main to upd-flux June 17, 2024 13:57

Andrei Kvapil (kvaps) merged commit 54017b6 into cozystack:upd-flux Jun 17, 2024

This was referenced Jun 17, 2024

FluxCD Updates #170

Merged

Refactor Flux 2.3 update #172

Merged

Kingdon Barrett (kingdonb) deleted the flux-operator branch June 18, 2024 11:33

Kingdon Barrett (kingdonb) added a commit to kingdonb/cozystack that referenced this pull request Jun 22, 2024

Migrate from fluxcd-community charts to Flux-Operator (cozystack#166)

a76a03a

Signed-off-by: Kingdon Barrett <kingdon+github@tuesdaystudios.com>

Aleksei Sviridkin (lexfrei) mentioned this pull request May 4, 2026

fix(objectstorage-controller): retry on Bucket conflict during BucketAccess reconcile #2529

Merged

3 tasks

Uh oh!

Conversation

Kingdon Barrett (kingdonb) commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kingdon Barrett (kingdonb) commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kingdon Barrett (kingdonb) Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

Andrei Kvapil (kvaps) Jun 15, 2024

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Kingdon Barrett (kingdonb) commented Jun 13, 2024

Uh oh!

Andrei Kvapil (kvaps) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andrei Kvapil (kvaps) commented Jun 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kingdon Barrett (kingdonb) commented Jun 15, 2024

Uh oh!

Kingdon Barrett (kingdonb) commented Jun 15, 2024

Uh oh!

Kingdon Barrett (kingdonb) commented Jun 15, 2024

Uh oh!

Andrei Kvapil (kvaps) Jun 15, 2024

Choose a reason for hiding this comment

Uh oh!

Kingdon Barrett (kingdonb) Jun 16, 2024

Choose a reason for hiding this comment

Uh oh!

Kingdon Barrett (kingdonb) Jun 16, 2024

Choose a reason for hiding this comment

Uh oh!

Kingdon Barrett (kingdonb) Jun 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andrei Kvapil (kvaps) Jun 16, 2024

Choose a reason for hiding this comment

Uh oh!

Andrei Kvapil (kvaps) Jun 16, 2024

Choose a reason for hiding this comment

Uh oh!

Kingdon Barrett (kingdonb) commented Jun 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kingdon Barrett (kingdonb) commented Jun 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Stefan Prodan (stefanprodan) commented Jun 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Kingdon Barrett (kingdonb) commented Jun 13, 2024 •

edited

Loading

Kingdon Barrett (kingdonb) commented Jun 13, 2024 •

edited

Loading

Andrei Kvapil (kvaps) left a comment •

edited

Loading

Andrei Kvapil (kvaps) commented Jun 15, 2024 •

edited

Loading

Kingdon Barrett (kingdonb) Jun 16, 2024 •

edited

Loading

Kingdon Barrett (kingdonb) commented Jun 16, 2024 •

edited

Loading

Kingdon Barrett (kingdonb) commented Jun 16, 2024 •

edited

Loading