Motivations
At IBM, I work on a product called Cloud Pak for AIOps. It is a self-hosted IT operations management tool that runs on Kubernetes. Because it is self-hosted and is composed of dozens of microservices, one of the key challenges we face is keeping installation and upgrade simple.
We use a handful of operators (each with one or more custom resources and controllers) to deploy and manage all the Kubernetes resources within the application. These controllers manage a wide array of resources, both ones native in Kubernetes as well as custom resources.
Of the many resources managed by our Go-based controllers, some provide Go types while others are managed through unstructured (a map[string]interface{} in Go with Kubernetes type meta). Across all of these objects, there are often common actions we wish to take — applying labels, setting replica counts, configuring owner references, etc.
With github.com/griffindvs/stencil, I started putting together some tools to assist with these types of operations. This library is built on many of the same core functions as Manifestival, but with an extension to Go typed Kubernetes objects which can take advantage of controller-runtime caching. After spending some time with the library, I did not end up completely satisfied with the user experience, but I find some of the tools and ideas useful in other areas.
Generic transformations
As provided in Manifestival, a useful way for conducting generic transformations on Kubernetes resources is to manage them with unstructured. A basic transformer could apply an owner reference to an object:
1// Transformer conducts in-place modifications to a provided object.
2type Transformer func(u *unstructured.Unstructured) error
3
4// OwnerReferenceTransformer returns a transformer that adds
5// an owner reference to the given object.
6func OwnerReferenceTransformer(
7 k8sClient client.Client, owner client.Object,
8) Transformer {
9 return func(u *unstructured.Unstructured) error {
10 if u == nil {
11 return nil
12 }
13
14 if k8sClient == nil {
15 return errors.New("unable to add owner reference with nil client")
16 }
17
18 ownerGvk, err := apiutil.GVKForObject(owner, k8sClient.Scheme())
19 if err != nil {
20 return err
21 }
22
23 u.SetOwnerReferences(
24 []metav1.OwnerReference{*metav1.NewControllerRef(owner, ownerGvk)},
25 )
26 return nil
27 }
28}A more complex transformation might take into account the current state of the object in the cluster. In stencil, I refer to these as context-aware transformers:
1// ContextAwareTransformer conducts in-place modifications to a
2// provided object with the context of the current object on the cluster.
3type ContextAwareTransformer func(desired *unstructured.Unstructured, current *unstructured.Unstructured) error
4
5// NewContextAwareTransformer returns a ContextAwareTransformer
6// with the current object on the cluster.
7func NewContextAwareTransformer(
8 ctx context.Context,
9 k8sClient client.Client,
10 fn ContextAwareTransformer,
11) Transformer {
12 return func(u *unstructured.Unstructured) error {
13 if u == nil {
14 return nil
15 }
16
17 // Get the current object on the cluster.
18 var current = &unstructured.Unstructured{}
19 current.SetGroupVersionKind(u.GroupVersionKind())
20 err := k8sClient.Get(ctx, client.ObjectKeyFromObject(u), current)
21 if err != nil {
22 if kerrors.IsNotFound(err) {
23 return fn(u, nil)
24 }
25
26 return err
27 }
28
29 // Run the provided transformer with both the new and current objects.
30 return fn(u, current)
31 }
32}A context-aware transformation can be used for things like preventing inadvertent reductions in scale that could lead to failures. For example, perhaps the volume capacity in the claim template of a StatefulSet is configurable, but you want to prevent attempts to reduce the capacity. Before you apply the updated StatefulSet, you could transform the resource to retain the larger capacity from the object on the cluster.
Converting between representations
To transform objects as unstructured while retaining some of the benefits of Go typed objects, we need a mechanism to switch between the representations. Moving from typed to unstructured is fairly simple:
1// ToUnstructured converts the given client.Object to an unstructured.
2func ToUnstructured(o client.Object) (unstructured.Unstructured, error) {
3 u, err := runtime.DefaultUnstructuredConverter.ToUnstructured(o)
4 return unstructured.Unstructured{Object: u}, err
5}The one catch is that Go typed objects do not include type meta, since that information is inherent in the Go type. To address this, we can set the GVK of the unstructured after the conversion:
1// SetGvkFromScheme populates the GroupVersionKind of an
2// unstructured object using the type stored in the client
3// scheme for the given client.Object.
4func SetGvkFromScheme(
5 k8sClient client.Client,
6 unstruct *unstructured.Unstructured,
7 typed client.Object,
8) error {
9 if k8sClient == nil {
10 return errors.New("unable to set GVK from scheme with nil client")
11 }
12
13 if unstruct == nil {
14 return errors.New("unable to set GVK on nil unstructured")
15 }
16
17 typedGvk, err := apiutil.GVKForObject(typed, k8sClient.Scheme())
18 if err != nil {
19 return err
20 }
21
22 unstruct.SetGroupVersionKind(typedGvk)
23 return nil
24}Moving the other direction is a bit more challenging. To do this generically, we need to create a pointer to whatever the type is that we determine at runtime:
1// NewRuntimeType return a pointer to a new value with
2// type determined at runtime.
3func NewRuntimeType[T any](t T) any {
4 return reflect.New(reflect.TypeOf(t).Elem()).Interface()
5}We can then switch from unstructured to that type:
1// Declare a pointer to the type of object (determined at runtime).
2var typed = common.NewRuntimeType(obj)
3
4// Convert from unstructured back to the typed object.
5err = runtime.DefaultUnstructuredConverter.FromUnstructured(u.Object, typed)
6if err != nil {
7 return nil, err
8}To then treat that type as a Kubernetes object, we need to assert that it implements client.Object:
1// ClientObjectInterface returns the reflect.Type for the
2// client.Object interface.
3func ClientObjectInterface() reflect.Type {
4 return reflect.TypeOf((*client.Object)(nil)).Elem()
5}
6
7// ImplementsClientObject checks if the provided object implements
8// client.Object, and if so, returns that object as a client.Object.
9func ImplementsClientObject(obj any) (client.Object, error) {
10 objectType := reflect.TypeOf(obj)
11 if objectType.Implements(ClientObjectInterface()) {
12 clientObject, ok := obj.(client.Object)
13 if !ok {
14 return nil, fmt.Errorf(
15 "unable to convert %v to client.Object", reflect.TypeOf(obj),
16 )
17 }
18
19 return clientObject, nil
20 }
21
22 return nil, fmt.Errorf(
23 "%v does not implement %v", objectType, ClientObjectInterface(),
24 )
25}