Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Last Updated: Mar 27, 2024

Config Connector

Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Speaker
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM

Introduction🧑‍🏫

So are you wondering what Config Connector is? And how can it help in boosting your business?

Config connector

 

Don't worry, Ninja! Just follow the article till the end. This blog will discuss the importance of config connector and its management and the commercial advantages it may bring to your company.

Google Cloud resource management is made possible using the open-source Kubernetes addon Config Connector. Many cloud-native development teams utilize various configuration systems, APIs, and tools to manage their infrastructure.

By using Config Connector, your environments can gain from how Kubernetes manages Resources, including:

  • Access control using RBAC(Role-based access control ).
  • For visibility, events
  • Use a single source for configuration and desired state management to reduce complexity.
  • Consistency for loosely coupled dependencies throughout time.

The complexity and cognitive load on developers can be decreased by managing your Google Cloud infrastructure in the same way you manage your Kubernetes apps.

How Config Connector works

Through Config Connector, Kubernetes Custom Resource Definitions (CRDs) and controllers are provided. The Custom Resource Definitions allow Kubernetes to create and manage Google Cloud resources when you configure and apply Objects to your cluster.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Choosing an installation type

Firstly you need to install Config Connector and then create your first resource. The controllers in Config Connector eventually reconcile your environment with your intended state.

You can install Config Connector in one of two ways:

The Config Connector add-on lets you install Config Connector during cluster creation.

To use manual installation, you need to download and use a Kubernetes Operator. Manual installations track the current release of Config Connector more closely; the Operator applies new versions faster than the add-on. 
 

The Service Usage API is used by Config Connector to enable service APIs. You must allow the Service Usage API to perform these steps. Using the Google Cloud CLI, you may enable this API:

gcloud services enable serviceusage.googleapis.com

Getting started with Config Connector 

Let us now understand how to enable the Google Cloud API, Create and manage a Pub/Sub topic by using the basics of Config Connector resource management. 

config connector img

Discovering the available Google Cloud resources

 You can create with  Config Connector to see what kinds of Google Cloud resources, run:

kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true

Enabling Pub/Sub service 

Applying a YAML configuration to your cluster will enable the Pub/Sub API using Config Connector:

  1. Create a file named enable-pubsub.yaml and copy the YAML below into it:
apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
kind: Service
metadata:
  name: pubsub.googleapis.com
spec:
  projectRef:
    external: projects/PROJECT_ID

 

  1. To apply the configuration to your cluster, use kubectl apply. Enable the Pub/Sub API by running the following command:
kubectl apply -f enable-pubsub.yaml

Creating Pub/Sub instance 

A file named pubsub-topic.yaml should be created using the given content:

apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubTopic
  metadata:
    annotations:
      cnrm.cloud.google.com/project-id: PROJECT_ID
    labels:
      LABEL_KEY:LABEL_VALUE
    name: TOPIC_NAME

 

Config Connector creates the resource when you create it if it doesn't already exist. Config Connector acquires and manages the resource if a Google Cloud resource with the same name already exists.

Describing a resource 

To get details on a resource using kubectl describe.

Run the following command to view all your Pub/Sub topics:

kubectl describe pubsubtopics

Verifying that a resource is ready

You can check the status.condition of a Pub/Sub topic after you've created it. For instance, use the following command to check if your Pub/Sub topic resource is fully ready. Run the command:

kubectl wait --for=condition=READY pubsubtopics TOPIC_NAME

Updating a resource

By making updates to your YAML file and re-applying it with kubectl, you can update the metadata on your resources.

  1. Change the label by modifying the metadata section of your pubsub-topic.yaml file:
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubTopic
  metadata:
    annotations:
      cnrm.cloud.google.com/project-id: PROJECT_ID
    labels:
      NEW_LABEL_VALUE
    name: TOPIC_NAME

2. To update the resource, use kubectl apply. Run the command below:

kubectl describe pubsubtopics

3. Check the Pub/Sub example for the name change:

kubectl describe pubsubtopics

Deleting a resource

For resource deletion, use kubectl delete. For instance, use your pubsub-topic.yaml file and run kubectl delete to delete the PubSubTopic you created previously:

kubectl delete -f pubsub-topic.yaml

Config Connector deletes the PubSubTopic resource by default. Take a look at the instructions in Managing and deleting resources if you prefer to keep this resource.

Installing with the GKE add-on

Let's examine how to use the Config Connector add-on to install Config Connector on a Google Kubernetes Engine (GKE) cluster.

Installing the Config Connector add-on

By creating a new GKE cluster or enabling an existing cluster, you can use the Config Connector add-on. You configure your Config Connector installation using your Google service accounts and namespaces after installing the Config Connector add-on.

Setting up a GKE 

The Config Connector add-on is usable with both new and existing clusters.

  • With the Config Connector add-on enabled creating a new cluster
    Using the Google Cloud console or the gcloud CLI, you can create a GKE cluster.
  • On an existing cluster enabling the Config Connector add-on 
    With gcloud or the Google Cloud console, you can enable the Config Connector add-on on an existing GKE cluster.

Creating an identity 

creating identity image

By authenticating with an Identity and Access Management (IAM) service account and using GKE's Workload Identity to connect IAM service accounts with Kubernetes service accounts, Config Connector can create and manage Google Cloud resources.

  1. Firstly create an IAM service account. 
  2. Give the IAM service account elevated project permissions:
  3. Create a binding between the IAM service account and the predefined Kubernetes service account that Config Connector uses by creating an IAM policy.

Configuring Config Connector

For the ConfigConnector CustomResource, create a configuration file and apply it using the kubectl command to complete the installation. The Google Cloud Resource CRDs and Config Connector components are installed in your cluster by the Config Connector Operator.

Specifying where to create your resources

Similar to how you would organize resources in Google Cloud, Config Connector allows you to manage resources by project, folder, or organization.

Config Connector requires that you first configure where your resources will be created. The construction of the resource is determined by an annotation on either the resource configuration or an existing Namespace by Config Connector. SeeOrganizing resources for more details.

Verifying your installation

The namespace cnrm-system is where Config Connector runs all of its components. To verify whether the Pods are ready or not, run the following command:

kubectl wait -n cnrm-system \
      --for=condition=Ready pod --all

Configuring your kubectl client 

By default, Config Connector expects the resource's namespace to match the Google Cloud project ID where it's created.

You can simplify commands and avoid adding --namespace to each kubectl command by changing the default context's namespace. To do so, run the following command:

kubectl config set-context --current --namespace NAMESPACE_NAME

Managing and deleting resources 

Let us now see how Config Connector manages an existing resource and handles deleting resources.

Acquiring a BigQuery dataset

When the values in the resource name and the manifest match, Config Connector takes ownership of the resources. The resource's ID is used if it lacks a name (such as a Project ID, for instance).

You may learn how Config Connector manages existing resources by generating an empty BigQuery dataset and then acquiring the dataset using Config Connector.

  1. With bq, make a BigQuery dataset with the name bigquerydatasetsample.
bq --location=US mk \
--dataset \
--default_table_expiration 3600 \
--description description \
PROJECT_ID:bigquerydatasetsample

 

  1. Into a file called bq-sample.yaml, paste the following content.
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
  kind: BigQueryDataset
  metadata:
    name: bigquerydatasetsample
  spec:
    defaultTableExpirationMs: 3600000
    description: "BigQuery Dataset Sample"
    friendlyName: bigquerydataset-sample
    location: US

 

  1. Apply the yaml to your cluster.
kubectl apply --namespace CC_NAMESPACE -f bq-sample.yaml

 

  1. To view the dataset's details, use kubectl describe.
kubectl describe --namespace CC_NAMESPACE bigquerydataset bigquerydatasetsample

 

Deleting the dataset

By default, deleting an item from your cluster will also destroy any resources that Config Connector has acquired and managed. Set the resource's deletion policy

The dataset will be removed from BigQuery, for instance, if the manifest used to obtain bigquerydataset-sample is deleted.

  1. The kubectl delete command can be used to remove the bigquerydataset-sample dataset.
kubectl delete --namespace CC_NAMESPACE -f bq-sample.yaml


The deletion is verified by the kubectl output.

bigquerydataset.bigquery.cnrm.cloud.google.com "bigquerydatasetsample" deleted

  1. To confirm that the dataset is no longer available, use bq.
bq show PROJECT_ID:bigquerydatasetsample

Importing and exporting resources 

You can export your current resources to Config Connector YAML files using the config-connector tool. Once you've done that, you can use kubectl to import the resources into Config Connector and apply them.

Installing config-connector

  1. Download the most recent binary tar file of the config-connector:
gsutil cp gs://cnrm/latest/cli.tar.gz .

2. After that you need to extract the tar file:

tar zxf cli.tar.gz

3. A binary executable for MacOS, Linux, and Windows is included in the tar file. Choose the binary that fits your OS:

  • Linux: ./linux/amd64/config-connector
  • Windows: ./windows/amd64/config-connector
  • MacOS: ./darwin/amd64/config-connector

4. Store the binary to a location by copying on your ${PATH}. For example, you could use mv to move it into /usr/local/bin on Linux and MacOS.

mv linux/amd64/config-connector /usr/local/bin

5. Enable the Cloud Asset Inventory API on your Google Cloud Identity project with gcloud if you wish to utilise the config-connector tool to export directly from Cloud Asset Inventory.

gcloud services enable cloudasset.googleapis.com

Using Secrets to store sensitive data 

secrets image

In your GKE cluster, you can use a  Secret to store secret configuration data like passwords and access keys.

 

Let us see an overview of how to use Secrets with Config Connector.

Using a Secret when creating a resource

When creating a resource, you can use the information contained in a Secret. By creating a Secret that includes a password and referencing it for a user on a Cloud SQL database, you can transmit a secret to a resource.

Updating Secrets

Config Connector refreshes the resource when it next reconciles your intended state after you apply an update to a Secret. Config Connector will update the user's password if, for instance, you change the Secret that is used as the password for a SQL user.

Deleting Secrets

Config Connector won't remove the resource's reference to a Secret if you delete one it is using. For instance, the password will stay on the SQLUser if you create it with a password that refers to a Secret and then delete the Secret.

Troubleshooting

Config Connector will generate a DependencyNotFound or DependencyInvalid Event if you create a resource that refers to an invalid Secret.

Secrets and multiple projects

If you are using Config Connector to manage multiple projects, you must apply secrets in every Namespace that corresponds to a project since Kubernetes does not permit access to secrets across Namespaces.

Monitoring Config Connector with Prometheus 

Metrics from the Config Connector can be collected and shown using Prometheus.

Scraping metrics

By scraping an HTTP endpoint, Prometheus gathers metrics. The configuration of Prometheus and the Config Connector scrape endpoints are covered in this section.

Config Connector scrape endpoints

Service endpoints for Config Connector are at cnrm-controller-manager-service and cnrm-resource-stats-recorder-service on port 8888. The annotations prometheus.io/scrape: "true" and prometheus.io/port: "8888" are present on these services.

Configuring Prometheus

You may need to configure Prometheus for Kubernetes Service Discovery (SD) to find scrape targets from the Kubernetes REST API before you can scrape metrics.

Example queries

The PromQL query language is used by Prometheus. Any valid PromQL query can be configured to cause Prometheus to generate alerts.

 

Query reconciliation by status and resource kind

By resource kind and status, you can see the total number of failed reconcile requests.
 

Check the aggregate status of resources by kind and Namespace

The number of resources in a Namespace is visible.

 

Query the utilization of reconcile workers per resource kind

(configconnector_reconcile_occupied_workers_total / configconnector_reconcile_workers_total)

 

Enabling resource name labels

Metrics are typically aggregated based on the kind of resource (for example, PubSubTopic). Metrics that are aggregated by individual resources can be enabled.

Follow these steps to enable resource name aggregation.

  1. Editing the cnrm-controller-manager StatefulSet object
     
  2. Finding the array spec.args and add --resource-name-label=true.

 

Now it’s time for the questions. Let us now move to FAQs.

FAQs

Also see, kubernetes interview questions

Frequently Asked Questions

What is a Google connector?

All of your Google Workspace data is automatically indexed by Google Cloud Search. A connector is a program you can create on your own to index data stored in a third-party repository. In addition to your repository, a separate program, a script that runs in its process, or another script can all be connectors.

What is a configuration connector?

You can manage Google Cloud resources with Kubernetes thanks to the open-source Config Connector add-on. To manage their infrastructure, many cloud-native development teams use a mix of configuration systems, APIs, and tools.

Describe GCP Deployment Manager.

The creation and maintenance of Google Cloud resources are automated via the infrastructure deployment service called Google Cloud Deployment Manager.

How does Config Connector work?

Kubernetes Custom Resource Definitions (CRDs) and controllers are provided through Config Connector. When you configure and apply Objects to your cluster, Kubernetes can create and manage Google Cloud resources via the Config Connector CRDs.

What is a workload identity namespace?

Workload Identity enables workloads in your GKE clusters to access Google Cloud services by impersonating Identity and Access Management (IAM) service accounts. On Autopilot clusters, Workload Identity is enabled by default.

 

Conclusion

In this article, we thoroughly discussed Config Connector. I hope that this article has helped you to enhance your knowledge regarding the working, installation, creating, updating and deleting secrets as well as secrets and monitoring-prometheus

Coding Ninja img

 

Please refer to our guided paths on Coding Ninjas Studio to learn more about DSA, Competitive Programming, JavaScript, System Design, etc. And also, enroll in our courses and refer to the mock test and problems available. Have a look at the interview experiences and interview bundle for placement preparations.

Do upvote our blog to help other ninjas grow.

Happy Learning!

Topics covered
1.
Introduction🧑‍🏫
2.
How Config Connector works
3.
Choosing an installation type
4.
Getting started with Config Connector 
4.1.
Discovering the available Google Cloud resources
4.2.
Enabling Pub/Sub service 
4.3.
Creating Pub/Sub instance 
4.4.
Describing a resource 
4.5.
Verifying that a resource is ready
4.6.
Updating a resource
4.7.
Deleting a resource
5.
Installing with the GKE add-on
5.1.
Installing the Config Connector add-on
5.2.
Setting up a GKE 
6.
Creating an identity 
6.1.
Configuring Config Connector
6.2.
Specifying where to create your resources
6.3.
Verifying your installation
7.
Configuring your kubectl client 
8.
Managing and deleting resources 
8.1.
Acquiring a BigQuery dataset
8.2.
Deleting the dataset
8.3.
Importing and exporting resources 
9.
Installing config-connector
10.
Using Secrets to store sensitive data 
10.1.
Using a Secret when creating a resource
10.2.
Updating Secrets
10.3.
Deleting Secrets
10.4.
Troubleshooting
10.5.
Secrets and multiple projects
11.
Monitoring Config Connector with Prometheus 
11.1.
Scraping metrics
11.2.
Config Connector scrape endpoints
11.3.
Configuring Prometheus
12.
Example queries
12.1.
Query reconciliation by status and resource kind
12.2.
Check the aggregate status of resources by kind and Namespace
12.3.
Query the utilization of reconcile workers per resource kind
13.
Enabling resource name labels
14.
Frequently Asked Questions
14.1.
What is a Google connector?
14.2.
What is a configuration connector?
14.3.
Describe GCP Deployment Manager.
14.4.
How does Config Connector work?
14.5.
What is a workload identity namespace?
15.
Conclusion