Join our - Lead the Future of API Dev: A New Approach with Blackbird -Webinar on September 19thRegister Now

Back to blog
KUBERNETES

How to Debug Kubernetes Services With VS Code and Telepresence

Jake Beck
April 23, 2024 | 15 min read

Many organizations using Node.js adopt cloud native development practices with the goal of shipping features faster. The technologies and architectures may change when we move to the cloud, but the fact remains that we all still add the occasional bug to our code. The challenge here is that many of your existing local debugging tools and practices can’t be used when everything is running in a container or on the cloud. A change in approach is required!

In this article, I’ll show you how to debug Kubernetes services using Telepresence and VScode. This will enable you to still utilize your local debugging tools even though your microservice application runs remotely.

Local Debugging with Mocks and Stubs Only Gets You So Far

One of the biggest challenges when working with a microservice system is spinning up all of your services locally when testing and debugging. Once you get to a certain number of services it becomes physically impossible to run these via a local Kubernetes cluster like minikube or Kind.

The first approach often taken is to create mocks, stubs, or doubles, perhaps using something library-based like Jest, or something out of process like Mountebank. Although valuable, these doubles can often encode incorrect assumptions, or they diverge over time against the real implementation. Often you are forced to sacrifice realistic testing compared to what is running within your production environment, and you end up pushing code to production with uncertainty.

In an ideal world, you would be testing against all of your services running in a cluster. Accordingly, the next approach engineers take is to attempt to remotely debug their services running in a remote cluster. Often this cluster is shared; perhaps each team has its own cluster, or maybe you are using a communal staging or pre-production cluster.

Difficulties with Debugging Node Apps Running in Kubernetes

Remote debugging against node.js apps running in Kubernetes can be challenging. The primary issue is exposing the debug ports for your locally running IDE or debugger to connect to. You can expose the port in your Kubernetes Service YAML, but this can be a security risk if this is deployed to production, so you typically have to maintain two copies of the YAML when using this approach. You can use

kubectl port-forward
to set up a local connection to a specific Pod and debug port, but you will have to re-issue the command every time the Pod name changes.

As an alternative, you can spin up a local copy of the service under test and attempt to use

kubectl port-forward
to expose remote services (via localhost and their ports). This means you can debug the copy of your Kubernetes service without needing to connect your debugger to the remote cluster. However, the challenge here is that often the environment variable configuration for your service is only available in the Kubernetes cluster. You now have to recreate this locally.

Telepresence can be used to overcome both of these challenges. By setting up a two way proxy between your local machine and remote Kubernetes cluster you can debug Kubernetes services locally as if they were running in the cluster i.e. your service can connect to other remote services simply using the k8s Service names and ports. Telepresence also allows the easy exporting of environment variables, which you can load into your local IDE or debugger.

Let’s now walk through a worked example of how to set all of this up.

Tutorial: VS Code + Telepresence = Easy Node.js K8s Debugging

Step 1: Deploy a Sample Microservice Application

In this tutorial, we’ll build on a sample application that was introduced in a previous article. All the instructions you need to debug the “DataProcessingService” node.js application are included in the article you are now reading. We assume you have access to a Kubernetes cluster, either a remote cluster or something like minikube running locally that you can pretend is a remote cluster. We also assume you have a current version of Node installed, alongside VS Code.

You can find a detailed explanation of each service in the original article, and the architecture diagram below should provide a high-level overview of the dependencies between services:

K8s services example
K8s services example

In this architecture diagram, you’ll notice that requests from users are routed through an ingress controller to our services. For simplicity’s sake, we’ll skip the step of deploying an ingress controller in this tutorial. If you’re ready to use Telepresence in your own setup and need a simple way to set up an ingress controller, we recommend checking out the Edge Stack API Gateway.

First, let’s deploy the sample application to your Kubernetes cluster:

If you run

kubectl get service
you should see something similar to this:

kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dataprocessingservice ClusterIP 10.3.255.141 <none> 3000/TCP 5s
kubernetes ClusterIP 10.3.240.1 <none> 443/TCP 17m
verylargedatastore ClusterIP 10.3.241.162 <none> 8080/TCP 4s
verylargejavaservice ClusterIP 10.3.243.172 <none> 8080/TCP 4s

Step 2: Set up your Local Node.js Development Environment and VS Code

You will need to configure your local development environment to debug the

DataProcessingService
Kubernetes service. As you can see in the architecture diagram above, the
DataProcessingService
is dependent on both the
VeryLargeJavaService
and the
VeryLargeDataStore
, so to make a change to this service, we’ll have to interact with these other services as well. You can imagine that the web page generating monolith “
VeryLargeJavaService
” and “
VeryLargeDataStore
” are too resource hungry to run on your local machine.

So, let’s get started with using our new approach to debugging!

  1. Clone the repository for this application from GitHub.

git clone https://github.com/datawire/edgey-corp-nodejs.git

2. Navigate into the directory and install the dependencies via NPM

npm install

3. After NPM has finished downloading the dependencies, start VS Code and select “Open” from the “Welcome” screen. Navigate to the

DataProcessingService
and click the “Open” button.


data processing
data processing

4. After the project loads into VS Code, open the

app.js
file in the editor by double-clicking on this in the Explore window.

explore window
explore window

5. Start the application in debug mode by clicking on ‘Run and Debug’ icon in the left side navigation pane, and then clicking the ‘Run and Debug’ button and selecting ‘Node.js’ from the dropdown menu that is shown.

Visual Studio Code
Visual Studio Code

6. The application will start in debug mode, and the (1) debug control panel is shown, alongside the (2) debug console (where application logging will be shown) and the (3) call stack explorer

example
example

7. In a terminal window, type the command curl

localhost:3000/color
to see that your locally running debug service is returning the color blue.

$ curl localhost:3000/color

“blue”

You now have your local service loaded into your IDE and running in debug mode! Now connect this to the remote Kubernetes cluster.

Step 3: Install and Configure Telepresence

Instead of fiddling with remote debugging protocols and exposing ports via

kubectl port-forward
to access services running in our remote Kubernetes cluster we are going to use Telepresence, an open source Cloud Native Computing Foundation project. Telepresence creates a bidirectional network connection between your local development and the Kubernetes cluster to enable fast, efficient Kubernetes development.

  1. Install Telepresence CLI (macOS version). See the documentation for a Linux version.

# Mac OS X

sudo curl -fL https://app.getambassador.io/download/tel2/darwin/amd64/latest/telepresence -o /usr/local/bin/telepresence

#Linux

sudo curl -fL https://app.getambassador.io/download/tel2/linux/amd64/latest/telepresence -o /usr/local/bin/telepresence


2. Make the binary executable

sudo chmod a+x /usr/local/bin/telepresence


3. Test Telepresence by connecting to the remote cluster

telepresence connect


4. Send a request to the remotely running

DataProcessingService
:

“green”


5. Notice two things here:

A. You can refer to the remote Service directly via its internal cluster name as if your development machine is inside the cluster

B. The color returned by the remote

DataProcessingService
is “green”, versus the local result you saw above of “blue”

Great! You’ve successfully configured Telepresence. Right now, Telepresence is “intercepting” (discussed below) the request you’re making to the Kubernetes API server, and routing over its direct connection to the cluster instead of over the Internet.


Step 4: Intercept Remote Traffic and Debug Your Local Kubernetes Service

An intercept is a routing rule for Telepresence. You can create an intercept to route all traffic intended for the

DataProcessingService
in the cluster to the local version of the
DataProcessingService
running in debug mode on port 3000.


Create the intercept


telepresence intercept dataprocessingservice — port 3000

You will see something similar to the following output:

Using deployment dataprocessingservice
intercepted
Intercept name : dataprocessingservice
State : ACTIVE
Destination : 127.0.0.1:3000
Volume Mount Error: macFUSE 4.0.5 or higher is required on your local machine
Intercepting : all TCP port connections

Access the application directly with Telepresence. Visit http://verylargejavaservice:8080 in your browser. Again, Telepresence is intercepting requests from your browser and routing them directly to the Kubernetes cluster. You should see a web page that displays the architecture of the system you have deployed into your cluster:

edgy corp
edgy corp

Note that the color of the title and

DataProcessingService
box is blue. This is because the color is being determined by the locally running copy of the
DataProcessingService
as the Telepresence intercept is routing the remote cluster traffic to this.


Within VS Code, scroll the editor window with the

app.js
file content showing so that you can see line 42. Click once in the margin to the left of the line number to set a breakpoint. A red dot should appear both where you clicked and also within the code on that line to indicate a breakpoint has successfully been set. This breakpoint will be triggered when the “color” endpoint of the
DataProcessingService
is called.

Example
Example

In your browser, visit http://verylargejavaservice:8080 again. Notice how VS Code immediately jumps to the foreground on your desktop with the (1) breakpoint hit. You can view the (2) call stack in the bottom left corner of the Debug window, and you can also see the (3) current variables involved with the execution of this service in the “Variables” panel to the left of the editor window.

At this point, you can perform all of the debug actions you typically can, e.g. inspecting variable values, changing variables, (4) stepping through and over code, and halting execution.

Example
Example

In the Variables pane on the left side of the screen, click once on the “Closure” section to expand the list of variables shown (3 in the image above).

Locate the color variable and double-click on the value ‘blue’. This should be editable. Change this value to ‘orange’ and press enter (1, below). Next, click the “Continue” icon in the debug control panel (2).


Example
Example

Your browser window should complete reloading and display an orange color for the title and

DataProcessingService
box:

Edgy Corp Webapp
Edgy Corp Webapp

Success! You have successfully made a request to the remote

VeryLargeJavaService
and Telepresence has intercepted the call this service has made to the remote
DataProcessingService
and rerouted the traffic to your local copy running in debug mode!

In addition to rapidly inspecting and changing variables locally, you can also step through the execution of the local service as if it were running in the remote cluster. You can view data passed into the local service from the service running in the remote cluster and interact with other services running in the cluster as if you were also running here.

Bonus: Loading K8s Environment Variables into VS Code

Telepresence also allows you to locally access the environment variables your service has access to in the remote cluster. This is useful if you store essential configuration in Kubernetes environment variables for service discovery, selection of runtime profile names, feature flag config, etc.

You can leave the intercept you created above and modify the Telepresence command line flags to export the environment variables into an env formatted file:

telepresence leave dataprocessingservice
telepresence intercept dataprocessingservice — port 3000 — env-file=my-envs.env
Using deployment dataprocessingservice
intercepted
Intercept name : dataprocessingservice
State : ACTIVE
Destination : 127.0.0.1:3000
Volume Mount Error: macFUSE 4.0.5 or higher is required on your local machine
Intercepting : all TCP port connections
$ head my-envs.json
DATAPROCESSINGSERVICE_PORT=tcp://10.3.255.141:3000
DATAPROCESSINGSERVICE_PORT_3000_TCP=tcp://10.3.255.141:3000
DATAPROCESSINGSERVICE_PORT_3000_TCP_ADDR=10.3.255.141
DATAPROCESSINGSERVICE_PORT_3000_TCP_PORT=3000
DATAPROCESSINGSERVICE_PORT_3000_TCP_PROTO=tcp
DATAPROCESSINGSERVICE_SERVICE_HOST=10.3.255.141
DATAPROCESSINGSERVICE_SERVICE_PORT=3000
KO_DATA_PATH=/var/run/ko
KUBERNETES_PORT=tcp://10.3.240.1:443
KUBERNETES_PORT_443_TCP=tcp://10.3.240.1:443

This env file can be loaded into VS Code by clicking on the Launch dropdown, selecting “Add Configuration”, and then selecting “Node.js” from the “Select Environment” dropdown that is shown.

Example
Example

A new ‘launch.json’ file is created and opened in the editor. VS Code uses this file to configure the local debug environment.

Locate the ‘program’ properties within the

launch.json
file, add a comma to the end of this line, and then add an
envFile
property that points to the Telepresence env file you generated above. If you ran the Telepresence intercept and created the corresponding env file in the
DataProcessingService
project folder, you can use the ${workspaceFolder} token followed by your env file name, as shown below.

example
example

Once you have added the envFile property, you will need to stop and start the debug process (simply reloading the debug process will not provide the expected results). Now the application you are debugging can read the environment variables as if this process was running within the remote cluster.

Here is an example of adding a variable watch for the

DATAPROCESSINGSERVICE_PORT
env passed in via the method above:

Example
Example

You can also locally access volumes mounted into your remote Services with Telepresence. This is useful if you are storing configuration, tokens, or other state required for the proper execution of the service. We’ll cover this in more detail in a future tutorial.

Learn More About Telepresence

In this article, you’ve learned how to use Telepresence to easily debug Kubernetes services. Now, instead of trying to mock out dependencies or fiddle around with remote debugging, you can iterate quickly with an instant feedback loop when locally debugging using your favorite IDE and tools.