How to Push Apps to Cloud Foundry With a Sidecar to Stream Metrics to Prometheus
source link: https://blog.anynines.com/how-to-push-apps-to-cloud-foundry-with-a-sidecar-to-stream-metrics-to-prometheus/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Introduction
Retrieving application metrics is usually not difficult. Simply implement a metrics endpoint and let Prometheus, for example, query that endpoint. But why is it the problem when you deploy an application on Cloud Foundry?
If you deploy your application to a platform like Cloud Foundry, you will eventually run into problems. Most likely the endpoint cannot be easily retrieved from a running Prometheus instance to be scraped.
Reasons could be e.g. the endpoint is not reachable from the network where the Prometheus instance is running also there are most probably multiple instances of your application running and each time Prometheus retrieves the application, you are redirected to a different instance via the Gorouter as he acts like a load balancer.
Although Prometheus does not work this way, there is a way around it. Instead of having Prometheus scrape the metrics endpoint, the metrics can be sent for example to a Graphite Exporter endpoint, which is then scraped by Prometheus.
Assuming that a Prometheus service instance is deployed with a Graphite Exporter you only need a sidecar deployed in your application to stream the metrics to the Graphite Exporter.
In this example, a simple application that provides a metrics endpoint is deployed to Cloud Foundry. The sidecar is packaged in a buildpack and deployed in parallel with the application via the application manifest. The actual application is not changed in the process. Thus, the effort is reduced to customizing the manifest. Only a few lines are added.
Let’s take a look at what we have.
The App
The application is written in Go and provides basic metrics in addition to a static page via a separate metrics endpoint using client libraries provided by Prometheus.
The Sidecar
For the sidecar, this example uses the telegraf-buildpack. It includes Telegraf which is a plugin-driven server agent for collecting and sending metrics. It is coupled to your application, scraps the metrics endpoint, and forwards them to a configured Graphite Exporter endpoint. In addition to the application metrics, the telegraf-buildpack is configured to let Telegraf also forward the system metrics from your application container. Collecting system metrics is a separate topic that is worth discussing. I recommend checking out this video.
Prerequisites
To deploy the application we need a platform like Cloud Foundry and a Prometheus instance. The Cloud Foundry installation we want to use is a9s Public PaaS. It offers us the necessary flexibility and we can deploy a Prometheus instance here.
The Prometheus service is the a9s Prometheus which contains Prometheus, the Prometheus Alertmanager, a Graphite Exporter and also a Grafana dashboard connected to Prometheus.
a9s Prometheus is not enabled in all accounts by default, please ask our support to get a preview.
Deploy a9s Prometheus
a9s Prometheus does not provide the ability to display application metrics by default. This is because the Graphite Exporter has a mapping configuration that transfers Graphite-compliant metrics into Prometheus metrics that are enriched with tags. By default, the Graphite Exporter is set to strictly adhere to these mappings and reject others. However, you can change this configuration by setting the mapping_strict_match property to false.
$ cf create-service a9s-prometheus prometheus-single-medium prometheus -c ‘{ “mapping_strict_match”: false }’
Before we can push the application, we have to wait until the service is deployed. You can use this command to query the status.
$ cf service prometheus
Showing info of service prometheus in org demonstrations / space demonstrations as [email protected]…
name: prometheus
service: a9s-prometheus
tags:
plan: prometheus-single-medium
description: This is a service creating and managing dedicated Prometheus to monitor applications and service instances, powered by the anynines Service Framework
documentation: https://www.anynines.com
dashboard: https://a9s-dashboard.de.a9s.eu/service-instances/8f866593-b6bc-4dac-9caf-902677d83b04
service broker: a9s-prometheus
This service is not currently shared.
Showing status of last operation from service prometheus…
status: create succeeded
message:
started: 2022-01-31T11:10:38Z
updated: 2022-01-31T11:12:50Z
Upgrades are not supported by this broker.
The status should be create succeeded like in the example above.
Check out the Sample Application
While waiting we can check out the application with git clone and have a look at the application manifest.
$ git clone https://github.com/anynines/a9s-metrics-sample.git
Change into the directory.
$ cd a9s-metrics-sample
The Application Manifest File
The repository already contains the manifest file. There are three important parts.
- In the `buildpacks` section, the `telegraf-buildpack` is already at the beginning of the list. Do not change the order, as this may cause a deployment error.
`https://gitlabci.exxeta.com/paas_buildpacks/telegraf-buildpack.git`
- The buildpack is designed to automatically use a bound a9s Prometheus instance. Therefore, you must define the service to bind in the `services` section. Cloud Foundry and the a9s Service Broker will then automatically take care of the necessary parameters to connect to the Prometheus instance.
- If you use another Prometheus offering or the instance name in VCAP_SERVICES or your application differs from a9s-prometheus you have to set the env variables GRAPHITE_HOST and GRAPHITE_PORT.
env:
GRAPHITE_HOST: phd31bf28-prometheus.node.dc1.consul.dsf2
GRAPHITE_PORT: 9109
Push the App
When the service instance is ready, simply push the application with the cf push command.
$ cf push
Example Dashboard
When the application is pushed, we can open the Grafana dashboard in our service instance and use the example dashboard provided in the repository to see the metrics. Just copy the JSON data into a new dashboard.
That’s it.
Images
a9s Prometheus Architecture
Sidecar
Communication Restrictions
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK