This was originally posted on OpenShift's blog.
Regardless of whether you are developing traditional applications or microservices, you will inevitably have a requirement to monitor the health of your applications, and the components upon which they are built.
Furthermore, users of OpenShift Container Platform will often have their own existing enterprise standard monitoring infrastructure into which they will be looking to integrate applications they build. As part of Red Hat's Professional Services organisation, one of the monitoring platforms I encounter whilst working with OpenShift in the field is the AppDynamics SaaS offering.
In this post, I will run through how we can take a Source-to-Image (S2I) builder, customise it to add the monitoring agent, and then use it as the basis of a Fuse Integration Services application, written in Apache Camel, and using the Java CDI approach.
Before getting into the process of modifying the S2I builder image and building the application, the first thing we need to do is register with the AppDynamics platform. If you're an existing consumer of this service, then this step obviously isn't required!
Either way, once registered, we need to download the Java agent. In this example, I've used the Standalone JVM agent, but there are many more options to choose from, and one of those may better suit your requirements.
There are two primary ways you can go about adding the AppDynamics Java agent to your image.
Firstly, you can use Source-To-Image (S2I) to add the Java agent to the standard fis-java-openshift base image at the same time as pulling in all your other dependencies – mainly source code and libraries.
Secondly, you can extend the fis-java-openshift S2I builder image itself, add your own layer containing the Java agent, and use this new image as the basis for your builds.
When using S2I to create an image, OpenShift can execute a number of scripts as part of this process. The two scripts we are interested in in this context are assemble and run.
In the fis-java-openshift image, the S2I scripts are located in /usr/local/s2i. We can override the actions of these scripts by adding an .s2i/bin directory into the code repository, and creating our new scripts there.
As can be seen above, we actually get this script to execute the original assemble script before we add the AppDynamics agent – this way, if the Maven build fails, we haven't wasted any time downloading any artifacts we're not going to use.
The run script is going to be the script that sets up the environment to allow us to use the AppDynamics Java agent, and – you've guessed it – run our app! Just as with the assemble script, we still want run to carry on executing our application when our customisations are complete. Therefore, all we do here is get it to check for the presence of an environment variable, and if it's found, configure the environment to use AppDynamics.
In this case, we're looking for a variable called APPDYNAMICS_AGENT_ACCOUNT_NAME. After all, if we haven't configured any credentials for the Java agent, then it can't connect to the AppDynamics SaaS anyway.
Finally, to bring this all together, we can use a Template to pull all of these components together, begin the build process, and deploy our application.
The Java agent needs to be hosted somewhere accessible to your build process. It also needs to be version controlled separately from the build, which adds extra build management overhead.
It will be downloaded every single time you run a build – not the most efficient way of deploying it if you have an effective CI / CD pipeline and are doing multiple builds per hour!
Whilst it's simpler to configure, it can present confusing problems during the build process. For example, if your assemble script creates some directories for your application to use (logging directories, for example), you may need to think about how your build and application are being executed, and who owns what in that process.
Regardless of these minor issues, this is still a powerful (and useful!) mechanism, and as such I have provided a sample repository that allows you to execute an S2I build that should pull in the Java agent and run it alongside an application.
NOTE: If you're still interested in using the S2I process, and want to know more about how to configure the Java agent with environment variables, skip ahead to 'Adding the AppDynamics agent to a FIS application'.
My preference for using the AppDynamics Java agent with applications built on FIS (and for similar use cases), is to add it into the base image once, so that it is accessible by the any applications based on that image.
In this Dockerfile, we are adding the content of the appdynamics directory in our Git repository to the fis-java-openshift base image, and altering its permissions so that it is owned by the JBoss user in that image.
Note the use of the --context-dir switch pointing to the directory containing the Dockerfile. This informs OpenShift that it needs to look in a sub-directory, not the root of the Git repository, for its build artifacts.
Step 6 : ENV "OPENSHIFT_BUILD_NAME" "fis-java-appdynamics-1" "OPENSHIFT_BUILD_NAMESPACE" "dev1" "OPENSHIFT_BUILD_SOURCE" "https://github.com/benemon/fis-java-appdynamics" "OPENSHIFT_BUILD_COMMIT" "d025f9961896b25fcae479d62779ae455df334d3"
Step 7 : LABEL "io.openshift.build.commit.message" "Updated the FIS build artifacts" "io.openshift.build.source-location" "https://github.com/benemon/fis-java-appdynamics" "io.openshift.build.source-context-dir" "src/main/docker" "io.openshift.build.commit.author" "Benjamin Holmes \[email protected]\u003e" "io.openshift.build.commit.date" "Sat Jul 9 11:26:11 2016 +0100" "io.openshift.build.commit.id" "d025f9961896b25fcae479d62779ae455df334d3" "io.openshift.build.commit.ref" "master"
NOTE: As an alternative to a standard Dockerfile build, we can use the Kubernetes Fluent DSL to generate the BuildConfig and ImageStream objects as part of a Template that will tell OpenShift to do a Dockerfile build based on the supplied project content. Using Kubernetes DSL is optional (you are more than welcome to define the objects manually), but as a Java developer this is a simple process to understand, it allows you to version control your whole image build process, and also falls nicely into the 'configuration as code' discipline so prominent in the DevOps world. An example of how to use the Fluent DSL is supplied in the Github repository for the AppDynamics base image.
Please note that lines have been removed from the above objects for the sake of brevity.
Once the build of the fis-java-appdynamics image has completed successfully, we will have a new base image present in our namespace that contains the AppDynamics agent plugin.
Given that I have elected to follow the second method of creating a new base image with the AppDynamics Java agent added to it, I now need a way of configuring it.
NOTE: These steps are much the same as those performed if you were to use the S2I builder process. However you can see the subtle differences, such as the addition of JAVA_OPTIONS is performed by the .s2i/bin/run scriptas opposed to the DeploymentConfig in the Template in the sample repository here.
The AppDynamics agent follows a similar agent model to many other profiling tools, in that it is added to a JVM using the -javaagent switch. When thinking in terms of immutable containers, we obviously want this whole configuration process to be as loosely coupled from the application image as possible.
With this in mind, the simplest way to configure the AppDynamics Java agent is via environment variables. This is helpful, as the AppDynamics agent prioritises environment variables over any other forms of configuration it may have available to it (such as controller-info.xml within the agent distribution). The AppDynamics Agent Configuration guide has further information.
One option we have here is to hard code all of the requisite environment variables into an application DeploymentConfig. However, in this brave new world of immutable containers, short lived cloud applications workloads, and CI/CD pipelines we can be a bit cleverer than that.
Using a Template for the application, we can still define all of the environment variables required by the AppDynamics agent, but we can also use a mixture of templated parameters, and Kubernetes' Downward API to effectively allow the container to introspect itself at runtime, and feed useful information about itself to the agent.
Note the use of Downward API reference for APPDYNAMICS_TIER_NAME and APPDYNAMICS_AGENT_NODE_NAME.
This creates the template within the current OpenShift project. This should be the same project in which you have done the fis-java-appdynamics build, otherwise OpenShift won't be able to locate the new base image!
When we present this via the OpenShift Web Console, we are shown a much more user friendly version of the above, allowing you to key in your AppDynamics account details without the need to store them in potentially troublesome static configuration files within the container.
Once all the fields have been completed, click on 'Create', and (assuming all mandatory fields have been filled in), a screen will be presented confirming the successful creation of all template objects.
Once this Template has been instantiated successfully, OpenShift will start a build against the source code branch, using fis-java-appdynamics as the S2I builder image.
Be aware that this project repository contains a standard Maven settings.xml which can be used to define how Maven resolves the build dependencies. If you experience long build times, this file can be updated to resolve to a local Maven repository, such as Sonatype Nexus, or JFrog Artifactory.
log4j:WARN No appenders could be found for logger (com.singularity.MissingMethodGenerator).
Started AppDynamics Java Agent Successfully.
Drilling down into the Application in the Dashboard also confirms that the Downward API has done its job, and we've automatically pulled in both the container name, and the Kubernetes namespace.
In OpenShift's container logs we can see these requests coming into the application, either via the Web Console or via the CLI.
We can even drill down into the Web Service endpoints themselves, and examine the levels of load each is experiencing.
We can also look at the load and response times being experienced by users of the application service. Whilst this particular view gives an amalgamation of data, it's a simple operation to drill down into an individual JVM to see how it's performing.
I have barely scratched the surface of what we can monitor, log, and alert on with OpenShift, Fuse Integration Services, and AppDynamics. Hopefully though, it gives you a glimpse of what is possible using the tools provided by the OpenShift Container Platform, and a template for not only integrating AppDynamics, but also other useful toolsets that follow a similar agent model.
Please note branches and tags in these repositories.
The post Exploring the AppDynamics Integration with OpenShift Container Platform appeared first on Application Performance Monitoring Blog | AppDynamics.
In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.
DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.