Giter Club home page Giter Club logo

cloud-debug-java's Introduction

Java Snapshot Debugger Agent

Snapshot debugger agent for Java 7, Java 8 and Java 11.

Project Status: Archived

This project has been archived and is no longer supported. There will be no further bug fixes or security patches. The repository can be forked by users if they want to maintain it going forward.

Overview

Snapshot Debugger lets you inspect the state of a running cloud application, at any code location, without stopping or slowing it down. It is not your traditional process debugger but rather an always on, whole app debugger taking snapshots from any instance of the app.

Snapshot Debugger is safe for use with production apps or during development. The Java debugger agent adds less than 10ms to the request latency when a debug snapshot is captured. In most cases, this is not noticeable to users. Furthermore, the Java debugger agent does not allow modification of application state in any way, and has close to zero impact on the app instances.

Snapshot Debugger attaches to all instances of the app providing the ability to take debug snapshots and add logpoints. A snapshot captures the call-stack and variables from any one instance that executes the snapshot location. A logpoint writes a formatted message to the application log whenever any instance of the app executes the logpoint location.

The Java debugger agent is only supported on Linux at the moment. It was tested on Debian Linux, but it should work on other distributions as well.

Snapshot Debugger consists of 3 primary components:

  1. The Java debugger agent (requires Java 7 and above).
  2. A Firebase Realtime Database for storing and managing snapshots/logpoints. Explore the schema.
  3. User interface, including a command line interface snapshot-dbg-cli and a VSCode extension

Installation

The easiest way to install the Java debugger agent is to download the pre-built package from the Internet.

For most enviroments, use:

mkdir /opt/cdbg
wget -qO- https://github.com/GoogleCloudPlatform/cloud-debug-java/releases/latest/download/cdbg_java_agent_gce.tar.gz | \
    tar xvz -C /opt/cdbg

For Google App Engine Java 8 Standard Environment:

mkdir /opt/cdbg
wget -qO- https://github.com/GoogleCloudPlatform/cloud-debug-java/releases/latest/download/cdbg_java_agent_gae_java8.tar.gz | \
    tar xvz -C /opt/cdbg

Alternatively you can build the Java debugger agent from source code:

git clone https://github.com/GoogleCloudPlatform/cloud-debug-java.git
cd cloud-debug-java
chmod +x build.sh
./build.sh

# For all supported environments other than GAE Java8 Standard:
ls cdbg_java_agent_gce.tar.gz

# For Google App Engine Java 8 Standard Environment:
ls cdbg_java_agent_gae_java8.tar.gz

Note that the build script assumes some dependencies. To install these dependencies, run this command:

On Debian 9:

sudo apt-get -y -q --no-install-recommends install \
    curl gcc build-essential libssl-dev unzip openjdk-8-jdk \
    cmake python maven

Alpine Linux

The Java agent is not regularly tested on Alpine Linux, and support will be on a best effort basis. The Dockerfile shows how to build a minimal image with the agent installed.

Historical note

Version 3.x of this agent supported both the now shutdown Cloud Debugger service (by default) and the Snapshot Debugger (Firebase RTDB backend) by setting the use_firebase flag to true. Version 4.0 removed support for the Cloud Debugger service, making the Snapshot Debugger the default. To note the use_firebase flag is now obsolete, but still present for backward compatibility.

Setup

The Java debugger agent is a JVMTI agent that needs to be enabled when JVM starts with the -agentpath option of the Java launcher. Most of the debugger options are configured through system properties.

For example:

java -agentpath:/opt/cdbg/cdbg_java_agent.so -jar ~/myapp.jar

By default the Java debugger agent assumes that it runs on Google Cloud Platform and obtain the credentials from the local metadata service. To use the Java debugger agent outside Google Cloud Platform requires setting up a service account.

You can customize the behavior of the agent by passing arguments to it. Multiple arguments can be passed by separating them using commas without spaces, as follows:

java -agentpath:/opt/cdbg/cdbg_java_agent.so=--arg1=val1,--arg2=val2 -jar ~/myapp.jar

Configuring the Firebase Realtime Database URL

It may be necessary to specify the database's URL, which can be done as follows:

-Dcom.google.cdbg.agent.firebase_db_url=https://my-database-url.firebaseio.com

or

-agentpath:/opt/cdbg/cdbg_java_agent.so=--firebase_db_url=https://my-database-url.firebaseio.com

By default the Java agent will check for a configured database first at https://PROJECT_ID-cdbg.firebaseio.com, and then failing that https://PROJECT_ID-default-rtdb.firebaseio.com. If your database has an address different from either of those, the URL needs to be specified. In general if either of the flags --database-id or --location were specfied when running the snapshot-dbg-cli init command to create the database this will be necessary.

Application Servers

Java application servers usually start through a bootstrap process, and each application server has its own way of customizing Java options.

Tomcat

Add this line to /etc/default/tomcat7 or /etc/default/tomcat8:

JAVA_OPTS="${JAVA_OPTS} -agentpath:/opt/cdbg/cdbg_java_agent.so"

If you run Tomcat in a Docker container, add this line to Dockerfile instead:

ENV JAVA_OPTS -agentpath:/opt/cdbg/cdbg_java_agent.so

Jetty

Add cdbg.ini file to /var/lib/jetty/start.d:

--exec
-agentpath:/opt/cdbg/cdbg_java_agent.so

Extra Classpath

The Java debugger agent needs to be able to find the application classes when it's running in an application server like Tomcat or Jetty. By default, it looks for the exploded root war directory. In other words, if you deployed a ROOT.war in Tomcat, the agent can find it without additional configuration.

However, if you deployed your WAR file with a different name (e.g., myapp.war), or that the exploded WAR directory is not under the default exploded root war directory (e.g., your exploded war is under /opt/tomcat/webapps/myapp), then you must let the agent know the full path to your application's classes using the cdbg_extra_class_path parameter.

-agentpath:/opt/cdbg/cdbg_java_agent.so=--cdbg_extra_class_path=/opt/tomcat/webapps/myapp/WEB-INF/classes

You can specify multiple paths by using a : (colon) as the path delimiter.

-agentpath:/opt/cdbg/cdbg_java_agent.so=--cdbg_extra_class_path=/opt/tomcat/webapps/myapp/WEB-INF/classes:/another/path/with/classes

Naming and Versioning

Developers can run multiple applications and versions at the same time within the same Google Cloud Platform project. You should tag each app version with the Cloud Debugger to uniquely identify it in the Cloud Debugger user interface.

To tag the application and it's version, please add these system properties:

-Dcom.google.cdbg.module=my-app-name
-Dcom.google.cdbg.version=my-app-version

Use module to name your application (or service). Use version to name the app version (e.g. build version). The UI will display the running version as module - version.

Logging

By default the Java debugger aget writes its logs to cdbg_java_agent.INFO file in the default logging directory. It is possible to change the log directory as following:

-agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/my/log/dir

Alternatively you can make the Java Cloud Debugger log to stderr:

-agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1

Service Account

To use the Java debugger agent on machines not hosted by Google Cloud Platform, the agent must use a Google Cloud Platform service account credentials to authenticate with the Cloud Debugger Service.

Use the Google Cloud Console Service Accounts page to create a credentials file for an existing or new service account. The service account must have at least the Stackdriver Debugger Agent role. If you don't have a Google Cloud Platform project, you can create one for free on Google Cloud Console.

Once you have the service account JSON file, deploy it alongside the Java debugger agent.

To use the service account credentials add this system property:

-Dcom.google.cdbg.auth.serviceaccount.jsonfile=/opt/cdbg/gcp-svc.json

Alternatively, you can set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the JSON file path instead of adding the auth.serviceaccount.jsonfile system property.

Other JVM Languages

Scala

Debugging Scala applications is supported; however, expressions and conditions must be written using the Java programming language syntax.

Kotlin

Debugging Kotlin applications is supported; however, expressions and conditions must be written using the Java programming language syntax.

Many Kotlin-specific features can be used in conditions and expressions with simple workarounds:

// Main.kt
private fun getGreeting() {
  return "Hello world!"
}
class Main {
  companion object {
    fun welcome() {
      return getGreeting()
    }
  }
}

Package-level functions can be accessed by qualifying them with the name of the file and a Kt suffix. For instance, the getGreeting function above can be used in an expression as MainKt.getGreeting()

Companion object methods can be accessed by qualifying them with the Companion keyword. For instance, the welcome function above can be used in an expression as Main.Companion.welcome()

cloud-debug-java's People

Contributors

b-daniels avatar cbaueratwork avatar cushon avatar dlj-nan avatar dragoluigi avatar emrekultursay avatar erezhaba avatar gk5885 avatar jasonborg avatar joaoandremartins avatar kluever avatar kssilveira avatar laramiel avatar lesv avatar louis-ye avatar maximgoldin avatar mctavish avatar mkanat avatar nlewycky avatar oquenchil avatar punzki avatar r4nt avatar rluble avatar rogeeff avatar rrch avatar shengwu avatar shijun-z avatar vladlf avatar vladmos avatar xinghuadou-google avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cloud-debug-java's Issues

GceMetadataQuery failures when upgrading to App Engine standard + Java 11 bundled services

We are currently migrating our App Engine standard application from Java 8 to Java 11 using the bundled services (https://cloud.google.com/blog/products/serverless/support-for-app-engine-services-in-second-generation-runtimes).

We made it as far as that the application seems to be running as expected.
The following errors are showing up in our logs every 10 seconds, however:

Failed to query GCE metadata service

and then

java.io.IOException: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:70)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:609)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:488)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.readResponse(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.queryMetadataAttribute(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GceMetadataQuery.getProjectId(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GcpHubClient.registerDebuggee(Unknown Source)
Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call urlfetch.Fetch in a thread that is neither the original request thread nor a thread created by ThreadManager
at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:844)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:117)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:54)
... 6 more

The App Engine standard Cloud Debugger setup for Java (https://cloud.google.com/debugger/docs/setup/java#gae-standard) says that no configuration is required.

So is this a bug or is there some configuration required after all?

cdbg_extra_class_path read before Tomcat deploys the war file.

The documentation describes the way to add extra Classpath for Tomcat applications. Following the instruction I would get an "File was not found in the executable" error from the Stackdriver UI. It looks like the agent loads the extra ClassPath files before Tomcat deployed the war file.

UnsupportedOperationException on Java 11

When using the agent in a Java 11 binary, I gete an UnsupportedOperationException when trying to resolve a source location

Java exception Ljava/lang/UnsupportedOperationException; thrown at ClassPathLookup.resolveSourceLocation: java.lang.UnsupportedOperationException

Unfortunately there's no stack trace logged.

The README mentions Java 7+ are supported, but I wonder if it fails on Java 10+, which changed permissions for mucking with the class loader.

Add support for dockerized apps

It would be very helpful to have either instructions or just a docker image to use cloud debugger for applications running in a dockerized environment

GKE - Debugger crashes on boot

Hi there,

I'm trying to run the java cloud debugger inside GKE.
Unfortunately, once the VM starts, the cloud debugger crashes with:

I0724 13:56:49.782296 36 jni_logger.cc:31] Initializing ClassPathLookup, default classpath: true, extra classpath: []
I0724 13:56:49.881603 36 jni_logger.cc:31] Total size of indexed resources database: 3092 bytes
I0724 13:56:49.902678 36 jvm_internals.cc:131] ClassPathLookup constructor time: 159468 microseconds
I0724 13:56:49.903113 36 yaml_data_visibility_config_reader.cc:37] debugger-blacklist.yaml was not found. Using default settings.
E0724 13:56:49.920346 36 jni_utils.h:372] GcpHubClient.: java.lang.RuntimeException: Failed to initialize service account authentication
at com.google.devtools.cdbg.debuglets.java.GcpEnvironment.getMetadataQuery(Unknown Source)
at com.google.devtools.cdbg.debuglets.java.GcpHubClient.(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.google.devtools.cdbg.debuglets.java.ServiceAccountAuth
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at com.google.devtools.cdbg.debuglets.java.InternalsClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
... 2 more
E0724 13:56:49.925649 36 jni_bridge.cc:50] Failed to instantiate HubClient Java class
E0724 13:56:49.925870 36 worker.cc:145] HubClient not available: debugger thread can't continue.
I0724 13:56:49.925930 36 jvmti_agent_thread.cc:103] Agent thread exited: CloudDebugger_main_worker_thread

Any ideas how to get this running?

encoding_util.cc prevents compilation on macOS

A change introduced between v2.16 and 2.18 contributes a new encoding_util.cc file. This file does not compile cleanly on macOS (Xcode 8 specifically) failing with the following error:

encoding_util.cc:74:20: error: comparison of constant 128 with expression of type 'char' is always true [-Werror,-Wtautological-constant-out-of-range-compare]
    } else if (cur < 0x80) {
               ~~~ ^ ~~~~
1 error generated.
make: *** [encoding_util.o] Error 1

For support in Cloud Foundry we'd like to continue to be able to compile on macOS and would like an update to the code to make this a bit more portable.

Archive this repository

As stated in the readme, this product will be unsupported after 31 August 2023 and will be archived.

This issue will track any remaining cleanup on the repository and the archival.

Note that the archival will not prevent use, and users are welcomed to fork the repository in order to fix bugs, update dependencies, or add features.

Passing certain agentlib settings causes the agent not to work

I was testing Google Cloud Debugger and had left these jvm parameters in the invocation of my application:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

These settings are recommended by IntelliJ when doing remote debugging, using the IntelliJ debugger.
Trying to create a snapshot resulted in this unhelpful error in the Google Cloud Debugger UI:
Internal error at jvm_breakpoint.cc:641
.. and these logs in the container:

I0406 11:00:14.996963    29 jvm_breakpoints_manager.cc:153] Setting new breakpoint: 5dbf896f4d957-87a1-177f
I0406 11:00:14.997048   151 jvmti_agent_thread.cc:95] Agent thread started: CloudDebugger_transmission_thread
I0406 11:00:15.012601    29 jvm_breakpoint.cc:263] Breakpoint 5dbf896f4d957-87a1-177f initialized to pending state, path: example-cloud-app/src/main/java/com/bol/axlexamplecloud/camping/api/CampingController.java, line number: 31, resolved class signature: Lcom/bol/axlexamplecloud/camping/api/CampingController;, resolved method name: getAllCampings, adjusted line number: 31
E0406 11:00:15.012631    29 jvm_breakpoint.cc:105] GetLineNumberTable failed, error: 99
E0406 11:00:15.012635    29 jvm_breakpoint.cc:631] Resolved source location not found, class signature: Lcom/bol/axlexamplecloud/camping/api/CampingController;, method: getAllCampings, adjusted line: 31
I0406 11:00:15.012676    29 jvm_breakpoints_manager.cc:263] Breakpoint 5dbf896f4d957-87a1-177f removed from active breakpoints list

It's unclear to me how the agentlib parameters disrupt the agent. Some way of communicating this back to the user (docs, error message, startup failure), could prove useful.

Can´t get it to run on Amazon Linux

I can´t get it to run on Amazon Linux.

Reported message:

Error opening zip file or JAR manifest missing : /opt/cdbg/cdbg_java_agent.so
Error occurred during initialization of VM
agent library failed to init: instrument

Has anyone successfully run it on AWS Linux?

Best

André

lazy start agent

we running our app in GKE, with google-appengine/openjdk image. with debugger and profiler enabled
starting application now takes 2x times longer . Is there a way we can speed up the start time by doing some lazy start on debugger and profilers

Project ID being injected from SA

The projectID where the debugger registers is being picked up from service account json file.

@Key("project_id") private String projectId = null;

This creates a problem when you have service account from different project but want to use debugger in another project. Providing environment variable PROJECT_ID should override the projectId from service account .

Add support for JSON service account key

Currently the debugger only takes in p12 key, which then necessitates all of the service account metadata properties to be specified at the command line.

It'd be great to add the JSON key option.

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

  • build/jarjar.jar
  • third_party/antlr/antlr_tool.jar
  • third_party/antlr/libantlr.jar

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.


⚠️ There is an updated version of this policy result! Click here to see the latest update


Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Include source version information

I have this working but the Google Cloud Debug console says it cannot find the source version information. All works well if I point it at local files but it would be great if I can point it at my Cloud Repository.

In the docs it mentions generating source-context.json. The command doesn't seem to exist anymore. Has this been deprecated?

Breakpoint expiry not working as expected when talking to Firebase Backend

The internals of the Java Agent are looking for a 'createTime' field when computing the expiration time for breakpoints. In Snapshot Debugger mode (talking to a Firebase backend), the field is actually called 'createTimeUnixMsec', and has a different format. As a result the agent simply uses the current time as the create time.

Agent never shows up in list of applications when using with Cloud Run

I'm trying to use the java cloud debugger with cloud run and no matter what I do I can't seem to get the applications to show up. I've followed the instructions here and my application starts up without any indication if the agent started or not. I am struggling to figure out how to debug this in order to get the debugger to work.

Screen Shot 2021-10-29 at 7 24 42 PM

I've created an out of the box Quarkus app with the same configuration as my real application minus my project ID and cloud run service ID in the cloudbuild.yaml

quarkus-cloud-debugger.zip

I would love some help to figure out what needs to be done in order to use the debugger.

Support GKE environments where Service Accounts are in a different Google Cloud project than the container

We'd like to use cloud-debug-java in our GKE environment, where the containers run in a GKE cluster in Google cloud project A, but the Service Account lives in Google cloud project B.

We use Workload Identity.

The debugger agent will contact the GKE metadata service to retrieve the project ID, which will return the project ID where the container is running, project ID A.
With the Service Account being in project ID B, permission will be denied when calling debuggees/register.

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "status": "PERMISSION_DENIED"
  }
}

In order to work around this, we've patched the agent to allow overriding both the project ID and project number.

I see someone in the past also ran into this: #15
The solution there was to use a second service account for debugging. This approach however works counter to one of the main benefits of using Workload Identity, which is reducing the (security) pain of managing service account secrets.

Since our company adopted Workload Identity, using service account keys has been banished as much as possible.
Issue via Google Cloud support https://issuetracker.google.com/issues/228214070

`bazel test` fails under Bazel 6

With Bazel 6, bazel test //tests/... fails. The output contains the following error messages:

ERROR: .../external/bazel_tools/platforms/BUILD:89:6: in alias rule @bazel_tools//platforms:windows: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
ERROR: .../external/bazel_tools/platforms/BUILD:89:6: Analysis of target '@bazel_tools//platforms:windows' failed
ERROR: .../external/com_google_googletest/BUILD.bazel:116:11: errors encountered resolving select() keys for @com_google_googletest//:gtest_main

Under Bazel 5.4 everything works.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.