# Troubleshooting

## Logging

Our libraries use the Java logging API via `java.util.logging` package.
Configuring logging level reveals various facts that help your troubleshooting,
including:

- The timing of underlying client-server communication
- Request and response message headers
- Verbose messages in underlying dependency libraries

While there are various ways to configure the logging,
to quickly enable verbose logging for the Google Cloud Java libraries, create
a file `logging.properties` with the following content:

```
# run java program pointing to this properties file with the java arg
#   -Djava.util.logging.config.file=path/to/logging.properties
handlers=java.util.logging.ConsoleHandler
java.util.logging.SimpleFormatter.format=%1$tF %1$tT,%1$tL %4$-8s %3$-50s - %5$s %6$s%n

# --- ConsoleHandler ---
java.util.logging.ConsoleHandler.level=ALL
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
.level=INFO

# --- Specify logging level for certain packages ---
# com.google.api is for HTTP 1.1 layer
com.google.api.level=ALL
# io.grpc is for gRPC + Netty layer
io.grpc.level=FINE
# com.google.auth is for authentication
com.google.auth.level=FINE

# Example when we want to specify storge library's level. This works when
# the target Cloud library uses the logging API.
com.google.cloud.storage.level=INFO
```

and run your application with `-Djava.util.logging.config.file=path/to/logging.properties`
as the "VM argument" (not "Program argument").

If you use IntelliJ, you specify the VM argument in "Run/Debug Configuration":

![Screenshot of IntelliJ configuration](docs/logging_vm_options.png)

If the JVM of your program is running with the configuration correctly, you see
the FINE-level logging in your console. Example output:

```
2023-04-05 13:03:01,761 FINE     com.google.auth.oauth2.DefaultCredentialsProvider  - Attempting to load credentials from well known file: /usr/local/google/home/suztomo/.config/gcloud/application_default_credentials.json
2023-04-05 13:03:01,847 FINE     io.grpc.ManagedChannelRegistry                     - Unable to find OkHttpChannelProvider
java.lang.ClassNotFoundException: io.grpc.okhttp.OkHttpChannelProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
...
```

Note that you may see many stacktraces printed there.
As long as it's "FINE" level, they are not errors.

This is just one way to configure logging level.
For more details about Java logging API usage, see [Java Logging Overview](
https://docs.oracle.com/javase/8/docs/technotes/guides/logging/overview.html).

## ALPN is not configured properly

If you see exceptions related to `ALPN is not configured properly`, such as:

```
Caused by: java.lang.IllegalArgumentException: ALPN is not configured properly. See https://github.com/grpc/grpc-java/blob/master/SECURITY.md#troubleshooting for more information.
```

Please use the [compatibility checker](https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-util/google-cloud-compat-checker) to see if your environment is compatible with grpc-based clients. The incompatibility can mean that:
- You are not on a [supported platform](https://github.com/googleapis/google-cloud-java/#supported-platforms).
- There are classpath conflicts with `netty`.
- Or, you are seeing any of the conflicts specified in [gRPC Troubleshooting guide](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#troubleshooting).

## ClassNotFoundException, NoSuchMethodError, NoClassDefFoundError

These errors are usually caused by having multiple versions or conflicting versions of the same dependency in the classpath.
Usually these dependency conflicts occur with `guava` or `protobuf-java`.

There may be multiple sources for classpath conflicts:
- Multiple versions of the same transitive dependency in the dependency tree
- Your runtime classpath has different versions of dependencies than what you specified in the build

For example, if you have a direct or a transitive dependency on Guava version 19.0, 
and `google-cloud-java` uses Guava version 30.0, 
then `google-cloud-java` could be using Guava methods that don't exist in Guava 19.0,
and cause `NoSuchMethodError`.

Similarily, if your classpath has an older version of `protobuf-java`, 
but `google-cloud-java` requires a newer version,
 then you might see `NoClassDefFoundError` that fails to initialize `google-cloud-java` classes. For example:

```
java.lang.NoClassDefFoundError: Could not initialize class com.google.pubsub.v1.PubsubMessage$AttributesDefaultEntryHolder
```

### Validate the conflict

Check the dependency tree to see if you have multiple versions of the same dependencies:

```
$ mvn dependency:tree
```

Look for versions of potentially conflicting dependencies like `guava`, `protobuf-java`, etc.

If you experience the error only during runtime, then your runtime environment
might  be introducing conflicting JARs into your runtime classpath. A typical case
is that Hadoop, Spark, or other server software that your application runs on
has conflicting versions `netty`, `guava`, or `protobuf-java` JARs in the classpath.

### Detecting conflicts during build

To detect dependency linkage errors at compile time, add the 
[Linkage Checker Enforcer Rule](https://github.com/GoogleCloudPlatform/cloud-opensource-java/tree/master/enforcer-rules)
in your pom.xml:

```
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-enforcer-plugin</artifactId>
        <version>3.0.0-M3</version>
        <dependencies>
          <dependency>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>linkage-checker-enforcer-rules</artifactId>
            <version>1.5.7</version>
          </dependency>
        </dependencies>
        <executions>
          <execution>
            <id>enforce-linkage-checker</id>
            <!-- Important! Should run after compile -->
            <phase>verify</phase>
            <goals>
              <goal>enforce</goal>
            </goals>
            <configuration>
              <rules>
                <LinkageCheckerRule
                    implementation="com.google.cloud.tools.dependencies.enforcer.LinkageCheckerRule"/>
              </rules>
            </configuration>
          </execution>
        </executions>
      </plugin>
```

There is no way to detect runtime classpath conflicts though. You need to be fully
aware of which JARs/classes are included in the runtime classpath
as every server environment is different.

### Resolving the conflict

There are different strategies to resolve conflicts, but you must understand the root cause of the conflicts. For example:

- If you have control over the dependency tree, upgrade
  offending dependencies (for example, upgrading Guava version). This is the 
  least hackish approach, but it is a lot of work that can require multiple releases
   of multiple libraries to sync everything up. 
- If you can't modify and push new versions of your dependencies, import
  `com.google.cloud:libraries-bom:25.1.0` (or a more recent version) and use that to
  select consistent dependency versions. This is the easiest route. 
  For example, this is how you can depend on consistent versions of Guava and 
  `com.google.cloud:google-cloud-storage` without explicitly setting the version of either one:
  
```
  ...
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>libraries-bom</artifactId>
        <version>25.1.0</version>
        <type>pom</type>
        <scope>import</scope>
       </dependency>
     </dependencies>
  </dependencyManagement>
  ...
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
    </dependency>
    ...
  </dependencies>
  ...
```

- If changing dependency versions causes other failures, 
  consider [shading dependencies](https://maven.apache.org/plugins/maven-shade-plugin/)
  that conflict with `google-cloud-java`.

  For example, to shade `guava` and `protobuf-java`:

```
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>...</version>
    <executions>
      <execution>
        <phase>package</phase>
        <goals>
          <goal>shade</goal>
        </goals>
        <configuration>
          <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
          <relocations>
            <!-- move protobuf to a shaded package -->
            <relocation>
              <pattern>com.google.protobuf</pattern>
              <shadedPattern>myapp.shaded.com.google.protobuf</shadedPattern>
            </relocation>
            <!-- move Guava to a shaded package -->
            <relocation>
              <pattern>com.google.common</pattern>
              <shadedPattern>myapp.shaded.com.google.common</shadedPattern>
            </relocation>
          </relocations>
        </configuration>
      </execution>
    </executions>
  </plugin>
```