java.net.UnknownHostException: metadata.google.internal

[vhnguyenae]

Hi, my Spark application used to works fine on my local machine, but today after 2~3 months, I relaunch it and encounter this error below (stack trace in detail).

gcsConnector version: 3.0.4 (even if I update to latest version 3.1.2 I get the same error)

sparkConfig:

spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
google.cloud.auth.type=SERVICE_ACCOUNT_JSON_KEYFILE
spark.hadoop.google.cloud.auth.service.account.enable=true
spark.hadoop.google.cloud.auth.service.account.json.keyfile=${PathToMyServiceAccountJsonFile}

Error in stacktrace:

Caused by: java.io.IOException: Error accessing gs://vh-wat-test-eu-central-1-crowley-results/import/contents/project_hash=p9d5fc41d58cbc82b7e3e36097649c71/crawl_id=241128-0342
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:2184)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfo(GoogleCloudStorageImpl.java:2073)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystemImpl.listFileInfo(GoogleCloudStorageFileSystemImpl.java:932)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.lambda$listStatus$13(GoogleHadoopFileSystem.java:833)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
        at com.google.cloud.hadoop.fs.gcs.GhfsGlobalStorageStatistics.trackDuration(GhfsGlobalStorageStatistics.java:114)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.trackDurationWithTracing(GoogleHadoopFileSystem.java:764)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.listStatus(GoogleHadoopFileSystem.java:816)
        at org.apache.hadoop.fs.Globber.listStatus(Globber.java:128)
        at org.apache.hadoop.fs.Globber.doGlob(Globber.java:291)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2142)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.lambda$concurrentGlobInternal$32(GoogleHadoopFileSystem.java:1546)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: ComputeEngineCredentials cannot find the metadata server. This is likely because code is not running on Google Compute Engine.
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.ComputeEngineCredentials.getMetadataResponse(ComputeEngineCredentials.java:366)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:271)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:270)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:267)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$RefreshTask.run(OAuth2Credentials.java:635)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:582)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:233)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:183)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.RetryHttpInitializer.initialize(RetryHttpInitializer.java:80)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize(ChainingHttpRequestInitializer.java:52)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:455)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:565)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.listStorageObjectsAndPrefixesPage(GoogleCloudStorageImpl.java:1425)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.listObjectInfoPage(GoogleCloudStorageImpl.java:1598)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystemImpl.listFileInfoForPrefixPage(GoogleCloudStorageFileSystemImpl.java:891)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.flatGlobInternal(GoogleHadoopFileSystem.java:1576)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.lambda$concurrentGlobInternal$31(GoogleHadoopFileSystem.java:1545)
        ... 6 more
Caused by: java.net.UnknownHostException: metadata.google.internal
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
        at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
        at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
        at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.auth.oauth2.ComputeEngineCredentials.getMetadataResponse(ComputeEngineCredentials.java:364)
        ... 28 more

I noticed the message ComputeEngineCredentials cannot find the metadata server. This is likely because code is not running on Google Compute Engine and java.net.UnknownHostException: metadata.google.internal
But I'm not running the application on GCE, why does it try to access the metadata google internal server?
Can someone help me to debug? Let me know if you need more information 😅

[surjikal]

By default, the connector uses COMPUTE_ENGINE auth, that's why it hits the metadata server; It looks like the google.cloud.auth.type setting isn't effective.

Try with:

spark.hadoop.fs.gs.auth.type=SERVICE_ACCOUNT_JSON_KEYFILE
spark.hadoop.fs.gs.auth.service.account.json.keyfile=${PathToMyServiceAccountJsonFile}

[vhnguyenae]

Ah nice thank you, the config spark.hadoop.fs.gs.auth.type=SERVICE_ACCOUNT_JSON_KEYFILE worked 👍,
But now I have this new error: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Options$OpenFileOptions

It's similar to #1099 (I also saw you commented in that thread)

Noted that if I use the version: hadoop2-2.2.27 it worked, but not with 3.1.2 not 3.0.4. I always use shaded version since I assemble the jar file for my Spark app.

Do you have any idea why or how to fix it?

25/05/14 12:53:58 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Options$OpenFileOptions
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.openFileWithOptions(GoogleHadoopFileSystem.java:1322)
        at org.apache.hadoop.fs.FileSystem$FSDataInputStreamBuilder.build(FileSystem.java:4768)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:92)
        at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.<init>(HadoopFileLinesReader.scala:63)
        at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.readFile(JsonDataSource.scala:132)
        at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.$anonfun$buildReader$2(JsonFileFormat.scala:126)
        at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:155)
        at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:140)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:217)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:279)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithKeys_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.Options$OpenFileOptions
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 31 more

[surjikal]

@vhnguyenae Yeah...

To fix that one, I installed hadoop (brew install hadoop on macos), then I set up the following env vars in my dotfiles:

# Uncomment this, if you don't have HOMEBREW_PREFIX in your env already... or hardcode it I guess!
# HOMEBREW_PREFIX=$(brew --prefix)

SPARK_DIST_CLASSPATH=$(hadoop classpath)
export SPARK_DIST_CLASSPATH
export HADOOP_HOME="${HADOOP_HOME:-"${HOMEBREW_PREFIX}/opt/hadoop/libexec"}"
export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-"${HADOOP_HOME}/etc/hadoop"}"

The above fixed spark for me. I think spark-env.sh is the "proper" way of doing it (as opposed to dotfiles):
https://spark.apache.org/docs/latest/hadoop-provided.html

I really didn't want to have to install hadoop, so I spent some time trying to figure out if there was a magic jar / dependency combo I could use to make it all work. I couldn't figure it out.

[Ferdinanddb]

I encountered this issue as well. The workaround of installing hadoop and setting the SPARK_DIST_CLASSPATH variable did the trick but I was not really happy with that since I did not want to install hadoop (within my Spark container image).

My trick was to build my Spark container and install the hadoop-common-3.3.6.jar (previous versions does not have OpenFileOptions so choose version >= 3.3.6 !):

FROM apache/spark:3.5.6-java17-python3

USER root

RUN wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/3.3.6/hadoop-common-3.3.6.jar && mv hadoop-common-3.3.6.jar /opt/spark/jars/

# ...

USER spark

After that, I can run Spark jobs with the gcs-connector shaded jar without errors, for example with the following setting:
"spark.jars": "https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/3.1.1/gcs-connector-3.1.1-shaded.jar".

Hope this will help someone.

[djice]

By default, the connector uses COMPUTE_ENGINE auth, that's why it hits the metadata server; It looks like the google.cloud.auth.type setting isn't effective.

Try with:

spark.hadoop.fs.gs.auth.type=SERVICE_ACCOUNT_JSON_KEYFILE
spark.hadoop.fs.gs.auth.service.account.json.keyfile=${PathToMyServiceAccountJsonFile}

Hi,

I have the same issue in the same context (spark). I have a

User class threw exception: java.io.UncheckedIOException: java.io.FileNotFoundException: /user/test/apps/conf/google_cloud_auth.json (No such file or directory) at com.google.cloud.spark.bigquery.SparkBigQueryUtil.createGcsPath(SparkBigQueryUtil.java:141)

I put the json file on HDFS...

My fear is that this file must be copied on each spark executors. Is it true ?

Thanks