PretrainedPipeline error #6152

mingsea · 2021-09-23T22:36:24Z

mingsea
Sep 23, 2021

Hi,

I tried to use Spark NLP with jupyter notebook, following instructions and sample code in the NLP link: https://github.com/JohnSnowLabs/spark-nlp. However, i found errors "Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize.
: java.lang.NullPointerException". in calling PretrainedPipeline as below: Any help? Thank you very much.

My system settings:
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
Spark version: 3.1.2
SparkNLP version: 3.2.3
python version: 3.7.5

recognize_entities_dl download started this may take some time.
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-1-d228dfeffc49> in <module>
     14 
     15 # Download a pre-trained pipeline
---> 16 pipeline = PretrainedPipeline('recognize_entities_dl', lang='en')
     17 
     18 # Your testing dataset

/opt/anaconda3/envs/test/lib/python3.7/site-packages/sparknlp/pretrained.py in __init__(self, name, lang, remote_loc, parse_embeddings, disk_location)
    136     def __init__(self, name, lang='en', remote_loc=None, parse_embeddings=False, disk_location=None):
    137         if not disk_location:
--> 138             self.model = ResourceDownloader().downloadPipeline(name, lang, remote_loc)
    139         else:
    140             self.model = PipelineModel.load(disk_location)

/opt/anaconda3/envs/test/lib/python3.7/site-packages/sparknlp/pretrained.py in downloadPipeline(name, language, remote_loc)
     70     def downloadPipeline(name, language, remote_loc=None):
     71         print(name + " download started this may take some time.")
---> 72         file_size = _internal._GetResourceSize(name, language, remote_loc).apply()
     73         if file_size == "-1":
     74             print("Can not find the model to download please check the name!")

/opt/anaconda3/envs/test/lib/python3.7/site-packages/sparknlp/internal.py in __init__(self, name, language, remote_loc)
    230     def __init__(self, name, language, remote_loc):
    231         super(_GetResourceSize, self).__init__(
--> 232             "com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize", name, language, remote_loc)
    233 
    234 

/opt/anaconda3/envs/test/lib/python3.7/site-packages/sparknlp/internal.py in __init__(self, java_obj, *args)
    163         super(ExtendedJavaWrapper, self).__init__(java_obj)
    164         self.sc = SparkContext._active_spark_context
--> 165         self._java_obj = self.new_java_obj(java_obj, *args)
    166         self.java_obj = self._java_obj
    167 

/opt/anaconda3/envs/test/lib/python3.7/site-packages/sparknlp/internal.py in new_java_obj(self, java_class, *args)
    173 
    174     def new_java_obj(self, java_class, *args):
--> 175         return self._new_java_obj(java_class, *args)
    176 
    177     def new_java_array(self, pylist, java_class):

/opt/spark-3.1.2/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
     64             java_obj = getattr(java_obj, name)
     65         java_args = [_py2java(sc, arg) for arg in args]
---> 66         return java_obj(*java_args)
     67 
     68     @staticmethod

/opt/anaconda3/envs/test/lib/python3.7/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1303         answer = self.gateway_client.send_command(command)
   1304         return_value = get_return_value(
-> 1305             answer, self.gateway_client, self.target_id, self.name)
   1306 
   1307         for temp_arg in temp_args:

/opt/spark-3.1.2/python/pyspark/sql/utils.py in deco(*a, **kw)
    109     def deco(*a, **kw):
    110         try:
--> 111             return f(*a, **kw)
    112         except py4j.protocol.Py4JJavaError as e:
    113             converted = convert_exception(e.java_exception)

/opt/anaconda3/envs/test/lib/python3.7/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize.
: java.lang.NullPointerException
	at com.amazonaws.ClientConfiguration.getProxyPasswordEnvironment(ClientConfiguration.java:950)
	at com.amazonaws.ClientConfiguration.getProxyPassword(ClientConfiguration.java:977)
	at com.amazonaws.ClientConfiguration.<init>(ClientConfiguration.java:377)
	at com.amazonaws.client.builder.AwsClientBuilder.resolveClientConfiguration(AwsClientBuilder.java:169)
	at com.amazonaws.client.builder.AwsClientBuilder.access$000(AwsClientBuilder.java:54)
	at com.amazonaws.client.builder.AwsClientBuilder$SyncBuilderParams.<init>(AwsClientBuilder.java:505)
	at com.amazonaws.client.builder.AwsClientBuilder.getSyncClientParams(AwsClientBuilder.java:441)
	at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
	at com.johnsnowlabs.client.aws.AWSGateway.getAmazonS3Client(AWSGateway.scala:72)
	at com.johnsnowlabs.client.aws.AWSGateway.client$lzycompute(AWSGateway.scala:50)
	at com.johnsnowlabs.client.aws.AWSGateway.client(AWSGateway.scala:39)
	at com.johnsnowlabs.client.aws.AWSGateway.getMetadata(AWSGateway.scala:77)
	at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.downloadMetadataIfNeed(S3ResourceDownloader.scala:62)
	at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.resolveLink(S3ResourceDownloader.scala:68)
	at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.getDownloadSize(S3ResourceDownloader.scala:145)
	at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.getDownloadSize(ResourceDownloader.scala:372)
	at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader$.getDownloadSize(ResourceDownloader.scala:483)
	at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize(ResourceDownloader.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

maziyarpanahi · 2021-09-24T08:26:03Z

maziyarpanahi
Sep 24, 2021
Maintainer

Hi,
Could you please share

what is your operating system?
how do you install PyPI spark-nlp (conda or pypi)
and please provide your full code?

0 replies

mingsea · 2021-09-24T16:30:25Z

mingsea
Sep 24, 2021
Author

Hi, * The operating system is macOS Catalina * I used "pip install spark-nlp==3.2.3 pyspark" to install spark-nlp * source code: (I follow exact link: https://github.com/JohnSnowLabs/spark-nlp) # Import Spark NLP from sparknlp.base import * from sparknlp.annotator import * from sparknlp.pretrained import PretrainedPipeline import sparknlp # Start SparkSession with Spark NLP spark = sparknlp.start() # Download a pre-trained pipeline pipeline = PretrainedPipeline('explain_document_dl', lang='en') # Your testing dataset text = """ The Mona Lisa is a 16th century oil painting created by Leonardo. It's held at the Louvre in Paris. """ # Annotate your testing dataset result = pipeline.annotate(text) Thanks, Will

…

On Fri, Sep 24, 2021 at 1:26 AM Maziyar Panahi ***@***.***> wrote: Hi, Could you please share - what is your operating system? - how do you install PyPI spark-nlp (conda or pypi) - and please provide your full code? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6152 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKXE33YSQDKOLMCE2JVHYLUDQY2LANCNFSM5EUV53CQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

15 replies

mingsea Oct 1, 2021
Author

@maziyarpanahi Any more help? Thank you very much

maziyarpanahi Oct 2, 2021
Maintainer

@mingsea It seems to be about proxy and nothing being able to download the pipeline now.

do you have anything inside ls ~/.aws just to check
let's try this offline to be sure there is nothing else wrong:
Downloading the pipeline from here: https://nlp.johnsnowlabs.com/2021/03/23/explain_document_dl_en.html
extracting the pipeline (just unziping it)
and use the following code instead:

from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
import sparknlp

# Start SparkSession with Spark NLP
spark = sparknlp.start()

# path to the extracted pretrained pipeline, for instance
pipeline = PretrainedPipeline.fromDisk('~/Downlaods/explain_document_dl_en_3.0.0_3.0_1626763223904')

mingsea Oct 3, 2021
Author

Thanks @maziyarpanahi . I followed your instructions and tried to download from the below link, but failed it with following error: Any comments? Thanks.

https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/explain_document_dl_en_3.0.0_3.0_1626763223904.zip

This XML file does not appear to have any style information associated with it. The document tree is shown below. <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>KXHE79D8XW00CJ2K</RequestId> <HostId>uiVgFO3NY5huBOKOCzKuHoykWg3c+WNxpfNV3E4Tak4Q/Qi+crcqVnRpzr6L/XbuSwRkt//3nZ0=</HostId> </Error>

maziyarpanahi Oct 4, 2021
Maintainer

Sorry about that, the URL was wrong on that page, you can try again with the new URL (I fixed the URL on that page)

https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/explain_document_dl_en_3.0.0_3.0_1616473268265.zip

mingsea Oct 6, 2021
Author

It works now. Thank you very much. Do you know what is best pretrained model for multiple-classification, like customer feedback? Thanks @maziyarpanahi

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PretrainedPipeline error #6152

{{title}}

Replies: 2 comments 15 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

PretrainedPipeline error #6152

mingsea Sep 23, 2021

Replies: 2 comments · 15 replies

maziyarpanahi Sep 24, 2021 Maintainer

mingsea Sep 24, 2021 Author

mingsea Oct 1, 2021 Author

maziyarpanahi Oct 2, 2021 Maintainer

mingsea Oct 3, 2021 Author

maziyarpanahi Oct 4, 2021 Maintainer

mingsea Oct 6, 2021 Author

mingsea
Sep 23, 2021

Replies: 2 comments 15 replies

maziyarpanahi
Sep 24, 2021
Maintainer

mingsea
Sep 24, 2021
Author

mingsea Oct 1, 2021
Author

maziyarpanahi Oct 2, 2021
Maintainer

mingsea Oct 3, 2021
Author

maziyarpanahi Oct 4, 2021
Maintainer

mingsea Oct 6, 2021
Author