Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datapusher not working on my local enviroment #73

Open
JeffreyVRiemsdijk opened this issue Jun 10, 2021 · 2 comments
Open

Datapusher not working on my local enviroment #73

JeffreyVRiemsdijk opened this issue Jun 10, 2021 · 2 comments

Comments

@JeffreyVRiemsdijk
Copy link

My environment is localhost with nothing changed except only these two settings in my hosts file on windows to try and find out if it worked but sadly enough both solutions do not work:

127.0.0.1 ckan
127.0.0.1 datapusher

After uploading a dataset i get this:

ckan-dev_1    | 2021-06-10 17:12:11,273 DEBUG [ckanext.datapusher.plugin] Submitting resource 1caafc34-ef92-4797-aaf2-d18693f9cfc4 to DataPusher
solr          | 2277933 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=name:"695790ff-759d-4f73-8eb4-4552c526b64f"+OR+id:"695790ff-759d-4f73-8eb4-4552c526b64f"&fq=site_id:"default"&rows=1&wt=json} hits=1 status=0 QTime=2
datapusher    | [pid: 11|app: 0|req: 1/1] 192.168.96.6 () {36 vars in 435 bytes} [Thu Jun 10 17:12:11 2021] POST /job => generated 700 bytes in 101 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)
solr          | 2278253 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
solr          | 2278253 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7aa047f1    
solr          | 2278601 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
solr          |         commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr/server/solr/ckan/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@336e3983; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_11,generation=37}
solr          |         commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr/server/solr/ckan/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@336e3983; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_12,generation=38}
solr          | 2278601 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.SolrDeletionPolicy newest commit generation = 38
solr          | 2278608 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f016b84[ckan] main]
solr          | 2278609 INFO  (searcherExecutor-7-thread-1-processing-x:ckan) [   x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@6f016b84[ckan] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_q(6.2.1):C10/1:delGen=1) Uninverting(_s(6.2.1):C1) Uninverting(_t(6.2.1):C1) Uninverting(_u(6.2.1):C1) Uninverting(_v(6.2.1):C1) Uninverting(_y(6.2.1):C1) Uninverting(_11(6.2.1):C1)))}
solr          | 2278611 INFO  (searcherExecutor-7-thread-1-processing-x:ckan) [   x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener done.
solr          | 2278611 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.DirectUpdateHandler2 end_commit_flush
solr          | 2278611 INFO  (searcherExecutor-7-thread-1-processing-x:ckan) [   x:ckan] o.a.s.c.SolrCore [ckan] Registered new searcher Searcher@6f016b84[ckan] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_q(6.2.1):C10/1:delGen=1) Uninverting(_s(6.2.1):C1) Uninverting(_t(6.2.1):C1) Uninverting(_u(6.2.1):C1) Uninverting(_v(6.2.1):C1) Uninverting(_y(6.2.1):C1) Uninverting(_11(6.2.1):C1)))}
solr          | 2278623 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.p.LogUpdateProcessorFactory [ckan]  webapp=/solr path=/update params={commit=true}{add=[ad8d186f2cbca8adf1c72ec67e71deb1 (1702200744687435776)],commit=} 0 398
ckan-dev_1    | 2021-06-10 17:12:12,107 INFO  [ckan.config.middleware.flask_app]  /dataset/done/resource/new render time 1.420 seconds
solr          | 2278830 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=name:"done"+OR+id:"done"&fq=site_id:"default"&rows=1&wt=json} hits=1 status=0 QTime=4
solr          | 2279326 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=50&df=text&fl=id+validated_data_dict&fq=%2Bowner_org:"6d6295db-36ff-40ba-a795-76ddd8106bc4"&fq=%2Bsite_id:"default"&fq=%2Bstate:active&sort=score+desc,+metadata_modified+desc&facet.mincount=1&rows=0&facet=false&wt=json} hits=11 status=0 QTime=3
solr          | 2279349 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=-1&facet.field=groups&facet.field=owner_org&fl=groups&fq=%2Bsite_id:"default"&fq=%2Bstate:active&facet.mincount=1&rows=2&facet=true&wt=json} hits=11 status=0 QTime=4
solr          | 2279824 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=50&df=text&fl=id+validated_data_dict&fq=%2Bowner_org:"6d6295db-36ff-40ba-a795-76ddd8106bc4"&fq=%2Bsite_id:"default"&fq=%2Bstate:active&sort=score+desc,+metadata_modified+desc&facet.mincount=1&rows=0&facet=false&wt=json} hits=11 status=0 QTime=1
solr          | 2279845 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=-1&facet.field=groups&facet.field=owner_org&fl=groups&fq=%2Bsite_id:"default"&fq=%2Bstate:active&facet.mincount=1&rows=2&facet=true&wt=json} hits=11 status=0 QTime=1
ckan-dev_1    | 2021-06-10 17:12:13,468 INFO  [ckan.config.middleware.flask_app]  /dataset/done render time 1.321 seconds

when i go to manage the resource and then datastore it says:
Error: Process completed but unable to post to result_url

When i click the upload to datastore it says
Error: Process completed but unable to post to result_url

And below that it display pending: 9 minutes

Aswell as when i try to preview the resource it keeps saying not available.

@ondics
Copy link

ondics commented Jun 28, 2021

Same problem here.

Standard setup. after docker-compose up, uploading a csv file anf starting the datapusher, the datapusher returns with

Error: Process completed but unable to post to result_url 

My .env (excerpt):

# Other services connections
CKAN_SOLR_URL=http://solr:8983/solr/ckan
CKAN_REDIS_URL=redis://redis:6379/1
CKAN_DATAPUSHER_URL=http://datapusher:8800
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000

Where to debug? What's the problem? Has it worked before?

@ondics
Copy link

ondics commented Jun 28, 2021

ok, solved:

The .env variables CKAN_SITE_URL and CKAN__DATAPUSHER__CALLBACK_URL_BASE must be identical.

If tey are not identical, the error is

Error: Process completed but unable to post to result_url

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants