In server mode after logging in "You have no projects yet"

[Physik310]

Hello Datashare-Community,

I am currently trying to run datashare in SERVER mode but after logging in with a newly created user I receive the following error message:

Using the documentation in https://github.com/ICIJ/datashare/wiki/Datashare-Server-Mode#more-settings I did the following:

• I created a new user:

root@localhost:~# redis-cli -h 127.0.0.1
127.0.0.1:6379> set foo '{"uid":"foo", "password":"fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9", "datashare_projects":["local-datashare"]}'

I edited the config file /root/.local/share/datashare/dist/datashare.conf:
#Datashare properties
parserParallelism=1
cors=no-cors
dataDir=/root/Datashare/local-datashare
redisPoolSize=1
protectedUriPrefix=/api/
mode=EMBEDDED
busType=memory
defaultProject=local-datashare
scrollSize=1000
ocr=true
settings=./dist/datashare.conf
messageBusAddress=redis://127.0.0.1:6379
tcpListenPort=8080
pluginsDir=/root/.local/share/datashare/plugins
queueType=memory
browserOpenLink=true
parallelism=4
scrollSlices=1
elasticsearchDataPath=/root/.local/share/datashare/index
elasticsearchAddress=http://localhost:9200
clusterName=datashare
nlpParallelism=1
dataSourceUrl=jdbc:sqlite:file:/root/.local/share/datashare/dist/datashare.db
defaultUserName=local
extensionsDir=/root/.local/share/datashare/extensions
queueName=extract:queue
redisAddress=redis://127.0.0.1:6379
sessionTtlSeconds=43200

-Then I run the Scan, where I placed one text file into /root/Datashare/local-datashare:

datashare --mode CLI --stages SCAN -d /root/Datashare/local-datashare --defaultProject local-datashare --queueName q1

In the output log it says “...ScannerVisitor - Completed scan of: "/root/Datashare/local-datashare"

-Then I run the indexing:
datashare --mode CLI --stages INDEX --ocr false --defaultProject local-datashare --queueName q1

The output log says:
....,129 [main] INFO IndexTask - disable OCR
....,133 [pool-2-thread-1] INFO IndexTask - Processing up to 4 file(s) in parallel
....,135 [pool-5-thread-1] INFO DocumentQueueDrainer - Draining to consumer until stopped or interrupted.
....,135 [pool-5-thread-1] INFO DocumentQueueDrainer - Continuous draining stopped.
....,135 [pool-2-thread-1] INFO DocumentQueueDrainer - Shutting down.
....,136 [pool-2-thread-1] INFO DocumentQueueDrainer - Awaiting completion up to 10 SECONDS.
....,136 [pool-2-thread-1] INFO DocumentQueueDrainer - Terminated.
....,136 [pool-2-thread-1] INFO IndexTask - drained 0 documents. Waiting for consumer to shutdown
....,138 [pool-2-thread-1] INFO DocumentConsumer - Shutting down.
....,138 [pool-2-thread-1] INFO DocumentConsumer - Awaiting completion up to 30 MINUTES.
....,138 [pool-2-thread-1] INFO DocumentConsumer - Terminated.
....,138 [pool-2-thread-1] INFO IndexTask - exiting
....,142 [main] INFO Indexer - Closing Elasticsearch connections
....,144 [main] INFO Indexer - Elasticsearch connections closed

-Then I started the server:
datashare --mode SERVER --authFilter org.icij.datashare.session.BasicAuthAdaptorFilter

The output log says at the end “..Fluent - Server started on port 8080”

I the browser I am prompted for the login credentials. After logging in I receive:

[pirhoo]

Hi @Physik310,

I see you have 0 documents indexed, I suspected the index not being created in ElasticSearch.

What do you see if you do curl http://localhost:9200/_cat/indices?

[Physik310]

Hi Pirhoo,

Thanks a lot. I receive the following:

root@localhost:~# curl http://localhost:9200/_cat/indices
green open .geoip_databases AZ_PBe0bQWq1rONiUMTwAw 1 0 40 37 37.7mb 37.7mb
yellow open {{ tJph32zUSReAeSPRj0KQ7Q 1 1 0 0 208b 208b
yellow open local-datashare pwKgeRvHQEGpp4_Aw7aCpg 1 1 0 0 208b 208b

[Physik310]

I just tried to create an index with project name local:
datashare --mode CLI --stages INDEX --ocr false --defaultProject local --queueName q1

Running your command shows a new index local:

root@localhost:~# curl http://localhost:9200/_cat/indices?
green  open .geoip_databases AZ_PBe0bQWq1rONiUMTwAw 1 0 40 37 37.7mb 37.7mb
yellow open {{               tJph32zUSReAeSPRj0KQ7Q 1 1  0  0   208b   208b
yellow open local-datashare  pwKgeRvHQEGpp4_Aw7aCpg 1 1  0  0   208b   208b
yellow open local            kpX7HYwjQEiS4uSSV58Fow 1 1  0  0   208b   208b

[Physik310]

It seems that the index is created but the document is not indexed. I can verify that there is a textfile in the specified dataDir:

root@localhost:~# ls /root/Datashare/local-datashare/
topics.txt

[Physik310]

I tried using a completely fresh debian 11 system setup and tried the docker installation as described in https://icij.gitbook.io/datashare/linux/install-datashare-on-linux . After running the datashare.sh script, the server starts but throws the following error after starting:
the start command:
./datashare.sh

The output:

7,410 [Thread-0] INFO  Fluent - Server started on port 8080
...
12,489 [pool-4-thread-1] INFO  IndexWaiterFilter - Ping failed. Waiting for Elasticsearch to be up java.net.ConnectException: Connection refused
13,490 [pool-4-thread-1] INFO  IndexWaiterFilter - Ping failed. Waiting for Elasticsearch to be up java.net.ConnectException: Connection refused
19,498 [pool-4-thread-1] INFO  IndexWaiterFilter - Ping failed. Waiting for Elasticsearch to be up java.net.ConnectException: Timeout connecting to [elasticsearch/172.18.0.3:9200]
25,506 [pool-4-thread-1] INFO  IndexWaiterFilter - Ping failed. Waiting for Elasticsearch to be up java.net.ConnectException: Timeout connecting to [elasticsearch/172.18.0.3:9200]

[Physik310]

Hello,
is it posslible that datashare needs the apache tika libraries? When I run the index mode I don't receive any errors but still no document is being indexed. Is it possible that I have to bind the apache tika libraries somehow?

I run the following:

sudo datashare --mode CLI --stages INDEX -d /root/Datashare --redisAddress redis://127.0.0.1:6379 --elasticsearchAddress http://127.0.0.1:9200 --dataSourceUrl $JDBCURL --rootHost https://127.0.0.1

And always receice 0 Documents drained:

`707 [pool-2-thread-1] INFO IndexTask - Processing up to 4 file(s) in parallel

710 [pool-5-thread-1] INFO DocumentQueueDrainer - Draining to consumer until stopped or interrupted.

711 [pool-5-thread-1] INFO DocumentQueueDrainer - Continuous draining stopped.

711 [pool-2-thread-1] INFO DocumentQueueDrainer - Shutting down.

711 [pool-2-thread-1] INFO DocumentQueueDrainer - Awaiting completion up to 10 SECONDS.

712 [pool-2-thread-1] INFO DocumentQueueDrainer - Terminated.

712 [pool-2-thread-1] INFO IndexTask - drained 0 documents. Waiting for consumer to shutdown

714 [pool-2-thread-1] INFO DocumentConsumer - Shutting down.

714 [pool-2-thread-1] INFO DocumentConsumer - Awaiting completion up to 30 MINUTES.

714 [pool-2-thread-1] INFO DocumentConsumer - Terminated.

714 [pool-2-thread-1] INFO IndexTask - exiting

719 [main] INFO Indexer - Closing Elasticsearch connections

722 [main] INFO Indexer - Elasticsearch connections closed
`

[caro3801]

Hello,
This issue might be related to our internal change of handling accesses ; we changed the data structure in the user's creation.
https://github.com/ICIJ/datashare/wiki/Datashare-Server-Mode/_compare/6fe03700f4ee6abf360c48c91d2c0de74ee978ed...48becd5aa6f38b8f8d6d7fcf5df3002a121fdad5