[Feature Request] Provide Image with Clouseau / Fulltext Search

[garrensmith]

From docker-couchdb: klaemo/docker-couchdb#48

It’d be cool to have a build of 2.0 that includes the Cloudant Fulltext Search setup described in https://cloudant.com/blog/enable-full-text-search-in-apache-couchdb/

Note, this is not a request to make the 2.0 image to include this, but a new image that is 2.0 + the full text stuff :)

[homerjam]

+1

I made https://github.com/homerjam/cloudant-search/tree/latest by following some instructions but I'm a total noob when it comes to making it production ready, adding auth/ssl etc. The other thing is it seems to wipe the database directory on restart when I use it locally - so really its nothing more than a proof of concept.

[wohali]

I'm afraid ASF restrictions will make this difficult for us, but we'll look into at least providing the Dockerfile. You may have to build it yourself.

[style95]

+1 on this

[endyjasmi]

Hello, I tried to make a variant of the latest CouchDB image (2.1.1) with search feature but encountered some problem. Hoping that somebody can give some pointer as to how to solve the problem.

Following is what I did:

1. Build a version of CouchDB (2.1.x) which integrate with Cloudant Dreyfus (master) using this page as guide. The result is in the branch 2.1.x-dreyfus.
2. Build a version of CouchDB Docker (master) which build the CouchDB (2.1.x-dreyfus) from step 1. The result of the build can be found at dreyfus branch. Feel free to check the changes with;

   git diff --no-index ./2.1.1/Dockerfile ./2.1.1-dreyfus/Dockerfile
   

3. Build a Cloudant Clouseau (master) docker image. The result is at Clouseau Docker repository.
4. After that, I try to run the system with the following docker compose configuration.

All the service is running. As I have disable-fauxton, I configure my couchdb using external fauxton.

npm install -g fauxton
fauxton

The problem occur when I try to test analyzer tokenization with;

POST /_search_analyze HTTP/1.1
Content-Type: application/json
{"analyzer":"standard", "text":"hello world"}

CouchDB return 500 Internal Server Error with response

{
    "error": "{badarg,[{erlang,monitor,[process,{analyzer,clouseau@clouseau}],[]},\n         {ioq,submit_request,2,[{file,\"src/ioq.erl\"},{line,156}]},\n         {ioq,maybe_submit_request,1,[{file,\"src/ioq.erl\"},{line,119}]},\n         {ioq,handle_info,2,[{file,\"src/ioq.erl\"},{line,92}]},\n         {gen_server,handle_msg,5,[{file,\"gen_server.erl\"},{line,599}]},\n         {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,237}]}]}",
    "reason": "{gen_server,call,\n            [ioq,\n             {request,{analyzer,clouseau@clouseau},\n                      {analyze,<<\"standard\">>,<<\"hello world\">>},\n                      undefined,<0.8648.0>,undefined},\n             infinity]}",
    "ref": 1634367512
}

Following is the output for docker-compose logs couchdb and docker-compose logs clouseau:
docker-compose logs.

Based on the logs and the research I conducted/understand, I assume that it is an error related to erlang rpc? Maybe the problem is caused by incompatible erlang version between CouchDB and Clouseau?

I have been working on this for the past week to bring it up to this point. Any helps is greatly appreciated. Do know that I have no erlang knowledge so all the curly bracket is alien to me.

Last but not least, thanks in advance and Happy New Year!

[grueni]

I followed thoroughly the instructions from ibm 2015.
You should read the instructions from cloudant 2017 too.
They differ in some points.

If you would like to use the tests in couchdb you should

• change the order of insertion in local.ini (see in issue 25). Otherwise the tests will fail.
• cherrypick this commit repairing the tests for the indices of type text with
  curl https://github.com/apache/couchdb/commit/52e9d894eecff0de1a7c75be24f219a43e680bd3.patch > patches/07-test-repair.patch

You may find my package for OpenIndiana in the branch couchdb-dreyfus. The folder patches contains the patches and among them patches 06-, 07- and 08- are relevant for the integration of dreyfus. In folder files you find dreyfus.js which should be copied to share/server.

All tests are now successfully passed.

AFAIK CouchDB and Clouseau should run in the same docker container. Maybe this the root cause of your rpc error.

Did you know this one with CouchDB 2.0?

edited:
It worked for CouchDB 2.1.1 (not trunk!), Erlang 19.3, current master of drefyus with latest OpenIndiana.

[lucidNTR]

two things missing for me until it worked: a) the standard cookie for clouseau is "monster" and needs to be set to the same value in the erlang flags. (would be happy for help how to set the flag in the maven scala startup command, i did not succeed in changing it) b) for some reason mango checks for dryfus with module_loaded(dryfus_index) but dryfus_index was never in my loaded modules, i patched it to check module_loaded(dryfus_index_manager) and since then it works in combination with all @grueni s patches. I will publish my docker container and post here in the next days. would be interesting to know the exact licensing issues, i would love to have this in the apache/couchdb-docker repo...

[lucidNTR]

https://github.com/neutrinity/ntr-couch-docker

[wohali]

We're tracking the upstream issue here: apache/couchdb#1269

Once this is resolved, expect to see official support in the image for these subsystems.

[wohali]

3.0 is going to include the Couch side of the search function natively. You'll need to run another container with the actual lucene runtime in it.

Once 3.0 releases we'll resolve this by providing a sample docker-compose file.

[wohali]

hey @willholley do you want to take a shot at a docker-compose for this + search? would be nice to put it in the docs for 3.0.0.

(Yes, I know there's already couchdb-helm...)

[kocolosk]

FYI the place where things get tricky with docker-compose compared to Kubernetes is that in Compose the containers are on separate networks, which means the Clouseau container can’t find the EPMD in the CouchDB container. In Kube they share the loopback network so one EPMD suffices.

I think one could create a multi-process container that runs both EPMD and Java, and then configures the two containers to look at the right links to find each other, it’s just a somewhat janky setup. Alternatively one could patch Clouseau to have a configurable location to look for EPMD, but we’re basically trying to avoid modifying that code at this point.

I would love to see a docker-compose setup, I think it would make development setups much simpler. Just wanted to explain why we don’t have this done already.

[homerjam]

Not an expert but would it be possible to use a proxy in the Clouseau container (suggested here)?

[rkleine]

Hi, here is a simple docker-compose example using search plugin without any hack, just standard compose yaml.

mkdir -p ./config/{couchdb,clouseau}

./config/clouseau/clouseau.ini

[clouseau]
name=clouseau@127.0.0.1
cookie=monster
dir=/data
max_indexes_open=500

./config/clouseau/log4j.properties

log4j.rootLogger=debug, CONSOLE
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %c [%p] %m%n

./config/couchdb/local.ini

[couch_httpd_auth]
secret = dev

[admins]
; admin = admin
admin = -pbkdf2-37aa363808ec5b83fc71cf2811479fbf1917f43d,2483e8ffe206819712f3bd1ba2308d41,10

curl -sSL -o clouseau-2.17.0-dist.zip "https://github.com/cloudant-labs/clouseau/releases/download/2.17.0/clouseau-2.17.0-dist.zip
# Unizp clouseau-2.17.0-dist.zip in clouseau-2.17.0

docker-compose.yml
This yaml expose 5984 to the host network, if you already using the 5984 change it on the yaml

version: '3.8'
services:
  couchdb1:
    image: couchdb:3.1.0
    restart: always
    ports:
    - 5984:5984
    environment:
      - NODENAME=10.0.0.10
      - ERL_FLAGS=-setcookie monster
    volumes:
    - ./config/couchdb:/opt/couchdb/etc/local.d
    - ./data/couchdb/1:/opt/couchdb/data
    networks:
      couchdb-net:
        ipv4_address: 10.0.0.10

  clouseau1:
    image: openjdk:8
    command: >
      java -server
      -classpath '/app/*'
      -Xmx2G -Dsun.net.inetaddr.ttl=30
      -Dsun.net.inetaddr.negative.ttl=30
      -Dlog4j.configuration=file:/config/log4j.properties
      -XX:OnOutOfMemoryError="kill -9 %p"
      -XX:+UseConcMarkSweepGC
      -XX:+CMSParallelRemarkEnabled com.cloudant.clouseau.Main /config/clouseau.ini
    restart: always
    depends_on:
    - couchdb1
    volumes:
    - ./config/clouseau:/config
    - ./data/clouseau/1:/data
    - ./clouseau-2.17.0:/app
    network_mode: service:couchdb1

networks:
  couchdb-net:
    name: couchdb-net
    driver: bridge
    ipam:
      config:
        - subnet: 10.0.0.0/24

docker-compose up

Check it http://127.0.0.1:5984

The "trick" is the network_mode: service:couchdb1 in the the service clouseau1. With this we are telling clouseau1 service to use the same network of couchdb1 service. So the ports (4369/epmd) are mapped to the clouseau1 container. For simplicity I just deploy one node, but we can add more easily, join them and deploy a loadbalancer with nginx/haproxy in the same docker-compse.yml

[wohali]

@rkleine Would you like to submit a PR against this repo with this setup? Just create a new subdirectory called clouseau-compose and place this content in it, plus a README.md that contains the info above.