couchbase-java-importer

This is a pluggable importer for Couchbase. So far it supports importing documents from a CSV file, a MongoDB or CouchDB instance.

How to Build

$ ./gradlew build

How to Use

$ cd build/distributions/
$ unzip couchbase-java-importer.zip

At this step you need to choose what kind of import you want to do. Configuration samples are available at the root of the repository.

Let's pretend you want to import a MongoDB collection called restaurants from the database test and add a field called couchbaseType with the value restaurant for every imported documents. Let's also pretend you have a local MongoDB and local Couchbase instance. For that you would need the following configuration

# Hostnames, comma separated list of Couchbase node IP or hostname
hostnames: localhost,127.0.0.1
# Buket name
bucket: default
# Bucket password
password:
# Log to write succesfully imported keys
successLogFilename: succes.out
# Log to write unsuccesfully imported keys
errorLogFilename: error.out
# Default RequestCancelledException delay in milliseconds and maximum number of retries
requestCancelledExceptionDelay: 31000
requestCancelledExceptionRetries: 100
# Default TemporaryFailureException delay in milliseconds and maximum number of retries
temporaryFailureExceptionDelay: 100
temporaryFailureExceptionRetries: 100
# Default upsert timeout in milliseconds
importTimeout: 500
# Choose between CSV, COUCHDB, MONGODB
choosenImporter: MONGODB
mongodb:
  # Give a valid connection string to connect to a MongoDB instance
  connectionString: "mongodb://127.0.0.1:27017/"
  # Name of the MondoDB database to connect to
  dbName: "test"
  # Name of the collection to import
  collectionName: "restaurants"
  # Couchbase does not have collection, we usually use a type field. As there could already be a type field in Mongo, you can specify another fieldName to be used as type
  typeField: "type"
  # type of the documents that will be imported
  type: "restaurant"

This is the content of the MongoDB sample configuration. To run the import copy the configuration file and run the importer:

$ cp ../../../application-mongodb.yml.sample application.yml 
$ ./bin/couchbase-java-importer

Once the import as ran you should have one file called success.out that contains the id of every document imported. If something went wrong you should also have a file called error.out.

Every configuration samples contains comments that should help you understand the various import options.

JDBC Configuration

Start by copying the JDBC sample configuration:

 cp ../../../application-jdbc.yml.sample application.yml 

Make sure you have configured your JDBC connection through the following properties:

spring.jpa.database=POSTGRESQL
spring.datasource.platform=postgres
spring.jpa.show-sql=true
spring.jpa.hibernate.ddl-auto=create-drop
spring.database.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://192.168.99.100:32778/dvdrental
spring.datasource.username=postgres
spring.datasource.password=password

More information available on the spring documentation.

Once you have your connection setup, you can configure what you want to import using the following filters used by the DatabaseMetaData Object.

choosenImporter: JDBC
jdbc:
  catalog: 
  schemaPattern: "public"
  tableNamePattern: 
  types:
  columnNamePattern:

All the selected tables and columns will be stored in a JsonDocument. It's key is defined by the tablesSchemaId property.

jdbc:
  # id of the document containing your database schema to be imported 
  tablesSchemaId: myDatabaseSchema

CouchDB

Start by copying the JDBC sample configuration:

 cp ../../../application-jdbc.yml.sample application.yml 

Then use the downloadURL property to define wich CouchDB documents you want to import. By default it uses the all_docs view with the include_docs flag set to true.

choosenImporter: COUCHDB
couchdb:
  # Download URL 
  downloadURL: http://127.0.0.1:5984/database_export/_all_docs?include_docs=true

CSV

Sample configuration for CSV import:

choosenImporter: CSV
csv:
  # CSV Separating char for rows
  columnSeparator: ';'
  # CSV quotes
  quoteChar: ''
  # Path to the CSV file to import
  csvFilePath: /home/couchbase/csvimporter/advocates.csv
  # Skip the first line of the CSV for field names
  skipFirstLineForNames: true
  # Any format usable by the Java SimpleDateFormat Class
  dateFormat: EEE MMM dd HH:mm:ss z yyyy
  # Language tag used by Java's Locale class
  languageTag: FR_FR
  # Number of columns to import
  totalcolumns: 10
  # Column index to use the column value as id
  keyColumIndex: 0
  # The value of this field will be added as key prefix
  keyPrefix: "advocate::"
  #Give the type of the columns, could be String, Long, Double, Boolean, Date. Must be the exact same size as the number of columns in your file
  columType:
     - STRING
     - STRING
     - STRING
     - STRING
     - STRING
     - DATE
     - LONG
  # Choose the name of the fields for each column, mandatory if skipFirstLineForNames is set to true.
  columName:
     - id
     - type
     - firstname
     - lastname
     - location
     - creationDate
     - count

MongoDB