tap-mambu

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

Pulls raw data from the Mambu 2.0 API
Extracts the following resources:
Outputs the schema for each resource
Incrementally pulls data based on the input state

Streams

branches (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/branches
Primary keys: id
Foreign keys: custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

communications (POST v2)

Endpoint: https://instance.sandbox.mambu.com/api/communications/messages:search
Primary keys: encoded_key
Foreign keys: group_key (groups), user_key (users), client_key (clients), loan_account_key (loans), deposit_account_key (deposits), sender_key (users), reference_id (from SMS dispatcher)
Replication strategy: Incremental (query all, filter results)
- Bookmark query field: creationDate
- Bookmark: creation_date (date-time)
Transformations: Fields camelCase to snake_case

centres (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/centres
Primary keys: id
Foreign keys: assigned_branch_key (branches), custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

clients (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/clients
Primary keys: id
Foreign keys: assigned_user_key (users), assigned_centre_key (centres), custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Full table
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

credit_arrangements (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/creditarrangements
Primary keys: id
Foreign keys: holder_key (?), custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Incremental (query all, filter results)
- Sort by: creationDate:ASC
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

custom_field_sets (GET v1)

Endpoint: https://instance.sandbox.mambu.com/api/customfieldsets
Primary keys: id
Foreign keys: None
Replication strategy: Full table
Transformations: Fields camelCase to snake_case

deposit_accounts (POST v2)

Endpoint: https://instance.sandbox.mambu.com/api/deposits:search
Primary keys: id
Foreign keys: assigned_branch_key (branches), credit_arrangement_key (credit_arrangements), assigned_user_key (users), assigned_centre_key (centres), custom_field_set_id, custom_field_id (custom_field_sets), account_holder_key (?), product_type_key (?)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
- Bookmark query field: lastModifiedDate
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

cards (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/deposits/[deposit_id]/cards
Primary keys: deposit_id, reference_token
Foreign keys: deposit_id (deposits)
Replication strategy: Full table (ALL for parent deposit_id)
Transformations: Fields camelCase to snake_case

deposit_products (GET v1)

Endpoint: https://instance.sandbox.mambu.com/api/savingsproducts/DSP
Primary keys: id
Foreign keys: None
Replication strategy: Incremental (query all, filter results)
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case

deposit_transactions (POST v2)

Endpoint: https://instance.sandbox.mambu.com/api/deposits/transactions:search
Primary keys: encoded_key
Foreign keys: branch_key (branches), centre_key (centers), user_key (users), linked_loan_transaction_key (loan_transactions), linked_deposit_transaction_key (deposit_transactions)
Replication strategy: Incremental (query all, filter results)
- Bookmark query field: creationDate
- Sort by: creationDate:ASC
- Bookmark: creation_date (date-time)
Transformations: Fields camelCase to snake_case

groups (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/groups
Primary keys: id
Foreign keys: client_key (clients), assigned_branch_key (branches), assigned_centre_key (centers), assigned_user_key (users), custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

loan_accounts (POST v2)

Endpoint: https://instance.sandbox.mambu.com/api/loans:search
Primary keys: id
Foreign keys: deposit_account_key (deposits), target_deposit_account_key (deposits), assig, ned_user_key (users), assigned_centre_key (centres), assigned_branch_key (branches), credit_arrangement_key (credit_arrangements), custom_field_set_id, custom_field_id (custom_field_sets), account_holder_key (?), product_type_key (?)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
- Bookmark query field: lastModifiedDate
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

loan_products (GET v1)

Endpoint: https://instance.sandbox.mambu.com/api/loanproducts
Primary keys: id
Foreign keys: None
Replication strategy: Incremental (query all, filter results)
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case

loan_transactions (POST v2)

Endpoint: https://instance.sandbox.mambu.com/api/loans/transactions:search
Primary keys: encoded_key
Foreign keys: branch_key (branches), centre_key (centers), user_key (users), parent_loan_transaction_key (loan_transactions), linked_loan_transaction_key (loan_transactions), linked_deposit_transaction_key (deposit_transactions)
Replication strategy: Incremental (query all, filter results)
- Bookmark query field: creationDate
- Sort by: creationDate:ASC
- Bookmark: creation_date (date-time)
- Lookback window: 14 day default, overridden by config
Transformations: Fields camelCase to snake_case

tasks (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/tasks
Primary keys: id
Foreign keys: created_by_user_key (users), assigned_user_key (users), custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

users (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/users
Primary keys: id
Foreign keys: branch_key (branches), assigned_branch_key (branches), custom_field_set_id, custom_field_id (custom_field_sets)
Replication strategy: Incremental (query all, filter results)
- Sort by: lastModifiedDate:ASC
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

gl_accounts (GET v1)

Endpoint: https://instance.sandbox.mambu.com/api/glaccounts
Primary keys: gl_code
Replication strategy: Incremental (query filtered based on date and account type)
- Bookmark: last_modified_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

gl_journal_entries (POST v1)

Endpoint: https://instance.sandbox.mambu.com/api/gljournalentries/search
Primary keys: entry_id
Replication strategy: Incremental (query filtered based on date)
- Bookmark: creation_date (date-time)
Transformations: Fields camelCase to snake_case, Abstract/generalize custom_field_sets

activities (GET v1)

Endpoint: https://instance.sandbox.mambu.com/api/activities
Primary keys: encoded_key
Replication strategy: Incremental (query all, filter results)
- Bookmark query field: timestamp
- Bookmark: timestamp (date-time)
Transformations: Fields camelCase to snake_case, de-nest activity

index rate sources (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/indexratesources
Primary keys: encoded_key
Replication strategy: Full table
Transformations: Fields camelCase to snake_case

installments (GET v2)

Endpoint: https://instance.sandbox.mambu.com/api/installments
Primary keys: encoded_key
Replication strategy: Incremental (query all, filter results)
- Bookmark query field: last_paid_date
- Bookmark: last_paid_date (date-time)
Transformations: Fields camelCase to snake_case

audit_trail (GET V1)

Endpoint: https://instance.sandbox.mambu.com/api/v1/events
Replication strategy: Incremental (query all, filter results)
- Bookmark query field: occurred_at
- Bookmark: occurred_at (date-time)
- Sort by: occurred_at:ASC
Transformations: Fields camelCase to snake_case

Quick Start

Install

Clone this repository, and then install using setup.py. We recommend using a virtualenv:

virtualenv -p python3 venv
source venv/bin/activate
python setup.py install
OR
cd .../tap-mambu
pip install .

Dependent libraries The following dependent libraries were installed.

pip install singer-python
pip install singer-tools
pip install target-stitch
pip install target-json

Create your tap's config.json file. The subdomain is everything before .mambu.com in the Mambu instance URL. For the URL: https://stitch.sandbox.mambu.com, the subdomain would be stitch.sandbox. Lookback window applies only to loan transactions stream.

{
    "username": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD",
    "apikey": "YOUR_APIKEY",
    "subdomain": "YOUR_SUBDOMAIN",
    "start_date": "2019-01-01T00:00:00Z",
    "lookback_window": 30,
    "user_agent": "tap-mambu <api_user_email@your_company.com>",
    "page_size": "500",
    "apikey_audit": "AUDIT_TRAIL_APIKEY",
    "window_size": 7
}

Note: The window_size parameter defaults to 1 day, which may cause slowdowns in historical sync for streams utilizing multi-threaded implementation. Conversely, using a larger window_size could lead to potential out-of-memory issues. It is advisable to select an optimal window_size based on the start_date and volume of data to mitigate these concerns.

Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

{
    "currently_syncing": "tasks",
    "bookmarks": {
        "audit_trail":"2021-05-20T09:59:09.780213Z", 
        "branches": "2019-06-11T13:37:51Z",
        "communications": "2019-06-19T19:48:42Z",
        "centres": "2019-06-18T18:23:53Z",
        "credit_arrangements": "2019-06-19T19:48:45Z",
        "custom_field_sets": "2019-06-11T13:37:56Z",
        "deposit_accounts": "2019-06-19T19:48:47Z",
        "cards": "2019-06-18T18:23:58Z",
        "deposit_products": "2019-06-20T00:52:49Z",
        "deposit_transactions": "2019-06-20T00:52:40Z",
        "groups": "2019-06-19T19:48:41Z",
        "loan_accounts": "2019-06-11T13:37:52Z",
        "loan_products": "2019-06-20T00:52:43Z",
        "loan_transactions": "2019-06-19T19:48:44Z",
        "tasks": "2019-06-18T18:23:55Z",
        "users": "2019-06-20T00:52:46Z",
        "gl_journal_entries": "2019-11-01T00:00:00.000000Z",
        "gl_accounts": {
            "INCOME": "2019-07-03T15:15:43.000000Z",
            "EQUITY": "2019-07-03T15:15:43.000000Z",
            "LIABILITY": "2019-07-03T15:15:43.000000Z",
            "ASSET": "2019-07-03T15:15:43.000000Z",
            "EXPENSE": "2019-07-03T15:15:43.000000Z"
        }
    }
}

Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

tap-mambu --config config.json --discover > catalog.json

See the Singer docs on discovery mode here. The default catalog.json that gets generated does not have any selected streams. Therefore you must enable them manually in order to actually retrieve data. Each stream in the catalog.json that you want retrieved is expected to have a data field in the root metadata block:

 "selected" : true

For example the following configuration will retrieve the branches stream:

"stream": "branches",
"metadata": [
  {
    "breadcrumb": [],
    "metadata": {
      "selected" : true,
      "table-key-properties": [
        "id"
      ],
      "forced-replication-method": "INCREMENTAL",
      "valid-replication-keys": [
        "last_modified_date"
      ],
      "inclusion": "available"
    }
  }

Run the Tap in Sync Mode (with catalog) and write out to state file

For Sync mode:

tap-mambu --config config.json --catalog catalog.json > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To load to json files to verify outputs:

tap-mambu --config config.json --catalog catalog.json | target-json > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To pseudo-load to Stitch Import API with dry run:

tap-mambu --config config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Test the Tap

While developing the Mambu tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

pylint tap_mambu -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

Pylint test resulted in the following score:

Your code has been rated at 9.87/10.

To check the tap and verify working:

tap-mambu --config config.json --catalog catalog.json | singer-check-tap > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Check tap resulted in the following:

The output is valid.
It contained 589 messages for 18 streams.

    24 schema messages
    503 record messages
    62 state messages

Details by stream:
+----------------------+---------+---------+
| stream               | records | schemas |
+----------------------+---------+---------+
| loan_products        | 2       | 1       |
| users                | 3       | 1       |
| gl_journal_entries   | 281     | 1       |
| branches             | 2       | 1       |
| groups               | 3       | 1       |
| credit_arrangements  | 2       | 1       |
| clients              | 104     | 1       |
| cards                | 1       | 3       |
| loan_accounts        | 7       | 1       |
| deposit_transactions | 32      | 1       |
| centres              | 2       | 1       |
| gl_accounts          | 18      | 5       |
| communications       | 1       | 1       |
| deposit_accounts     | 3       | 1       |
| tasks                | 8       | 1       |
| loan_transactions    | 9       | 1       |
| custom_field_sets    | 21      | 1       |
| deposit_products     | 4       | 1       |
+----------------------+---------+---------+

Name		Name	Last commit message	Last commit date
Latest commit History 111 Commits
.circleci		.circleci
.github		.github
mambu_tests		mambu_tests
schema_fetcher		schema_fetcher
tap_mambu		tap_mambu
tests		tests
.codeclimate.yml		.codeclimate.yml
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.pylintrc		.pylintrc
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
config.json.example		config.json.example
setup.py		setup.py
state.json.example		state.json.example

License

singer-io/tap-mambu

Folders and files

Latest commit

History

Repository files navigation

tap-mambu

Streams

Quick Start

About

Resources

License

Stars

Watchers

Forks

Languages