Use DataFrame as a NodeJS/npm C++ addon

[dereckmezquita]

Hello @hosseinmoein I wanted to thank you 👋 for all the work on this project.

I'm working on an npm package that would leverage DataFrame do you have any experience with this? Maybe someone else can discuss.

I am having some troubles with how to manage DataFrame as a dependency and include it in my npm package so that it builds when installed by users. Should I use conan or vckpg what scripts to include in my package.json so the pacakge installs DataFrame as a dependency and builds for users on install etc.

[hosseinmoein]

I am not familiar with npm packages etc. But exploring Conan and/or VCPKG is a good start.
I leave the discussion here, if anybody else can help

[dereckmezquita]

I've been continously working on this and figured out how to set this up. I'll include a simple "Hello world" for what I have so far.

DataFrame in an npm package "Hello world"

cmake

I used cmake-js for this. A lot of people suggest node-gyp but I find cmake easier to use and more flexible. I am still learning Cmake so please correct me or make suggestions if you see anything.

# https://nodejs.github.io/node-addon-examples/build-tools/cmake-js/
# https://learn.microsoft.com/en-us/vcpkg/get_started/get-started-vscode
cmake_minimum_required(VERSION 3.27.7)

# Set the path to find packages installed by vcpkg
# platform agnostic target triplet
file(GLOB VCPKG_PLATFORM_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/vcpkg_installed/*")

# Convert the list of directories into a format suitable for CMAKE_PREFIX_PATH
set(VCPKG_PREFIX_PATHS "")
foreach(DIR IN LISTS VCPKG_PLATFORM_DIRS)
  if(IS_DIRECTORY "${DIR}")
    list(APPEND VCPKG_PREFIX_PATHS "${DIR}")
  endif()
endforeach()

# Now set the CMAKE_PREFIX_PATH
set(CMAKE_PREFIX_PATH "${VCPKG_PREFIX_PATHS}" ${CMAKE_PREFIX_PATH})

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

cmake_policy(SET CMP0091 NEW)
cmake_policy(SET CMP0042 NEW)

project(datatable)

add_definitions(-DNAPI_VERSION=8)

include_directories(${CMAKE_JS_INC})
execute_process(
  COMMAND node -p "require('node-addon-api').include"
  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
  OUTPUT_VARIABLE NODE_ADDON_API_DIR
  OUTPUT_STRIP_TRAILING_WHITESPACE
)
# Fix for paths that contain spaces
string(REPLACE "\"" "" NODE_ADDON_API_DIR ${NODE_ADDON_API_DIR})
include_directories(${NODE_ADDON_API_DIR})

# Include all source files recursively from the cpp directory
file(GLOB_RECURSE SOURCE_FILES "cpp/*")

add_library(${PROJECT_NAME} SHARED ${SOURCE_FILES} ${CMAKE_JS_SRC})
set_target_properties(${PROJECT_NAME} PROPERTIES PREFIX "" SUFFIX ".node")
target_link_libraries(${PROJECT_NAME} ${CMAKE_JS_LIB})

if(MSVC AND CMAKE_JS_NODELIB_DEF AND CMAKE_JS_NODELIB_TARGET)
  execute_process(COMMAND ${CMAKE_AR} /def:${CMAKE_JS_NODELIB_DEF} /out:${CMAKE_JS_NODELIB_TARGET} ${CMAKE_STATIC_LINKER_FLAGS})
endif()

# dependencies
find_package(dataframe CONFIG REQUIRED)
target_link_libraries(datatable PRIVATE DataFrame::DataFrame)

vcpkg

I am using vcpkg to manage dependencies in manifest mode. They get installed local to the project: ls vcpkg_installed/arm64-osx/include/DataFrame

{
  "name": "your-project-name",
  "version-string": "1.0.0",
  "dependencies": [
    "dataframe"
  ]
}

Cpp

./cpp/main.cpp

#include <napi.h>

#include "DataFrameAddon/DataFrameAddon.h"

Napi::Object Init(Napi::Env env, Napi::Object exports)
{
    DataFrameAddon::Init(env, exports);

    return exports;
}

NODE_API_MODULE(datatable, Init)

#pragma once
#include <DataFrame/DataFrame.h>
#include <napi.h>
#include <vector>
#include <string>

using namespace hmdf;

class DataFrameAddon : public Napi::ObjectWrap<DataFrameAddon>
{
public:
    static Napi::Object Init(Napi::Env env, Napi::Object exports);
    DataFrameAddon(const Napi::CallbackInfo &info);
    Napi::Value ToString(const Napi::CallbackInfo &info);
private:
    StdDataFrame<uint64_t> df;
};

#include "DataFrameAddon.h"

Napi::Object DataFrameAddon::Init(Napi::Env env, Napi::Object exports)
{
    Napi::Function func = DefineClass(env, "DataFrame", {
        InstanceMethod("toString", &DataFrameAddon::ToString)
    });

    Napi::FunctionReference *constructor = new Napi::FunctionReference();
    *constructor = Napi::Persistent(func);
    env.SetInstanceData(constructor);
    exports.Set("DataFrame", func);
    return exports;
}

DataFrameAddon::DataFrameAddon(const Napi::CallbackInfo &info) : Napi::ObjectWrap<DataFrameAddon>(info)
{
    Napi::Env env = info.Env();

    std::vector<uint64_t> idx_col = {1, 2, 3, 4, 5, 6, 7};
    std::vector<int> int_col1 = {1, 2, -3, 100, 5};

    StdDataFrame<uint64_t> df;

    df.load_index(std::move(idx_col));
    df.load_column("int_col1", std::move(int_col1));

    this->df = df;
}

Napi::Value DataFrameAddon::ToString(const Napi::CallbackInfo &info)
{
    Napi::Env env = info.Env();
    return Napi::String::New(env, this->df.to_string());
}

TypeScript exports

./src/index.ts

import path from 'path';
const addon = require(path.join(__dirname, '../build/Release/datatable'));

export const DataFrame = addon.DataFrame;

usage

Now you can do this when you build and install the package:

import { DataFrame } from '@dereckmezquita/datatable';

const yeet = new DataFrame();

console.log(yeet);
console.log(yeet.toString());

(base) work@Derecks-MacBook-Air dev % ts-node index.ts
DataFrame {}
INDEX:7:<ulonglong>:1,2,3,4,5,6,7,

[hosseinmoein]

It is because to_string() requires template arguments template<typename ... Ts> that you are not providing.
you need this: .to_string<int>()

[dereckmezquita]

Yes that's right @hosseinmoein I noticed I have to pass template args to to_string. Is there a way I can just have it convert all columns; the whole DataFrame to a string instead?

In order to wrap this method and get it to work as a class member I had to do use a visitor:

Napi::Value DataFrameAddon::ToString(const Napi::CallbackInfo &info) {
    Napi::Env env = info.Env();

    // we want a visitor that converts all types of cols to string
    auto toStringVisitor = [](auto&& arg) -> std::string {
        return arg.template to_string<std::string, int64_t, double>();
    };

    // Use std::visit with the visitor to call the correct to_string method
    std::string result = std::visit(toStringVisitor, df);

    // Convert the result to a Napi::String and return it
    return Napi::String::New(env, result);
}

But I am realising that as I add more types/different columns to my DataFrame I'll have to add more type args to to_string so they convert as well.

Evidently I'm writing a wrapper for DataFrame :-)

Thus I want to allow for conevrsion of all cols to string being that the columns will be added dynamically and types are not known at compile time.

Keen to hear your input/corrections.

Correction: I'm still learning/understanding this. I used a visitor since I had made this->df a std::variant.

[hosseinmoein]

C++ is a statically typed language. That means all types must be known at compile time. Also, there is no way to store types in C++ and at runtime cast to the stored type. So, as I write in the docs, you must specify all column types at compile time (but only once. If you have 3 double columns, you specify double only once). There is no way around this, because of the nature of C++ language, and because I want to allow any types.
One way that comes to mind is for you to only allow certain types (for example only allow fundamental types). And then hardcode them.
Is there any way to allow your end user to specify type and pass it to C++?

[dereckmezquita]

Also, there is no way to store types in C++ and at runtime cast to the stored type.

Got it, of course.

What I can do though, is map between Napi/NodeJS (Napi = Node API) types to C++ types and cast using a series of if else statements whereby I test, if is X Napi type then cast to C++ type.

So, as I write in the docs, you must specify all column types at compile time (but only once. If you have 3 double columns, you specify double only once). There is no way around this, because of the nature of C++ language, and because I want to allow any types.

Question about this please. When you say you must specify all column types at compile time do you mean functions/methods acting on the DataFrame - such is the case with the df.to_string<T> method.

I'm almost certain this is what you mean because I've been able to create an empty DataFrame and then dynamically add columns of varying types to it without issue.

One way that comes to mind is for you to only allow certain types (for example only allow fundamental types). And then hardcode them.

Is there any way to allow your end user to specify type and pass it to C++?

Indeed, I've already started down this road, conveniently when a user passes data from NodeJS to a C++ add-on I can test the type passed by the user using different methods attached to those objects.

One nuance here though is that there is some amount of information lost when the user passes certain types such as numbers. As JavaScript has one main number type for both floats and ints. The solution here is to cast to a double.

Another idea I had is to allow the user to pass a type argument as a string. However, this seems less than ideal to me.

[hosseinmoein]

I think you are on the right path.