Plugin mechanism for compiled rust code

[marcmo]

we need to be able to link precompiled static libraries.

• libs need to be pre-built for different architectures
• should allow to pull in closed source functionality

First example will be the MDF library

[DmitryAstafyev]

Tech Overview

	Scripting	Sockets	Pipes	Memory	Wasm	Dynamic lib (FFI)	Static lib
Runtime loading -1 - no; 1 - yes	1	1	1		1	1
Requires interpreter or compiler -1 - requires; 0 - doesn't	-1	0	0		0	0
Performance estimation (theoretically) 1 - poor; 5 - best	2	3	3		3	5
Windows headache 0 - not expected -1 - probably	0	-1	0		0	0
Possibility avoid cloning 1 - no cloning; 0 - partly; -1- cloning	-1	-1	-1		1	1
Required rust unsafe 0 - yes; 1 - no;	1	1	1		1	0
Score	2	3	4	x	6	7	x
Disqualification				R.0			R.1

R.0. - This feature heavily depends on the system’s kernel, so it may hurt the “Cross-Compatibility” requirement. [1]
R.1. - Requires compilation with basic application; no run-time loading

Plugins targets

1. Source
2. Parser

Requirements for plugins system

• Setting/options of parser/source should be generic and represented as JSON object with the description of fields and types
• Key-parser-traits should excluded into a standalone crate to be reused in the scope of a parser plugin

Plugin output files

• lib's files
• manifest file (with description, version etc in JSON or toml and with definition of plugin's type: (parser, source or...))

note: store in the home folder ~/.chipmunk/plugins

Loading

• makes sense to load with chipmunk based on a list of active plugins from electron-layer

Opened questions

• distribution? possible ways: manual (copying files), github repo based, server based etc.

Overview of ByteSource and Parser traits

Motivation to use (try to use) crate [abi_stable](https://github.com/rodrimati1992/abi_stable_crates) is better support of types. But in general would be good to understand, which types we would like to use and if is there a way to stay on the level of primitives.

ByteSource trait

pub trait ByteSource: Send + Sync {
    fn consume(&mut self, offset: usize);
    fn current_slice(&self) -> &[u8];                    <== Key method
    fn len(&self) -> usize;
    fn is_empty(&self) -> bool {
        self.len() == 0
    }
    async fn reload(&mut self, filter: Option<&SourceFilter>) -> Result<Option<ReloadInfo>, Error>;
    async fn cancel(&mut self) -> Result<(), Error> {
        Ok(())
    }
    async fn income(&mut self, _msg: sde::SdeRequest) -> Result<sde::SdeResponse, Error> {
        Err(Error::NotSupported)
    }
}

The key-method is current_slice, which delivers a byte's slice for future processing by a parser. Actually ByteSource plugin can operate with primitive types only.

Conclusion: plugin can operate just with primitive types

Parser trait

#[derive(Debug)]
pub enum ParseYield<T> {
    Message(T),
    Attachment(Attachment),
    MessageAndAttachment((T, Attachment)),
}

pub trait Parser<T> {
    fn parse<'a>(
        &mut self,
        input: &'a [u8],
        timestamp: Option<u64>,
    ) -> Result<(&'a [u8], Option<ParseYield<T>>), Error>;
}

#[derive(Debug, Clone, Serialize)]
pub struct Attachment {
    pub name: String,
    pub size: usize,
    pub created_date: Option<String>,
    pub modified_date: Option<String>,
    /// The indexes of the message within the original trace (0-based).
    pub messages: Vec<usize>,
    pub data: Vec<u8>,
}

• as T we are using now LogMessage trait, which actually just inherit Display trait to convert bytes to a string. But this is only what we are doing always - converting LogMessage to string (always (!)). It means, to in current state we can just return from the plugin a string as bytes.
• Attachment actually also is a collection of primitive types.
• ParseYield can be represented as bytes using a simple own protocol (we can mark with the first byte a type of message and based on it parse in on a host)

Conclusion: plugin can operate just with primitive types in case of implementation our own simple protocol based on header and payload

References/links:

• [1] @kruss thanks for nice link: https://nullderef.com/blog/plugin-tech/

[DmitryAstafyev]

As for the first step we could implement BinaryByteSource (sources/src/binary/raw.rs) as a plugin. For the first prototype:

• hardcode path to plugin on rust level
• one option - file path as a string
• loading as soon as rust module is loaded
• others as simple as possible

Goals:

• confirm conception (possibility to use crate abi_stable)
• check performance
• check shutdown workflow
• check possibility to operate with byte slices to avoid cloning