Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Job Payload that can be shared amongst Tasks #26

Open
KenSuenobu opened this issue May 9, 2018 · 3 comments
Open

Create Job Payload that can be shared amongst Tasks #26

KenSuenobu opened this issue May 9, 2018 · 3 comments
Labels
difficult Feature that may be difficult to implement enhancement New feature or request help wanted Extra attention is needed serde (De)serialization server Server/Daemon Development

Comments

@KenSuenobu
Copy link
Owner

KenSuenobu commented May 9, 2018

Create a Job Payload object that can be shared amongst Tasks. This will eventually be the payload that is serializable (either as the entire payload, or in chunks as Spark handles it.)

This is not distributed, meaning, the data format will not be one that is serializable over a network. At least, not yet. That will be coming in a later release.

@KenSuenobu KenSuenobu added enhancement New feature or request help wanted Extra attention is needed difficult Feature that may be difficult to implement labels May 9, 2018
@KenSuenobu KenSuenobu added this to the 0.1.0 milestone May 11, 2018
@KenSuenobu KenSuenobu modified the milestones: 0.1.0, 0.3.0 May 26, 2018
@KenSuenobu
Copy link
Owner Author

This is really appropriate for job distribution and when the jobs run in a server environment. Tasks can currently share data between each other using a common class object, so this makes the ticket irrelevant for 0.1.0 milestone.

@KenSuenobu KenSuenobu added serde (De)serialization server Server/Daemon Development labels May 26, 2018
@KenSuenobu
Copy link
Owner Author

Sharable object must be a case class or something that can be serde using a library of sorts.

@KenSuenobu
Copy link
Owner Author

Good candidate for Arrow?

@KenSuenobu KenSuenobu modified the milestones: 0.3.0, 0.3.2 Jul 23, 2018
@KenSuenobu KenSuenobu removed this from the 0.3.2 milestone Dec 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
difficult Feature that may be difficult to implement enhancement New feature or request help wanted Extra attention is needed serde (De)serialization server Server/Daemon Development
Projects
None yet
Development

No branches or pull requests

1 participant