Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the memory backend usability and testing #1816

Merged
merged 1 commit into from Feb 20, 2018

Conversation

johnynek
Copy link
Collaborator

This makes the memory source and sinks more realistic: we can read and write using a Future, so this seems like it is actually something you could potentially use in production.

Some poor-man's benchmarking shows this is a lot faster than cascading local mode:

some example timings:

scalding: 1.24314 ms
cascading: 210.13103 ms

scalding: 0.497229 ms
cascading: 205.542811 ms

scalding: 1.300902 ms
cascading: 207.588402 ms

scalding: 9.135427 ms
cascading: 248.029985 ms

scalding: 0.4009 ms
cascading: 0.750325 ms

scalding: 0.276766 ms
cascading: 0.656262 ms

scalding: 6.968218 ms
cascading: 218.509877 ms

scalding: 3.855046 ms
cascading: 214.557583 ms

Of course we can optimize more and look better use parallelism.

@ianoc
Copy link
Collaborator

ianoc commented Feb 20, 2018

showing up red in ci

but all the changes lgtm, merge when green

@johnynek
Copy link
Collaborator Author

yeah, this hit #1814 and #1804
restarted.

@johnynek johnynek merged commit fe12eb6 into develop Feb 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants