Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read numpy input, preview code, test_run/quick_run #88

Open
thoth291 opened this issue Oct 1, 2020 · 3 comments
Open

Read numpy input, preview code, test_run/quick_run #88

thoth291 opened this issue Oct 1, 2020 · 3 comments
Labels

Comments

@thoth291
Copy link

thoth291 commented Oct 1, 2020

  1. I've created 2 operations.
    One creates a numpy array: data_output = np.arange(15).reshape([3,5])
    And another one adds 10 to that array: output_data = input_data + 10
    I don't understand how to do that - and can't find examples to learn from.
    Can someone help me with that or point to better docs?

  2. How to preview an operation's code from within the workflow graph?
    Does it always using un-depreciated version or one can still point to depreciated one? Id's are non-human readable - so navigating in operations tab might get more difficult with many similarly named operations.

2.5. How to completely delete an operation or perform a cleanup by removing unused ones into some sort of archive?

  1. How to test an operation without moving to a workflows and creating a test workflow?
    It would be great to create a quick run option before approving an operation for the final workflow.
    Technically speaking I could of created a simple workflow to test my currently-in-developement-not-yet-approved operation, and once it's tested I could approve it and push to the actual workflow where it could be used.
@khaxis
Copy link
Collaborator

khaxis commented Oct 13, 2020

  1. In the current state operations use files to share data between each other. In your case you will need to write your np array to a file and read it from another operation
with open(outputs['output'], 'w') as f:
    np.save(f, data_output)
....
# Operation 2
with open(inputs['foo']) as f:
    input_data  = np.load(f)
output_data = input_data + 10

Communication using the files is a use basic case. This way is easier to distribute computation across cluster of workers, also using operations written in different languages.
But it does not have to be this way. Plynx supports Executor plugin, that can interpret code and graph itself in your way. Please let me know if you want to hear more about it! Here are some basics https://plynx.readthedocs.io/en/latest/plugins.html#executors

@khaxis
Copy link
Collaborator

khaxis commented Oct 13, 2020

  1. Click on an operation. On the right panel, a link to it will appear under "this operation". Currently the only way to preview it.
    You think it will be useful to improve it?
    2.5. I am afraid you can not delete it. It's actually rather legacy from previous version and one use case. Without going too much into details, the operations are kind of deleted if you deprecate them with "MANDATORY_DEPRECATED" flag. That means users cannot reuse them in any of the graphs. Would it be helpful?

Actually not the first time people are asking about removing an operation! Thanks for noting, because I need more information about use cases

@khaxis
Copy link
Collaborator

khaxis commented Oct 13, 2020

  1. I am pretty sure, that would be an amazing feature! Unfortunately, didn't have time to do it. There is a lot of things to think of and implement. You think it should be higher in priority?

Currently you can click "preview" and run the resulting script locally... Not best option but definitely easiest for now

@khaxis khaxis added the feature label Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants