Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass in the filename to the mapper script as an argument #60

Open
jankatins opened this issue Jan 22, 2021 · 2 comments
Open

Pass in the filename to the mapper script as an argument #60

jankatins opened this issue Jan 22, 2021 · 2 comments

Comments

@jankatins
Copy link
Member

It would be nice in certain debugging scenarios to actually know the filename (or any parts in that filename) in the mapper scripts (e.g. to write it to the row in the final table).

It would basically adding -- "{self.file_name}"in the mapper script part of the pipe:

https://github.com/mara/mara-pipelines/blob/master/mara_pipelines/commands/files.py#L65-L71

                f'{uncompressor(self.compression)} "{pathlib.Path(config.data_dir()) / self.file_name}" \\\n' \
                + (f'  | {shlex.quote(sys.executable)} "{self.mapper_file_path()}" -- "{self.file_name}" \\\n' # <- changed

As far as I understand, as current mappers do not get any args, none should fail if they get one now... @hz-lschick @martin-loetzsch ?

@ghost
Copy link

ghost commented Jan 22, 2021

I currently don't use mapper scripts, so, feel free to change it 👍

@ghost
Copy link

ghost commented Jan 22, 2021

P.S. I already found several parts where breaking changes should be done. E.g the tables are still called after the old repo name (data integration) and the cli call 'mara_pipelines.ui.run' IMHO should be 'mara_pipelines.run'.
This could be all done in a new branch for version 4 to not make any breaking changes in version 3. Just an idea 💡. Would love a virtual community meet/call where such things could be discussed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant