backgroundjobs related magic functions are all missing

[digitalsatori]

sorry for the dumb question, but where are those magics:
%jobs, %bg etc.

ipython prompted Magic function 'xxx' not found whenever I type these magic phrase in ipython, and it seems really missing in the "core/magic.py" . I also found that nowhere in ipython actually refer to the lib/backgroundjobs.py where background jobs handling process defined.

[fperez]

Hi, I'm afraid it was a victim of the large refactoring that took place towards 0.11. I don't recall right now the precise reasons that led to this one being cut out, it could have been accidental, but @bgranger might have a better recollection as he did the grunt work of that large reorganization.

Part of the problem is that this functionality was all threads-based, and in python, starting background threads for anything cpu intensive isn't a very good idea. But I can see how it could be useful for certain scenarios, and if we can bring it back without creating problems for the new Qt console or notebook, we can look into it.

Could you give us some feedback on your use scenarios and how important this is for you? Feedback like this will help us gauge how critical this should be in terms of prioritizing effort.

Note that the code is all there, it's just in the 0.10.2 tag in the git repository. So it wouldn't be too hard to revive it, if someone steps up to help out and we do it with proper documentation and testing.

[digitalsatori]

Thank you very much for the reply.
I was just trying out stuffs mentioned in the on-line document, no
particular use scenario. I will be happy to do the test though if some of
you consider it no harm and bring it back.

On Mon, Oct 10, 2011 at 5:04 AM, Fernando Perez <
reply@reply.github.com>wrote:

Hi, I'm afraid it was a victim of the large refactoring that took place
towards 0.11. I don't recall right now the precise reasons that led to this
one being cut out, it could have been accidental, but @bgranger might have a
better recollection as he did the grunt work of that large reorganization.

Part of the problem is that this functionality was all threads-based, and
in python, starting background threads for anything cpu intensive isn't a
very good idea. But I can see how it could be useful for certain scenarios,
and if we can bring it back without creating problems for the new Qt console
or notebook, we can look into it.

Could you give us some feedback on your use scenarios and how important
this is for you? Feedback like this will help us gauge how critical this
should be in terms of prioritizing effort.

Note that the code is all there, it's just in the 0.10.2 tag in the git
repository. So it wouldn't be too hard to revive it, if someone steps up to
help out and we do it with proper documentation and testing.

Reply to this email directly or view it on GitHub:
#844 (comment)

[fperez]

On Sun, Oct 9, 2011 at 6:28 PM, digitalsatori
reply@reply.github.com
wrote:

Thank  you very much for the reply.
I was just trying out stuffs mentioned in the on-line document, no
particular use scenario. I will be happy to do the test though if some of
you consider it no harm and bring it back.

Well, there's a reasonable amount of work involved in bringing it
back, and I'm afraid I don't have the resources to work on this right
now. So it would require an interested user who needs it, to invest
some time on the effort. The main code is in lib/bacgkroundjobs.py,
and we can get the magic back from the 0.10.x tags. It would be a
matter of reworking that code, validating it in the various user
environments (terminal, qt console, notebook) and adding proper tests
to it.

Interesting and possibly useful, but at this point in time somewhat
low-priority, I'm afraid.

I'll leave this open though, so others can find it, and if an
interested user (including yourself) wants to jump on it we'll be
happy to review any relevant Pull Requests.

[fperez]

Please see gh-856 for further details. When that gets merged, some of this functionality will indeed come back.

[minrk]

closed by PR #856

[fperez]

@minrk, reopening it b/c I never brought %bg back. So there's a tiny bit of work left for some interested soul, but now with the bacgkround job manager present, updating the magic should be easy. I'd left this one open to remind me of that fact.

[minrk]

Oh, sorry. There was a series of Issues that should have been autoclosed by PRs that weren't, and I guess I got overzealous.

[fperez]

On Tue, Oct 18, 2011 at 4:33 PM, Min RK
reply@reply.github.com
wrote:

Oh, sorry.  There was a series of Issues that should have been autoclosed by PRs that weren't, and I guess I got overzealous.

No worries! I'm glad to see you closing, I definitely have a similar
urge to bring our open PR count closer to 0 and our open issue count
under control. Ideally we'd have by 0.12 just one or two lingering
open PRs, and I'd like to have our issue count below 100, with most of
those being low priority or enhancement.. Right now we have ~40 with
type-bug and prio-{med/high/critical}.

And an unknown number non-triaged (without labels).

Cheers,

f

[minrk]

On Tue, Oct 18, 2011 at 16:38, Fernando Perez <
reply@reply.github.com>wrote:

On Tue, Oct 18, 2011 at 4:33 PM, Min RK
reply@reply.github.com
wrote:

Oh, sorry. There was a series of Issues that should have been autoclosed
by PRs that weren't, and I guess I got overzealous.

No worries! I'm glad to see you closing, I definitely have a similar
urge to bring our open PR count closer to 0 and our open issue count
under control. Ideally we'd have by 0.12 just one or two lingering
open PRs, and I'd like to have our issue count below 100, with most of
those being low priority or enhancement.. Right now we have ~40 with
type-bug and prio-{med/high/critical}.

And an unknown number non-triaged (without labels).

I've used my issues script to keep a handle on unlabeled Issues. We have
only a couple that are none of:

A) assigned to a milestone
B) marked dormant
C) labeled status-active, with priority and type

I pretty aggressively labeled most things as 0.12 milestone, so we at least
look at them before deciding to push them back to 0.13.

Cheers,

f

Reply to this email directly or view it on GitHub:
#844 (comment)

[fperez]

On Tue, Oct 18, 2011 at 4:55 PM, Min RK
reply@reply.github.com
wrote:

I've used my issues script to keep a handle on unlabeled Issues.  We have
only a couple that are none of:

A) assigned to a milestone
B) marked dormant
C) labeled status-active, with priority and type

I pretty aggressively labeled most things as 0.12 milestone, so we at least
look at them before deciding to push them back to 0.13.

Excellent! BTW, mind putting your script in tools/? That way we can
all use it and refine it over time. I have github-stats in there, so
it might be worth merging some of the code which is probably
duplicate between the two...

No need for a PR for that, just go ahead and do it at your leisure.

[ahmadia]

This has been resolved with the new script magic, that provides a --bg flag.

Example:

%%script bash --bg --out script_out

sleep 10
echo hi!

[Carreau]

Thanks ! Closing then !

[Tagar]

It would be still great to have %bg back.. not just external script background execution.
We use ipython for Spark and some commands (like stats collection), may have to run for an hour,
but most of following cells not necessarily depend on its result. So it would be good to run
any cell in background, not just external scripts. Thank you.

[madanh]

I have Monte-Carlo simulations that run for around two hours, but may converge earlier. Useful conclusions and earlier convergence detection can be be done when running them in background and dumping intermediate results in a file. A perfect job for %bg, so please reopen

[Carreau]

I have Monte-Carlo simulations that run for around two hours, but may converge earlier. Useful conclusions and earlier convergence detection can be be done when running them in background and dumping intermediate results in a file. A perfect job for %bg, so please reopen

Magics do not have to be part of IPython to be available, you are free to publish a package on PyPI that exposes a %bg magic. Though from your use case, it seem like ipyparallel and using Python Futures might be more appropriate.