NumPy-like Functionality Request Rollup

[mruberry]

PyTorch often receives requests for NumPy-like functionality. This issue is a "rollup" of these requests. It will be continue to be updated with additional requests and helpful links for implementing them.

Newer contributors, checkout the Wiki and read the Contribution Guide for help getting started.

If one of these issues requires additional discussion then, to avoid cluttering this rollup, please create a new issue titled “Implementing [Request]”, label it with “module: numpy”, and start the more focused discussion there.

Functions (Updated for the PyTorch 1.8 release):

• divmod (reserved for onboarding)
• frexp
• dsplit, hsplit, and vsplit
• block
• diag_indices
• take_along_axis
• histogram
• histogram_bin_edges
• isin(Function request: np.isin #3025) (reserved for bootcamp)
• let clamp accept tensors (Support clamp() with tensor min and max #2793)
• corrcoef, requested in Function request: np.corrcoef #1254

Completed Functions

• diff, implemented by @soulitzer in Implement np.diff for single order differences #50569
• any and all supporting all dtypes, implemented by @kshitij12345 in [numpy] torch.{all, any} : Extend Dtype Support #44790
• broadcast_to, implemented by @RockingJavaBean in Implementing NumPy-like function torch.broadcast_to #48997
• xlogy, see original issue Implement torch.xlogy #22656, implemented by @kshitij12345 in [numpy] Add torch.xlogy #48777
• floatpower, implemented by @Kiyosora in Implement NumPy-like function torch.float_power() #44937
• msort, added by @Kiyosora in Implement NumPy-like function torch.msort() #48440
• ldexp, added by @ranman in Add basic ldexp operator for numpy compatibility #45370
• column_stack and row_stack, implemented by @RockingJavaBean in implement NumPy-like functionality column_stack, row_stack #46313
• kron, implemented by @IvanYashchuk in Added Kronecker product of tensors (torch.kron) #45358
• ravel, implemented by @ejguan in Implement ravel #46098
• swapaxes, implemented by @kshitij12345 in Add torch.swapdims and torch.swapaxes #46041
• tile, implemented by @arbfay in An implementation of torch.tile as requested in pytorch/pytorch#38349 #47974
• copysign, implemented by @ejguan in Implement copysign #46396
• maximum, minimum implemented by @RockingJavaBean in implement NumPy-like functionality maximum, minimum #42579
• dstack, hstack, vstack, implemented by @muthuArivoli in Implement hstack, vstack, dstack #42799
• arcosh, arcsinh, arctanh, implemented by @krshrimali in Add arcosh, arcsinh and arctanh to unary ops #38388
• atleast_1d, atleast_2d, atleast_3d, implemented by @kshitij12345 in Add torch.atleast_{1d/2d/3d} #41317
• deg2rad, rad2deg, implemented by @aayn in Implement rad2deg, deg2rad #38852
• fliplr, flipud, implemented by @kshitij12345 in Add torch.flip{lr, ud} #38599
• isneginf, isposinf, implemented by @RockingJavaBean in implement numpy-like functionality isposinf, isneginf #41588
• isreal, implemented by @Justin-Huber in torch.isreal #41298
• logaddexp, logaddexp2, implemented by @cloudhan in Implement logaddexp #38384
• signbit, implemented by @Kiyosora in Implementing NumPy-like function torch.signbit() #41589
• count_nonzero, implemented by @kshitij12345 in Add torch.count_nonzero #39992
• gcd, lcm, implemented by @aayn in Implement gcd, lcm #40651
• moveaxis (Add moveaxis function #36048), implemented by @kshitij12345 in Add torch.movedim #41480 (as movedim)
• quantile, implemented by @heitorschueroff in Initial implementation of quantile operator #39417
• nansum, implemented by @kshitij12345 in Add torch.nansum #38628
• hypot ([feature request] torch.hypot #22764), implemented by @muthuArivoli in Implement hypot #42291
• nextafter, implemented by @muthuArivoli in Implement torch.nextafter #42580

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @mruberry @rgommers @heitorschueroff
cc @aayn @n-gao

[kshitij12345]

Hi,

I'd like to work on fliplr and flipud

[rgommers]

@mruberry there's a number of functions in there that the NumPy devs consider mistakes or are outdated. We'd like to be able to deprecate those, but backwards compat discussions. Examples are heaviside (too trivial), kaiser and all other window functions (belongs in scipy.signal), ptp (unreadable, bad idea), percentile (quantile is almost identical but slightly better), sinc (belongs in scipy.special). All the *stack functions are kind of special cases of concatenate and stack, see numpy/numpy#7183.

A lot of the other functions (e.g. deg2rad) are simply utilities that would maybe be good to have somewhere, but probably not in the main torch namespace.

We'll also have to deal with existing incompatibilies between pytorch and numpy functions/objects of the same name, e.g. mapping axis to dim, the behavior of .view() and probably many more little things.

Would it make sense to curate the above list of desired functions a little, and revisit the separate torch.numpy namespace (#2228 (comment)) for adding the utility functions?

[mruberry]

Always great to hear from you, @rgommers!

Let's follow-up offline.

[mruberry]

Hi,

I'd like to work on fliplr and flipud

That's great! But no need to announce what you're working on ;) Go for it!

[carlosgmartin]

Should take be added to the list?

[carlosgmartin]

The following seems to work for quantile:

import torch
import numpy as np
from pdb import set_trace

def take(a, indices, axis):
    return a[tuple(
        slice(a.shape[dim]) if dim != axis else indices 
        for dim in range(a.ndim)
    )]

def quantile(a, q, axis):
    i = q * (a.shape[axis] - 1)
    i_hi = torch.ceil(i).long()
    i_lo = torch.floor(i).long()
    w_hi = i - i_lo
    w_lo = 1 - w_hi
    sort = a.sort(axis).values
    v_hi = take(sort, i_hi, axis)
    v_lo = take(sort, i_lo, axis)
    return w_lo * v_lo + w_hi * v_hi

while True:
    a = torch.randn(4, 5, 6).float()
    q = torch.rand(size=())
    axis = torch.randint(a.ndim, size=())
    try:
        x = quantile(a, q, axis)
        y = torch.tensor(np.quantile(a, q, axis)).float()
        assert torch.allclose(x, y, atol=1e-6)
    except AssertionError:
        print(x)
        print(y)
        set_trace()

Note that q could potentially be a tensor rather than just a scalar.

[mruberry]

The following seems to work for [quantile]...

Yes but we need a properly registered C++ implementation since PyTorch has both Python and C++ APIs.

[mruberry]

Should take be added to the list?

"take" isn't a great candidate at the moment because we already have torch.take.

[kshitij12345]

@mruberry
outer will just be an alias for torch.ger right?

>>> import torch
>>> import numpy as np
>>> np.outer(np.ones((5,)), np.linspace(-2, 2, 5))
array([[-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.]])
>>> torch.ger(torch.ones((5,)), torch.linspace(-2, 2, 5))
tensor([[-2., -1.,  0.,  1.,  2.],
        [-2., -1.,  0.,  1.,  2.],
        [-2., -1.,  0.,  1.,  2.],
        [-2., -1.,  0.,  1.,  2.],
        [-2., -1.,  0.,  1.,  2.]])

[carlosgmartin]

"take" isn't a great candidate at the moment because we already have torch.take.

torch.take doesn't seem to take an axis like the numpy version.

[mruberry]

outer will just be an alias for torch.ger right?

Good point! I'll remove it from the list since we'll be handling aliases later. Thanks @kshitij12345!

[mruberry]

"take" isn't a great candidate at the moment because we already have torch.take.

torch.take doesn't seem to take an axis like the numpy version.

Yes, sorry I should have elaborated that this list is intended for entirely new functions, not for functions that exist in PyTorch but don't have the same behavior as those functions in NumPy. Those are out of scope for this issue. (They're totally valid separate issues, however.)

[vadimkantorov]

One problem with special numpy-like nan* ops is that without an agreed and clearly pronounced principle, many other non-numpy ops may demand nan* versions as well: e.g. topk #16762

One problem about fliplr/flipud is that they assume about which dimensions they concern, if I understand well. But in PyTorch usecases, there can be more flexible choices (as opposed to NumPy): BCHW is more popular logical format, but if someone has BHWC logical format, then what will fliplr/flipud do?

[mruberry]

I don't think the behavior of fliplr and flipud is ambiguous. You are correct that their names suggest a certain perspective on the data, but that's typical of many functions.

From my discussion with @rgommers there probably won't be many (any?) new nan- functions, but the existing ones are frequently used and have been requested.

[vadimkantorov]

About nan: an alternative way could be adding some nan-controlling option to regular torch.mean/sum/... and then redirecting these numpy nan* versions to use that option enabled. Like that, if we have pytorch (not numpy) functions wanting to support nan skipping, the interface would be the same (e.g. skip_nan = True or nan_policy = 'skip') and no function nan* aliases will have to be created

[rgommers]

From my discussion with @rgommers there probably won't be many (any?) new nan- functions

Dug up the most recent discussion on this with a summary of the feeling of the NumPy devs about this: numpy/numpy#13198 (comment)

[vadimkantorov]

Another related: SciPy.Stats seems to have nan_policy concept (not saying that any choice is much better than the other - just that the made choice should be hopefully made clear in some new "NaN/Inf handling" docs section)

[alvgaona]

Awesome, thanks @rgommers

[rgommers]

One thing I never noticed before, and that I don't find discussed in issues, is that pytorch doesn't have nan, inf, pi and e constants. At least the first three are used a lot. The test suite is full of float('nan') and float('inf') which is ugly. Doing import math; nan = math.nan would be the alternative (that'd be reasonable). Anyone know if not implementing these constants was a conscious decision?

[mruberry]

One thing I never noticed before, and that I don't find discussed in issues, is that pytorch doesn't have nan, inf, pi and e constants. At least the first three are used a lot. The test suite is full of float('nan') and float('inf') which is ugly. Doing import math; nan = math.nan would be the alternative (that'd be reasonable). Anyone know if not implementing these constants was a conscious decision?

Per offline discussion, no it wasn't an especially conscious decision. We can probably just add these.

[vadimkantorov]

torch functions for torch.add / torch.delete

[Kiyosora]

这是来自QQ邮箱的自动回复邮件。   您好，我已收到您的邮件，请放心。

[mruberry]

Closing this issue. We've added the great majority of missing NumPy functions, and future requests can be handled as individual issues.

@vadimkantorov We have torch.add(), but would you like to file an issue requesting torch.delete()? delete is a tough name to reuse, however.

[vadimkantorov]

Oh, I think mistyped, I meant append / delete. Yeah, I'll create a separate issues for discussing these

[danijar]

@mruberry I just found this issue and was quite surprised that it's closed. It seems like even basic Numpy functions like np.concatenate() are not in the PyTorch namespace yet? Are there any plans to support a Numpy-like API? That's the default in JAX and TF2 has tf.experimental.numpy with fairly complete coverage. I'm happy to open a separate issue if this one was just to provide Numpy-like functionality rather than a Numpy-like API.

[mruberry]

@mruberry I just found this issue and was quite surprised that it's closed. It seems like even basic Numpy functions like np.concatenate() are not in the PyTorch namespace yet? Are there any plans to support a Numpy-like API? That's the default in JAX and TF2 has tf.experimental.numpy with fairly complete coverage. I'm happy to open a separate issue if this one was just to provide Numpy-like functionality rather than a Numpy-like API.

Hey @danijar! It's true there are some NumPy operations and aliases still not in PyTorch, but we thought it'd be easier to track them directly than using this rollup issue, that's all.

In PyTorch we have torch.cat and torch.concat, and we'd just want to add an alias for torch.concatenate, too. If you'd like to file an issue for that alias in particular that'd be great; it should be straightforward to add.

[danijar]

That's great to hear! I'm happy to file an issue. It would just be useful to see an overview of what's available and what's still missing, it could even be a spreadsheet. That way, I can see what I need to special case when converting a given code base from JAX to PT.