numpy.concat does not appear to work across broadcast axes (Trac #1518)

[thouis]

Original ticket http://projects.scipy.org/numpy/ticket/1518 on 2010-06-22 by trac user eob, assigned to unknown.

When I'm trying to concatenate two tensors together, the concatenate operation does not allow me to use broadcast dimensions (using newaxis) in one of them.

[thouis]

trac user eob wrote on 2010-06-22

Note: the workaround we are using is to tile the numpy.array that we wanted to use newaxis on, which works but at the expense of memory.

[thouis]

@pv wrote on 2010-06-24

Can you write a self-contained piece of example code showing what you are trying to do, what you would like to happen, and what actually happens.

From your description, it's not completely clear to me how you'd like things to work.

[thouis]

@pv wrote on 2010-06-24

If you mean this,

>>> x = np.array([1,2,3,4])
>>> y = np.array([[1,2],[3,4],[5,6],[7,8]])
>>> np.concatenate((y, np.tile(x, (4,1))), axis=1)
array([[1, 2, 1, 2, 3, 4],
       [3, 4, 1, 2, 3, 4],
       [5, 6, 1, 2, 3, 4],
       [7, 8, 1, 2, 3, 4]])

you can do it without copies:

def broadcast_view(x, ref):
    """Broadcast unit dimensions in `x` to match those in `ref` without copies"""
    strides = [0 if x.shape[j] == 1 else x.strides[j] for j in range(x.ndim)]
    shape = [ref.shape[j] if x.shape[j] == 1 else x.shape[j] for j in range(x.ndim)]
    from numpy.lib.stride_tricks import as_strided
    return as_strided(x, shape=shape, strides=strides)

>>> x = np.array([1,2,3,4])
>>> y = np.array([[1,2],[3,4],[5,6],[7,8]])
>>> np.concatenate((y, broadcast_view(x[None,:], y)), axis=1)
array([[1, 2, 1, 2, 3, 4],
       [3, 4, 1, 2, 3, 4],
       [5, 6, 1, 2, 3, 4],
       [7, 8, 1, 2, 3, 4]])
>>> broadcast_view(x[None,:], y).base.base.base is x
True

But yes, I guess np.concatenate should do this automatically.

[jluttine]

Any news/progress on np.concatenate supporting broadcasting?

[jaimefrio]

There has been some discussions about this on the list in the past, and the collective wisdom dictated that broadcasting in a concatenate operation was more likely to be masking unintended mistakes, than a useful functionality. We probably should close this.

In any case, if you would like to stirr that conversation up again, the right place would be the mailing list, not here.

[jaimefrio]

Should this be closed as a "won't fix"?

[shoyer]

Agreed. Note that manual broadcasting is now a little bit easier because we have broadcast_to.

[jaimefrio]

I just needed a tiny bit of positive reinforcement, closed it is. If anyone has a strong opinion against this, feel free to reopen.

[davidmashburn]

I found a solution that seems solid to me... just use broadcast_arrays in the form np.concatenate(np.broadcast_arrays(*arrays), axis=...). Ex: np.concatenate(np.broadcast_arrays(0, [1, 2], [[3, 4], [5, 6]]), axis=1) → array([[0, 0, 1, 2, 3, 4], [0, 0, 1, 2, 5, 6]])

[oulenz]

@davidmashburn I think there is a problem with this solution (and with manual broadcasting in general as suggested by others in this thread). I don't think it makes sense to broadcast in the dimension in which we concatenate. In other words, in your case I would only expect to end up with one column of zeros.

[boeddeker]

If someone else also finds this Issue, while searching for a solution of broadcasting in concatenate, I posted my solution at https://stackoverflow.com/questions/56357047/concatenate-with-broadcast/61061019#61061019