Theano dot products lead to segfault on Azure pipelines

[rodluger]

This is independent of starry but is causing many of the tests (in particularly those that call map.render()) to fail.

Here is a MWE of a theano operation that segfaults on Azure, and here are the failing test results. The operation is a very simple dot product of two large-ish matrices:

starry/tests/test_segfault.py

Lines 1 to 25 in 160cc49

	import numpy as np
	import theano
	import theano.tensor as tt
	import theano.sparse as ts
	import pytest
	tt.config.floatX = "float64"
	sizes = [10, 100, 1000, 10000, 100000]
	def dot_sum(x, y):
	dot = tt.dot(x, y)
	return tt.sum(dot)
	x = tt.dmatrix()
	y = tt.dmatrix()
	func = theano.function([x, y], dot_sum(x, y))
	@pytest.mark.parametrize("M", sizes)
	def test_dot_product(M, N=300, L=10):
	u = np.random.randn(M, N)
	v = np.random.randn(N, L)
	print(func(u, v))

Did the stack size change on Azure? Are we exceeding the RAM somehow? Looking into this.

[rodluger]

@twiecki Have you seen this before? It's likely an Azure RAM issue, but I was wondering if you've seen this before. The segfault happens when I try to dot a (100000, 300) matrix with a (300, 10) matrix using double precision. The issue is specific to compiled functions; calling .eval() on the node works fine.

[rodluger]

The culprit is the latest release of openblas (0.3.10, July 2020). Reverting to version 0.3.6 fixes this issue (see tests).

[rodluger]

@fbartolic @dfm and anyone else that runs into this:

This is a known issue in OpenBLAS:

numpy/numpy#16913
OpenMathLib/OpenBLAS#2732

The workaround is to

conda install openblas=0.3.6

for now if you find that starry is throwing segfaults.

Fore reference, here are the test results with openblas=0.3.6 (passing) and openblas=0.3.10 (failing).

[rodluger]

I haven't run into this issue lately, and the dev version of the code (on branch restructure, soon to be merged to master) is currently running fine on Github Actions with openblas==0.3.13.

https://github.com/rodluger/starry/runs/1953734532?check_suite_focus=true