Error: "numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares"

[NekoAlosama]

Describe the bug
When using results\update_results.py, the program produces the error shown in the title numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares.

To Reproduce
py -3 results\update_results.py on the current master.

Expected behavior
Results would be generated.

Console Output

(venv) %USERPROFILE%\Downloads\GitHub\AutoEq-master>py -3 results\update_results.py

Processing oratory1990 on-ear measurements...
1/259 (0.4%) 11s: Adam SP-5
 ** On entry to DLASCLS parameter number  4 had an illegal value
 ** On entry to DLASCLS parameter number  4 had an illegal value
Traceback (most recent call last):
  File "results\update_results.py", line 256, in <module>
    main()
  File "results\update_results.py", line 135, in main
    batch_processing(
  File "%USERPROFILE%\Downloads\GitHub\AutoEq-master\autoeq.py", line 84, in batch_processing
    peq_filters, n_peq_filters, peq_max_gains, fbeq_filters, n_fbeq_filters, fbeq_max_gain = fr.process(
  File "%USERPROFILE%\Downloads\GitHub\AutoEq-master\frequency_response.py", line 1665, in process
    self.smoothen_fractional_octave(
  File "%USERPROFILE%\Downloads\GitHub\AutoEq-master\frequency_response.py", line 1214, in smoothen_fractional_octave
    self.smoothed = self._smoothen_fractional_octave(
  File "%USERPROFILE%\Downloads\GitHub\AutoEq-master\frequency_response.py", line 1179, in _smoothen_fractional_octave
    y_normal = savgol_filter(y_normal, self._window_size(window_size), 2)
  File "%USERPROFILE%\AppData\Roaming\Python\Python38\site-packages\scipy\signal\_savitzky_golay.py", line 337, in savgol_filter
    coeffs = savgol_coeffs(window_length, polyorder, deriv=deriv, delta=delta)
  File "%USERPROFILE%\AppData\Roaming\Python\Python38\site-packages\scipy\signal\_savitzky_golay.py", line 139, in savgol_coeffs
    coeffs, _, _, _ = lstsq(A, y)
  File "%USERPROFILE%\AppData\Roaming\Python\Python38\site-packages\scipy\linalg\basic.py", line 1218, in lstsq
    raise LinAlgError("SVD did not converge in Linear Least Squares")
numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares

System:

• OS: Windows 10
• Python 3.8.7
• numpy-1.18.5
• scipy-1.3.3

Additional context
I did not use my fork of AutoEQ for this, so the changes on that fork shouldn't be affecting this.

[NekoAlosama]

Updating all of the modules in requirements.txt still reproduces this error. This probably means that something does have to be downgraded.

[jaakkopasanen]

Thanks for reporting. Do you happen to have a local copy which does work? I'm wondering if this is solvable by downgrading certain libraries. We also have this same issue in Impulcifer: jaakkopasanen/Impulcifer#51. This seems to be caused by a bug in Windows but I cannot reproduce it myself. I suspect that if the bug is in Windows, no matter what versions of the dependencies are used, the bug is going to be there.

Maybe I'll just do a fallback to a simple moving average filter if the Savitzky-Golay throws this error.

[NekoAlosama]

I should have a copy, but right now I'm away from the laptop with it. I should be able to check it in a few hours.

That Impulcifier issue was from "Jul 7, 2020", although I haven't experienced the bug at any time between then and "Nov 29, 2020". I've kept my Windows and Python packages up to date (even using --update-strategy eager on pip).

Maybe it's the Python version in conflict with the Windows version I'm using? I'll have to check the version on the laptop with the old copy.

[NekoAlosama]

@jaakkopasanen It's because of a Windows 10 update, and it seems the bug comes and goes with said updates.
Specifically, the version I found the bug on was Build 19042 with Python 3.8.7. I point this out because I can use Python 3.8.7 on the machine with the old copy on Build 18363 and no errors pop up.

According to this NumPy issue, a patch that fixes this is on the Developer (not Beta or Stable) update branch of Windows 10. I'll update to the Beta (or Developer if that doesn't work) and see if this patch is there. Apparently, the way to get into the Windows Insider Program (the way to get Release Preview, Beta, and Developer builds) is bugged; You can't change the build in Settings, as it doesn't save it(?), and you can't change it through insider.windows.com, since the option to do so doesn't load (even on Edge).

[jaakkopasanen]

Thanks for checking! I'll just implement the simple moving average fallback. It's perhaps not quite as good but asking users to switch to developer realse of Windows is not realistic. Hopefully the fix comes to proper release of Windows soon.

[NekoAlosama]

I'm going to keep a copy of the Savitzky-Golay filter copy of AutoEQ until an update comes. In that way, I can notify if it works on that build.

To continue with testing, I'll rollback my Windows 10 version to see if the bug was (re?)introduced in the current build.

Edit: I've rolled back to Build 18363 and the error doesn't appear. So this is definitely a Windows issue (that will hopefully be fixed in another update).

[NekoAlosama]

@jaakkopasanen
NumPy 1.19.5, according to the release notes, has a workaround instead of waiting for Microsoft to apply the fix. The current requirements.txt restrictions prevent NumPy from updating beyond 1.18.x. From what I've tested on my fork and copy of AutoEQ, I can update all the packages to their latest version and everything seems to be working (except for scipy, the latest 1.6.0 version has a defect that replaces the from constants line to from scipy.optimize._highs.constants, will be waiting for that to be fixed).

NumPy 1.19.5 is a short bugfix release. Apart from fixing several bugs, the main improvement is the update to OpenBLAS 0.3.13 that works around the windows 2004 bug while not breaking execution on other platforms. This release supports Python 3.6-3.9 and is planned to be the last release in the 1.19.x cycle.

[jaakkopasanen]

@NekoAlosama good news! I'll think I'm going to hold off with the moving average fallback since this looks to be fixed soon enough.

Thanks a lot for keeping up with this issue!

[jaakkopasanen]

@NekoAlosama I created a branch numpy-1.19.5 which contains new requirements.txt with updated versions for numpy, scipy, matplotlib and tensorflow. Seems to be working for me and I had no issues with scipy even though I updated it to 1.6.0. Could you try this branch out and confirm if it works for you.

[NekoAlosama]

@jaakkopasanen I have a small problem with scipy-1.6.0 when trying to use update_indexes.py

(venv) C:\*\Downloads\GitHub\AutoEq-numpy-1.19.5>py -3 results\update_indexes.py
Traceback (most recent call last):
  File "results\update_indexes.py", line 14, in <module>
    from frequency_response import FrequencyResponse
  File "C:\*\Downloads\GitHub\AutoEq-numpy-1.19.5\frequency_response.py", line 22, in <module>
    from constants import DEFAULT_F_MIN, DEFAULT_F_MAX, DEFAULT_STEP, DEFAULT_MAX_GAIN, DEFAULT_TREBLE_F_LOWER, \
ImportError: cannot import name 'DEFAULT_F_MIN' from 'scipy.optimize._highs.constants' (C:\*\AppData\Local\Programs\Python\Python38\lib\site-packages\scipy\optimize\_highs\constants.cp38-win_amd64.pyd)

As I've said already, this is a known bug with 1.6.0 and should be patched in another update. However, this can be worked around by placing the from constants import... line above the from scipy... import... lines.

So the beginning of frequency_response.py could look like:

import os
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import math
import pandas as pd
from io import StringIO
from constants import DEFAULT_F_MIN, DEFAULT_F_MAX, DEFAULT_STEP, DEFAULT_MAX_GAIN, DEFAULT_TREBLE_F_LOWER, \
    DEFAULT_TREBLE_F_UPPER, DEFAULT_TREBLE_MAX_GAIN, DEFAULT_TREBLE_GAIN_K, DEFAULT_SMOOTHING_WINDOW_SIZE, \
    DEFAULT_SMOOTHING_ITERATIONS, DEFAULT_TREBLE_SMOOTHING_F_LOWER, DEFAULT_TREBLE_SMOOTHING_F_UPPER, \
    DEFAULT_TREBLE_SMOOTHING_WINDOW_SIZE, DEFAULT_TREBLE_SMOOTHING_ITERATIONS, DEFAULT_TILT, DEFAULT_FS, \
    DEFAULT_F_RES, DEFAULT_BASS_BOOST_GAIN, DEFAULT_BASS_BOOST_FC, \
    DEFAULT_BASS_BOOST_Q, DEFAULT_GRAPHIC_EQ_STEP, HARMAN_INEAR_PREFENCE_FREQUENCIES, \
    HARMAN_ONEAR_PREFERENCE_FREQUENCIES, PREAMP_HEADROOM
from scipy.interpolate import InterpolatedUnivariateSpline
from scipy.signal import savgol_filter, find_peaks, minimum_phase, firwin2
from scipy.special import expit
from scipy.stats import linregress
from scipy.fftpack import next_fast_len
import numpy as np
import urllib
from time import time
from tabulate import tabulate
from PIL import Image
import re
import warnings
import biquad

instead of from constants import... being at the end.

Other than that, the new requirements.txt is doing fine. On my fork, I'll be updating to the latest version of the packages there in order to find when a bug comes.

[NekoAlosama]

@jaakkopasanen The branch works as long as I do the fix above. I'll close this when I can confirm that the next scipy release fixes the bug above.

[jaakkopasanen]

@NekoAlosama could you try if installing scipy~=1.5.0 would work? I don't think there's any real reason to update it to 1.6.

[NekoAlosama]

@jaakkopasanen scipy 1.5.4 still works. I'm just trying out 1.6.0 to see if it's still stable. You can decide if it's better to use it in the requirements.txt.

[NekoAlosama]

@jaakkopasanen You can close this issue when you update the requirements.txt on the master branch.

[jaakkopasanen]

@NekoAlosama I updated the scipy version to ~=1.5.4 in the numpy-1.19.5 branch. Could you do a fresh venv and check that everything works for you as expected. When I get a thumbs up from you, I'll merge to master.

[NekoAlosama]

Everything seems to be working out. Of course, if any problems pop up they can just be issued,

[jaakkopasanen]

Merged to master. Closing the ticket since this has been tested in the numpy-1.19.5 branch.