Attribute Error when Using Multiprocessing

[ZachPence]

Hello,
The following error is thrown when I attempt to use the multiprocessing module when the function contains Galois fields:

Exception in thread Thread-8: <-- Author's Note: Number varies. This is just an example
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 576, in _handle_results
    task = get()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'FieldArray_3_2' on <module 'galois._fields._factory' from '{directory to Python files}\\Python\\Python39\\site-packages\\galois\\_fields\\_factory.py'>

Below is an example demonstrating the point:

from galois import GF, Poly
from multiprocessing import Pool # Issue also arises with the "multiprocess" module

def f(inputValue, galoisField):
    # Do stuff with field
    return inputValue
    
def g(inputValue, fieldSize):
    GFp = GF(fieldSize)
    poly = Poly((inputValue,1), GFp)
    return poly
    

def main():
    p = 2 # Works just fine with binary
    # p = 3 # Switching to a non-binary field causes issues
    inputs = (1,2,3,4,5)
    
    # Passing in a galois field as an argument
    with Pool(2) as pool:
        results = [pool.apply_async(f, args=(i,GF(p))) for i in inputs]
        outputs = [r.get() for r in results]
    print(f"Output from f: {outputs}")
       
    # Constructing galois field inside the function
    with Pool(2) as pool:
       results = [pool.apply_async(g, args=(j,p)) for j in range(p)]
       outputs = [r.get() for r in results]
    print(f"Output from g: {outputs}")
     
if __name__ == "__main__":
    main()

Key Points:

• Both f and g work fine with binary (p=2)
• • I suspect that is because binary is treated as a special case in the source code
• Likewise, the exception is only thrown when p >= 3. Or at least, I have checked when p=3,5
• The exception is thrown both when the field itself is passed through as an argument (modeled using the function f) as well as when the field-size is an input and the field is constructed in the function (modeled with the function g).

Things I have tried:

• See if constructing the field in the function could be used as a workaround. It could not, hence why the function "g" is present.
• Running the program in the terminal. I'm using spyder/anaconda, so I wanted to see if that was part of the issue.
• Made sure the version of Galois and multiprocessing are compatible.
• • Galois version v.0.3.5, Multiprocessing v.2.6.2.1, Multiprocess v.0.70.14
• Switched from using "multiprocessing" to "multiprocess."

Things I have not tried:

• See if it is only an issue with "Pool".
• Used another method than apply_async, such as map.
• I've only noticed the exception is raised when "get()" is called, when it doesn't the code runs. Though I have not investigated further.
• Sometimes 2-3 processes will run before the exception is raised. I have not investigated further.

I do not know whether if this is an issue of Galois, multiprocessing, how they interact, or if it is on my end/human error.
Link to a similar issue: #388

[raeudigerRaeffi]

I have encountered the same problem, my current workaround is to also construct the field again in the called functions by passing the degree and characteristic, however the average time to reconstruct a field is around 0.2 - 0.7 secs which defeats the purpose.

[mhostetter]

@ZachPence sorry for the delayed reply.

Creating GFp = GF(p) once and then using GFp inside of with Pool(2) as pool: seems to re solve the issue. Is that satisfactory to you?

from multiprocessing import Pool  # Issue also arises with the "multiprocess" module

from galois import GF, Poly


def f(inputValue, galoisField):
    # Do stuff with field
    return inputValue


def g(inputValue, fieldSize):
    GFp = GF(fieldSize)
    poly = Poly((inputValue, 1), GFp)
    return poly


def main():
    # p = 2  # Works just fine with binary
    p = 3  # Switching to a non-binary field causes issues
    inputs = (1, 2, 3, 4, 5)

    # Passing in a galois field as an argument
    GFp = GF(p)
    with Pool(2) as pool:
        results = [pool.apply_async(f, args=(i, GFp)) for i in inputs]
        outputs = [r.get() for r in results]
    print(f"Output from f: {outputs}")

    # Constructing galois field inside the function
    with Pool(2) as pool:
        results = [pool.apply_async(g, args=(j, p)) for j in range(p)]
        outputs = [r.get() for r in results]
    print(f"Output from g: {outputs}")


if __name__ == "__main__":
    main()

$ python3 issue_497.py 
Output from f: [1, 2, 3, 4, 5]
Output from g: [Poly(1, GF(3)), Poly(x + 1, GF(3)), Poly(2x + 1, GF(3))]