Using Embedded Python with BWAPI

[Sneakyabc]

Hi there!

I've been searching for an appropriate place to ask this question, so hopefully this is a good place and someone can generously help me. I've been trying to develop an AI using pytorch to use in a BWAPI module bot. I've created a simple simulation environment for SC:BW in C++ so that I could train the pytorch models in my simulation rather than having to fiddle with constantly going in and out of the actual game. To use pytorch I've been running python code directly from C++ (#including <Python.h>), this link shows an example of how I've been using this:

https://stackoverflow.com/questions/37399930/how-do-you-run-a-python-script-from-a-c-program

I've finally gotten the AI model to a place where I'm ready to try using it within BWAPI, but I've run into some complications. I can include <Python.h> in the Example AI Module project and get it to run in the game no problem. But the moment I put in any function that actually makes use of Python.h (for instance, Py_Initialize() or PyObject_CallMethodObjArgs()) the AI module fails to load when starting the game ("Failed to load the AI Module ExampleAIModule.dll").

As this wasn't working, I also tried using TorchScript, which is a C++ library for importing wrapped Torch models, but I couldn't get that to compile.

Would any geniuses out there happen to know if this sort of thing should work with BWAPI, and what I might be missing? I'm not too familiar with compiling projects into DLLs so I'm hoping that it's a relatively simple fix but I know how complicated and messy these libraries and environments can be. I can provide more information if needed, as a quick explanation: I'm on windows 10, using VS 2017, I have 'Python310-32\include' folder included in the C/C++ Additional Include Directories options for the ExampleAIModule project and 'Python310-32\libs' added to the Linker Additional Library Directories options for the same project. With this and #include <Python.h> it compiles and runs in game just fine. But if I uncomment "Py_Initialize();" in the Module it will compile just fine and then fail to load once in game.

I'm guessing I'm not the first to try this sort of approach. I love programming and I love Starcraft so I'd be very bummed to abandon this project now, huge thanks for any help in advance.

[N00byEdge]

I would recommend opening your bot .dll in depends.exe to check if all its dll dependencies (like torch) are satisfied. You have to have all dlls your bot depends on in the DLL include path, for example in the same AI directory.

[N00byEdge]

The python debug dll would either be part of a debug build that you download or, if you are building python from source, created when you built python. If you used a release build of python, you have to build your bot in release mode and link against the release python library.

[Sneakyabc]

Okay... I'm not sure if this is progress but I've got things to report. Looking around randomly I noticed that the python310_d.dll Dependency Walker is talking about is located one folder up from the \python310-32\libs folder. I tried copying the DLL into the libs folder and nothing changed. However, after building, if I put the python310_d.dll in the same folder as the ExampleAIModule.dll, and then open the AI DLL with Dependency Walker, it DOES see the DLL and its functions.

---

---

Even if I drop both DLL files into the bwapi-data\AI folder the bot still doesn't load, though. Perhaps the BWAPI program is looking elsewhere for it?

From my understanding reading online, a DLL is dynamic, meaning the necessary function definitions are loaded at runtime. So the necessary DLL file must be present on a system in order for a program that needs its functions to run. Whereas a LIB file is (or at least can be) static, meaning the function definitions are pulled at compile time, so you don't need a DLL to run. Since I am pointing VS to LIB files and not the DLLs in the project settings, shouldn't it be compiling in a manner where I don't need the DLL Dependency Walker is seeing? Once I've built the bot, shouldn't I only have/need the ExampleAIModule.dll that is output? If I am expected to provide both DLL files, where would BWAPI be expecting to find it?

I feel like this is some kind of problem with how VS is linking everything together that I'm just not fully grasping.

[Sneakyabc]

Whelp, trying random things I figured out that if I put the python310_d.dll file in the Program Files (x86)\StarCraft folder, the AI module runs. I have VS attached to StarCraft here, which I couldn't do before because the module wasn't loading.

---

---

So I suppose this works, but why do I need the DLL in the StarCraft directory? I'm confused why the project isn't building the necessary functions into the AI Module's DLL so that I don't need another DLL for it to run. Is it common for other BWAPI bots to submit with additional DLLs they require? I'm guessing there's project settings I need to mess with to get that output.

I feel a little bad this didn't seem to be an issue with BWAPI at all and was just issues with me getting the project to link in the way I expect but hopefully if someone else has an issue like this this post will help. And thanks for the help, I know it's time consuming trying to follow what someone else is experiencing.

[N00byEdge]

I'm confused why the project isn't building the necessary functions into the AI Module's DLL so that I don't need another DLL for it to run

That's called static linking and requires a different build configuration. You should check the python resources to see how to statically link against python.

[Sneakyabc]

Yeah I have a few tabs queued up to read up on forcing static linkage. Rethinking how this should all work I think I don't want to use the Python libraries directly because then if I wanted to run my bot on another machine I'd have to ensure that other machine had python, numpy, pytorch, and everything installed on it. I think trying to get the TorchScript C++ library running would be better, but that is giving me unresolved reference errors and won't compile.
Always something to fix! Thanks for the help, I'll probably close the discussion since I think I understand why I was having problems before now.

[Sneakyabc]

Closed

[dgant]

If you get this project setup working I think folks would love to see it!