Support Windows. Help wanted 🙏🪟

[kantord]

At all steps, SeaGOAT is developed with the intention of guaranteeing Windows support. That being said, as I don't have a Windows device and I don't have expertise regarding Windows development workflows, I have not been able to verify that it works properly on Windows.

The main goal is to support WSL, but supporting Windows natively is also a goal if it's possible!

Help wanted!

Here is a rough checklist of the current status of Windows support:

• Enable CI checks for Windows
• Enable tests for WSL if it's possible and necessary with GitHub actions
• Make sure that all CI tests pass for Windows
• Manually test SeaGOAT on Windows
• Document installation steps on Windows
• Make sure that any required Windows-specific features are implemented
• Ensure it's easy to install on Windows using common tools
• Currently, the main problem is that I have not been able to try and run SeaGOAT manually on Windows.

Also, I have discovered that some of the tests are failing on Windows. So, I created a CI file specifically for Windows that only checks some of the tests that are passing without issue:

SeaGOAT/.github/workflows/test-windows.yml

Lines 56 to 58 in 1edae8e

	- name: Run pytest
	run: |
	poetry run pytest tests/test_repository.py -vvs --timeout=60

For comparison, the Linux tests look like this:

SeaGOAT/.github/workflows/test.yml

Lines 56 to 58 in 1edae8e

	- name: Run pytest
	run: |
	poetry run pytest . -vvs --timeout=60

[DeveloperPaul123]

I'm trying this out on a decently sized project and starting the server takes a long time. It seems that my computer's resources, however, are not being fully utilized. It might be good to look into maximizing CPU usage while the server is indexing the repo. Also, being able to exclude certain folders inside the repo folder would also be nice.

[kantord]

Reference in n

You are able to query the repo while it's being indexed, and the most useful files are being prioritized for indexing. For this reason the intended use case is to allow you to work normally on your computer while the repo is being indexed, which is why CPU utilization is not being maximized. However maybe it would make sense to allow the user to customize this behavior

[DeveloperPaul123]

Thanks for the tip, seems like a reasonable design choice.

Edit: I'm dumb, I installed ripgrepy via pip, not ripgrep via chocolatey

---

When I try to query, I'm getting an error however. It seems to think that I don't have ripgrep installed though I've ensured that I do.

2023-09-20 10:51:49,040 Handling task: query
Exception in thread Thread-1 (_worker_function):
Traceback (most recent call last):
  File "C:\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "C:\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\queue\base_queue.py", line 79, in _worker_function
    self._handle_task(context, task)
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\queue\base_queue.py", line 66, in _handle_task
    result = handler(context, *task.args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\queue\task_queue.py", line 78, in handle_query
    context["seagoat_engine"].fetch_sync(
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\engine.py", line 163, in fetch_sync
    loop.run_until_complete(self.fetch(*args, **kwargs))
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\nest_asyncio.py", line 99, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "C:\Python311\Lib\asyncio\tasks.py", line 269, in __step
    result = coro.throw(exc)
             ^^^^^^^^^^^^^^^
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\engine.py", line 149, in fetch
    results = await asyncio.gather(*async_tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\tasks.py", line 339, in __wakeup
    future.result()
  File "C:\Python311\Lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\sources\ripgrep.py", line 45, in fetch
    return _fetch(query_text, str(path), limit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\sources\ripgrep.py", line 16, in _fetch
    Ripgrepy(query_text, path)
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\ripgrepy\__init__.py", line 131, in __init__
    raise RipGrepNotFound("ripgrep not found")
ripgrepy.RipGrepNotFound: ripgrep not found

Output from pip list

Is there something I'm doing wrong?

[DeveloperPaul123]

Now that I've resolved the ripgrep issue 😅 , I'm getting another error

Exception in thread Thread-1 (_worker_function):
Traceback (most recent call last):
  File "C:\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "C:\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\queue\base_queue.py", line 79, in _worker_function
    self._handle_task(context, task)
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\queue\base_queue.py", line 66, in _handle_task
    result = handler(context, *task.args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\queue\task_queue.py", line 83, in handle_query
    results = context["seagoat_engine"].get_results(kwargs["limit_clue"])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\engine.py", line 208, in get_results
    sorted(
  File "C:\Users\Paul\.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\engine.py", line 212, in <lambda>
    + 0.3 * normalize_file_position(top_files[x.path])
                                    ~~~~~~~~~^^^^^^^^
KeyError: '<redacted>'

[GautierT]

@DeveloperPaul123 : it's because the file path in top_files is in linux format.

Temp fix :
Replace top_files[x.path] by top_files[Path(x.path).as_posix()] in engine.py (C:\Users\XXXXXX.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\engine.py)

[kantord]

Now that I've resolved the ripgrep issue 😅 , I'm getting another error

Out of curiosity, how did you resolve the ripgrep issue? I was thinking that if it wasn't detecting your ripgrep installation originally, it might be a bug in ripgrepy itself: https://github.com/securisec/ripgrepy maybe we could help them fix the problem or at least submit an issue there.

Also if you used a specific workaround we could mention that in the readme of SeaGOAT

[kantord]

@DeveloperPaul123 : it's because the file path in top_files is in linux format.

Temp fix : Replace top_files[x.path] by top_files[Path(x.path).as_posix()] in engine.py (C:\Users\XXXXXX.local\pipx\venvs\seagoat\Lib\site-packages\seagoat\engine.py)

ok, actually we have something similar here:

SeaGOAT/seagoat/utils/server.py

Lines 21 to 22 in 51ae32c

	def normalize_repo_path(repo_path: str) -> str:
	return str(os.path.normpath(Path(repo_path).expanduser().resolve()))

we can probably reuse that

[DeveloperPaul123]

Now that I've resolved the ripgrep issue 😅 , I'm getting another error

Out of curiosity, how did you resolve the ripgrep issue? I was thinking that if it wasn't detecting your ripgrep installation originally, it might be a bug in ripgrepy itself: https://github.com/securisec/ripgrepy maybe we could help them fix the problem or at least submit an issue there.

Also if you used a specific workaround we could mention that in the readme of SeaGOAT

I just installed ripgrep via chocolatey as I didn't have it installed previously. Maybe I missed something, but I didn't realize that ripgrep was another command line tool that I needed to install along with SeaGOAT. I had installed ripgrepy via pip, but didn't have ripgrep available. It seems that ripgrepy assumes you have ripgrep installed and that it is available on your path.

[kantord]

I have merged this, now with the latest version of SeaGOAT, the path-related error should not happen on Windows. If either of you can try it, please confirm if it works or not 🙏 #234

[DeveloperPaul123]

@kantord Thanks for the the fixes, things seem to be working correctly now!

[kantord]

@kantord Thanks for the the fixes, things seem to be working correctly now!

amazing! let me know if you encounter other problems. also please let me know about your use cases and whether or not you find that the results are useful for your use cases.

[DeveloperPaul123]

@kantord Thanks for the the fixes, things seem to be working correctly now!

amazing! let me know if you encounter other problems. also please let me know about your use cases and whether or not you find that the results are useful for your use cases.

Will do thanks!

As for my use case, I occasionally use it to find code in my projects. I may use it in the future to help me replace code blocks or explore legacy code bases.