Python 3 support for ShinySDR #145

quentinmit · 2020-07-21T09:49:36Z

I can't promise everything works but all the basic functionality seems to be there and working in my tests. I kept each logical change as a separate commit so you can easily review them. I believe all of these changes are backwards-compatible to Python 2.7. Several of the tests are failing but it seems like that was true before these commits as well. I can start trying to knock off the tests in this PR if you'd like, or I can leave that for another PR.

kpreid · 2020-07-22T01:45:15Z

My computer(s) are a bit in disarray right now, so I may be a while giving this a proper code review. So, thanks for doing this, and I'll give you some quick cheap observations. Unfortunately, the Travis CI integration seems to have not posted results here (I may have reauthorized it now, but I'm not sure) but here they are. There are two simple lints:

shinysdr/i/network/webapp.py:79:1: E305 expected 2 blank lines after class or function definition, found 0
shinysdr/i/network/export_http.py:35:1: F401 'shinysdr.i.pycompat.defaultstr' imported but unused

And there are several test failures in the Python 2.7 environment, all looking like they have to do with details of web serving (probably str/bytes/unicode confusion). It's okay to go from failing to failing on Python 3, but the tests must remain passing under Python 2 (until we drop Python 2 support, which will come no sooner than adding GNU Radio 3.8 support).

quentinmit · 2020-07-22T02:44:55Z

My computer(s) are a bit in disarray right now, so I may be a while giving this a proper code review. So, thanks for doing this, and I'll give you some quick cheap observations. Unfortunately, the Travis CI integration seems to have not posted results here (I may have reauthorized it now, but I'm not sure) but here they are. There are two simple lints:
shinysdr/i/network/webapp.py:79:1: E305 expected 2 blank lines after class or function definition, found 0
shinysdr/i/network/export_http.py:35:1: F401 'shinysdr.i.pycompat.defaultstr' imported but unused
And there are several test failures in the Python 2.7 environment, all looking like they have to do with details of web serving (probably str/bytes/unicode confusion). It's okay to go from failing to failing on Python 3, but the tests must remain passing under Python 2 (until we drop Python 2 support, which will come no sooner than adding GNU Radio 3.8 support).

I don't have a good Python 2 GR environment anymore. I guess I could see how many tests run without being able to import gnuradio. I'll take a stab at trying to fix the bytes confusion.

I didn't mention it in the PR description but this also runs on GNU Radio 3.8.

kpreid · 2020-07-22T02:47:33Z

OK, when I have my GR development environment back, I'll see if I can contribute any tweaks to get it working on both — or perhaps merge whatever subset of these commits are straightforwardly compatible.

…s they are part of a flowgraph

quentinmit · 2020-07-22T09:37:39Z

OK, when I have my GR development environment back, I'll see if I can contribute any tweaks to get it working on both — or perhaps merge whatever subset of these commits are straightforwardly compatible.

I just pushed a fix to the only test that I could find broken on 2.7 (shinysdr.i.test_db.TestDatabaseResource). And the lint fixes. With that I get:

Ran 547 tests in 7.796s

PASSED (skips=12, successes=535)

There are definitely still test failures left on Python 3.8, though.

I don't know how disarrayed your computers are right now, but if you have a computer that can run Docker you can pull the w1xm/shinysdr Docker image if you want a Python 3.8 + GR 3.8 + ShinySDR dev environment. There's also w1xm/base:python2 if you want GR 3.8 + Python 2.7, though it doesn't have ShinySDR preinstalled.

quentinmit · 2020-07-22T11:09:35Z

I can't leave it alone, so I ran trial3 inside docker and got it down to:

Ran 547 tests in 11.684s

FAILED (skips=5, failures=3, successes=539)

trial2 is still passing as well, and running it on Py3.8 + GR3.8 seems to cursorily work. As part of the most recent commit I think I fixed the OsmoSDR source (we don't use it, though, so I can't be sure).

The only remaining failing tests on Python 3.8 are:

shinysdr.i.network.test_export_ws.TestStateStream.test_two_references
shinysdr.i.test_main.TestMain.test_minimal
shinysdr.i.test_main.TestMain.test_persistence

AFAICT the last two (and maybe also the first?) appear to be testing the order of items in a dictionary, which was not defined behavior and changed in Python 3.7.

quentinmit · 2020-08-08T05:21:07Z

@kpreid Hey, this has been sitting for a couple weeks... any chance to look at this yet?

kpreid

Code review only; I haven't run these myself yet.

kpreid · 2020-08-08T14:00:01Z

setup.py

+try:
+    from urllib.request import urlretrieve
+except ImportError:
+    from urllib import urlretrieve


Add a comment indicating this is for version compatibility so we can strip it out later.

kpreid · 2020-08-08T14:00:19Z

shinysdr/i/blocks.py

+        self.__fft_sink = self.__fft_cell.create_sink_internal(numpy.dtype((numpy.int8, output_length)))
+        self.__scope_sink = self.__scope_cell.create_sink_internal(numpy.dtype(('c8', self.__time_length)))


Why are these being made attributes?

Letting Python garbage collect them while they're still in the flowgraph causes crashes now.

kpreid · 2020-08-08T14:02:24Z

shinysdr/i/db.py

@@ -186,11 +186,11 @@ def __init__(self, db, instantiate):
        self.__instantiate = instantiate

    def render_GET(self, request):
-        request.setHeader(b'Content-Type', b'application/json')
+        request.setHeader(b'Content-Type', b'application/json; charset=utf-8')


application/json does not have a charset parameter; this change and its other instances below should be reverted. https://tools.ietf.org/html/rfc7159

I get what the RFC says, but I believe this was in fact necessary to make it work because something else started setting/requiring it. I'll try to reproduce the error so I can post it here.

(I certainly didn't spontaneously decide to add this all over the place!)

I can't reproduce whatever error I saw (and this was months ago so I've long since forgotten). I reverted these changes for now as a separate commit so I can easily re-add them if/when I figure out why they were necessary in the first place.

shinysdr/i/network/export_http.py

shinysdr/i/network/session_http.py

kpreid · 2020-08-08T14:10:57Z

shinysdr/i/network/webapp.py

@@ -50,13 +51,43 @@
 from shinysdr.values import SubscriptionContext


+# txWS is unfortunately dead upstream, and has buggy Python 3 support.


:(

I have a dead branch that tries to move to Autobahn, but I was having trouble getting it to even show me any error stacks from my callbacks not working.

kpreid · 2020-08-08T14:15:02Z

shinysdr/i/network/webapp.py

    for resource_def in getPlugins(_IClientResourceDef, shinysdr.plugins):
        # Add the plugin's resource to static serving
-        plugin_resources.putChild(resource_def.key, resource_def.resource)
+        plugin_resources.putChild(resource_def.key.encode(), resource_def.resource)


This one definitely needs to be 'utf-8' as the plugin could put arbitrary characters in its names.

Does that mean it also needs to be URL-encoded? Otherwise done.

shinysdr/i/test_db.py

shinysdr/plugins/rtl_433.py

quentinmit · 2020-10-11T04:13:36Z

It's been three weeks, have you had a chance to get your GR development environment working yet?

kpreid · 2020-10-11T04:40:51Z

Sorry, I've been very busy dealing with various further life complications, and also a bit of obsession with another project. This is still on my todo list.

alphafox02 · 2021-01-16T15:45:56Z

I’ve been checking back here from time to time. I’d like to add shinysdr back to DragonOS (project of mine). It’s already got python3/gnuradio 3.8 included. Just curious if there’s any movement planned on this or if I should just try to piece it together based on this PR and test it that way? Thanks

alphafox02 · 2021-01-16T17:35:29Z

Tested it real quick and found that it works rather well. Thanks for doing this PR. I went ahead and also installed gr-air-modes and gr-dad. I had pretty much everything else that was needed, except gr-radioteletype which looks to be for an older gnuradio.

One thing I noticed, I can see mode S info in the terminal but am I supposed to be seeing the messages decoded somewhere in the shinysdr GUI? I have the map up and do not see anything there.

alphafox02 · 2021-01-17T01:07:58Z

I think I figured out that for whatever reason I’m not getting positions on Adsb, but I do see mode S messages.

Another note, I thought I’d get lucky and this would work,

mcapdeville/gr-radioteletype@69b6b18

But after installing the 3.8 version shiny SDR would not start till I uninstalled gr-radioteletype. Thought maybe I’d have all the features working, so close.

NeuroForLunch · 2021-05-01T18:12:57Z

Love this project. I hope this branch gets completed and this project upgraded to python 3!

revivec · 2023-02-18T03:13:40Z

我现在正在开始的迁移工作，从gnuradio3.7+python2.7迁移到gnuradio3.9+python3.8 正在迁移的过程中，目前通过python setup.py build 和 sudo python setup.py install 返回的错误提示修改了两处位置。

在setup.py 中 urllib改成了urllib.request
关于rational_resampler
#from gnuradio.filter import rational_resampler

try:
from gnuradio.filter import rational_resampler
except ImportError:
dirname, filename = os.path.split(os.path.abspath(file))
path.append(os.path.join(dirname, "bindings"))
from gnuradio.filter.filter_python import rational_resampler

并且升级了 Twisted six txWS PyEphem pyserial等相关依赖，但是目前还是存在问题。
是否有成功的给与我一些建议。谢谢。
`

s-martin · 2023-05-20T18:57:25Z

Hi, are there any plans to merge this PR?

alphafox02 · 2023-05-20T19:53:47Z

Be even better to see if support gr 3.9+

kpreid added the refactoring label Jul 22, 2020

quentinmit added 16 commits July 22, 2020 05:33

Update setup.py for Python 3

fa08731

Update testutil.py for Python 3

5d8c982

More Python 3 support

f16517b

Work around Twisted Python 3.8 compatibility issue

c7a0606

Call putChild with byte strings for Python 3 compatibility

1267477

Fix make_websocket_url to use strings on Python 3

7b882d4

Filenames and MIME types are strings in Python 3

9d77173

Return JSON as utf-8

97e2ef0

Parse WebSocket path correctly on Python 3

c9dd568

Python blocks must have a reference held on the Python side as long a…

480d0cf

…s they are part of a flowgraph

Monkey-patch txWS to fix it on Python 3

55f2ece

Fix BlockResource for Python 3

48bcccc

sort can fail on Python 3 if objects are not comparable

9089b68

Iterators no longer have a next() method in Python 3

1529d7f

Filenames are not bytes in Py3

291496f

Handle application/json headers with and without charset specified

b5b609c

quentinmit force-pushed the python3 branch from 583f58c to b5b609c Compare July 22, 2020 09:33

Address lint errors

3a7529b

More bytes/str changes that fix tests on Python 3.8

11b6d2f

quentinmit force-pushed the python3 branch from 8b1e6e2 to 11b6d2f Compare July 22, 2020 11:10

Fix lint

9835805

kpreid requested changes Aug 8, 2020

View reviewed changes

Address review comments

d4d6684

Test removing charset from application/json headers

0f0992b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3 support for ShinySDR #145

Python 3 support for ShinySDR #145

quentinmit commented Jul 21, 2020

kpreid commented Jul 22, 2020

quentinmit commented Jul 22, 2020

kpreid commented Jul 22, 2020

quentinmit commented Jul 22, 2020 •

edited

quentinmit commented Jul 22, 2020

quentinmit commented Aug 8, 2020

kpreid left a comment

kpreid Aug 8, 2020

quentinmit Sep 17, 2020

kpreid Aug 8, 2020

quentinmit Sep 17, 2020

kpreid Aug 8, 2020

quentinmit Sep 17, 2020

quentinmit Sep 17, 2020

kpreid Aug 8, 2020

kpreid Aug 8, 2020

quentinmit Sep 17, 2020

quentinmit commented Oct 11, 2020

kpreid commented Oct 11, 2020

alphafox02 commented Jan 16, 2021

alphafox02 commented Jan 16, 2021

alphafox02 commented Jan 17, 2021

NeuroForLunch commented May 1, 2021

revivec commented Feb 18, 2023 •

edited

s-martin commented May 20, 2023

alphafox02 commented May 20, 2023

		self.__fft_sink = self.__fft_cell.create_sink_internal(numpy.dtype((numpy.int8, output_length)))
		self.__scope_sink = self.__scope_cell.create_sink_internal(numpy.dtype(('c8', self.__time_length)))

		@@ -50,13 +51,43 @@
		from shinysdr.values import SubscriptionContext


		# txWS is unfortunately dead upstream, and has buggy Python 3 support.

Python 3 support for ShinySDR #145

Are you sure you want to change the base?

Python 3 support for ShinySDR #145

Conversation

quentinmit commented Jul 21, 2020

kpreid commented Jul 22, 2020

quentinmit commented Jul 22, 2020

kpreid commented Jul 22, 2020

quentinmit commented Jul 22, 2020 • edited

quentinmit commented Jul 22, 2020

quentinmit commented Aug 8, 2020

kpreid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

quentinmit commented Oct 11, 2020

kpreid commented Oct 11, 2020

alphafox02 commented Jan 16, 2021

alphafox02 commented Jan 16, 2021

alphafox02 commented Jan 17, 2021

NeuroForLunch commented May 1, 2021

revivec commented Feb 18, 2023 • edited

s-martin commented May 20, 2023

alphafox02 commented May 20, 2023

quentinmit commented Jul 22, 2020 •

edited

revivec commented Feb 18, 2023 •

edited