Pypi release #17

Athanaseus · 2021-05-19T06:30:10Z

Add setup file
Restructure src and binaries
Add github actions

haavee · 2021-05-19T10:30:01Z

I get the following problem trying to install on a (mildly vanilla) system, mostly seems to work except the dependency:

harro@hvvm:~/src/jiveplot (pypi_release)$ python setup.py install --prefix=/tmp/test-pypi
running install
running bdist_egg
running egg_info
writing requirements to jiveplot.egg-info/requires.txt
writing jiveplot.egg-info/PKG-INFO
writing top-level names to jiveplot.egg-info/top_level.txt
writing dependency_links to jiveplot.egg-info/dependency_links.txt
reading manifest file 'jiveplot.egg-info/SOURCES.txt'
writing manifest file 'jiveplot.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib.linux-x86_64-2.7' does not exist -- no Python modules to install

creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
installing scripts to build/bdist.linux-x86_64/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
creating build/bdist.linux-x86_64/egg/EGG-INFO/scripts
copying build/scripts-2.7/standardplots -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts
copying build/scripts-2.7/klk1 -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts
copying build/scripts-2.7/jplotter -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts
changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/standardplots to 755
changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/klk1 to 755
changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/jplotter to 755
copying jiveplot.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jiveplot.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jiveplot.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jiveplot.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jiveplot.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/jiveplot-1.0.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing jiveplot-1.0.0-py2.7.egg
Removing /tmp/test-pypi/lib/python2.7/site-packages/jiveplot-1.0.0-py2.7.egg
Copying jiveplot-1.0.0-py2.7.egg to /tmp/test-pypi/lib/python2.7/site-packages
jiveplot 1.0.0 is already the active version in easy-install.pth
Installing klk1 script to /tmp/test-pypi/bin
Installing standardplots script to /tmp/test-pypi/bin
Installing jplotter script to /tmp/test-pypi/bin

Installed /tmp/test-pypi/lib/python2.7/site-packages/jiveplot-1.0.0-py2.7.egg
Processing dependencies for jiveplot==1.0.0
Searching for ppgplot@ git+git://github.com/haavee/ppgplot.git#egg=ppgplot
Reading https://pypi.python.org/simple/ppgplot/
Couldn't find index page for 'ppgplot' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or working download links found for ppgplot@ git+git://github.com/haavee/ppgplot.git#egg=ppgplot
error: Could not find suitable distribution for Requirement.parse('ppgplot@ git+git://github.com/haavee/ppgplot.git#egg=ppgplot')

haavee · 2021-05-19T10:40:24Z

On another host where the python setup.py install --prefix=/tmp/test-pypi worked fine w/o dependency errors, can't run:

[eee (pypi_release)]Okay->/tmp/test-pypi/bin/jplotter
Traceback (most recent call last):
File "/tmp/test-pypi/bin/jplotter", line 4, in
import('pkg_resources').run_script('jiveplot==1.0.0', 'jplotter')
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 719, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources/init.py", line 1511, in run_script
exec(script_code, namespace, namespace)
File "/tmp/test-pypi/lib/python2.7/site-packages/jiveplot-1.0.0-py2.7.egg/EGG-INFO/scripts/jplotter", line 4, in
import('pkg_resources').run_script('jiveplot==1.0.0', 'jplotter')
ImportError: No module named jiveplot.functional

Athanaseus · 2021-05-19T10:58:21Z

Hey @haavee thanks for checking this.

The idea is to use pip which will automatically download all dependencies. Also, it will be a good idea to upload both jiveplot and ppgplot to PyPI (This can be automated). This will resolve the dependency issues.
Another issue might be that I tested this only in python3.

cd jiveplot/
pip install .

haavee · 2021-05-19T11:21:32Z

I see now: it requires Python >= 3.6 ... which is way too ambitious; we cannot run the s/w on our own systems then (we have systems that are still 2.7 based)

haavee · 2021-05-19T11:51:56Z

I found a couple of other issues with the install on one of the test systems:

missing system PGPLOT library (apt-get install giza-dev) fixed that, but that required
missing libX11 (apt-get install libx11-dev)

I changed the ppgplot repo URL to git+https://... (from pip install docs), but:

pip install .

still doesn't work, but this does:

python -m pip install -e "git+https://github.com/haavee/ppgplot.git#egg=ppgplot"

So don't see why the automatic (pip install .) doesn't work

Athanaseus · 2021-05-20T09:41:14Z

I removed the Py3 enforcement. I tested this using an 18.04 ubuntu docker image with python 2.7.17.

missing system PGPLOT library (apt-get install giza-dev) fixed that, but that required
missing libX11 (apt-get install libx11-dev)

The above dependencies will be resolved once we have a Debian package in KERN. For a pip or python installation, they will have to exist on the system.

Also, I noticed pip will install latest python-casacore which requires casacore version >= 3.1.1. (Also this will be resolved by the debian package). For now I had to add casacore from KERN-6.

haavee · 2021-05-20T14:49:16Z

OK, getting better, but not there yet; it does install & execute now.

But ...

The code supports dynamically loading a module+function for "postprocessing" (postprocess ../<module>.<function>).
All files (currently one) under postprocessing/... should be available after running jiveplot such that the following command should work:

 $> jplotter
 ...
 jcli> postprocess postprocessing/phatime.phaserate

In my local version I did:

 $> git mv postprocess jiveplot/

(to make it reside at the original relative level to the jplotter script) and edited jiveplot.egg-info/SOURCES to include jiveplot/postprocessing/phatime.py.

After (re)installing with that I see the phatime.py module being compiled, but the postprocess ... command still fails because the module itself (the raw Python code) is not installed but only the bytecompiled version I presume.

Athanaseus · 2021-05-20T15:50:42Z

After (re)installing with that I see the phatime.py module being compiled, but the postprocess ... command still fails because the module itself (the raw Python code) is not installed but only the bytecompiled version I presume.

Also, this is caused by the missing __init__.py file in the directory. I think it should be possible to run:

 $ jplotter
 jcli> postprocess phatime.phaserate

PS: Remeber to update pip pip install pip -U

haavee · 2021-05-27T14:44:25Z

(sorry for the lack of replies; been terribly busy with other things)
I did try several approaches but I cannot make the code load an (included) module from postprocessing/... yet.
I was going through this page https://docs.python.org/3/library/imp.html#imp.find_module but haven't made it work yet.

Athanaseus · 2021-06-01T11:24:05Z

Hey @haavee, so just to get this clear if this installation is done the old way this works? And can you please post the response after loading the module?

haavee · 2021-06-01T11:39:10Z

the problem is that in the "old way" there was never an installation :-(
you just run the Python script from the checked-out git repo and then it works.

The response if the command succeeds should be:

jcli> postprocess postprocessing/phatime.phaserate
postProcessing: postprocessing/phatime.phaserate

Athanaseus · 2021-06-02T06:39:03Z

ok i tried to figure out how the imp module works. Somehow this works if I do (from anywhere):

import imp
a, b, c = imp.find_module('jiveplot/postprocessing/phatime')
mod = imp.load_module('phatime', a, b, c)
mod.__dict__['phaserate']

This means in jplotter it works if:

jcli> postprocess postprocessing/phatime/phatime.phaserate
postProcessing: postprocessing/phatime/phatime.phaserate

I'm just bit confused by the redundancy of the phatime/phatime.

Found some instances that were not excercides during the Py2 -> Py2/3 compatibility yet

Finally figured out how to properly do it, so even from within pypi it is now possible to say: jcli> postprocess postprocessing/phatime.phaserate and expect it to work

haavee · 2021-08-18T14:42:48Z

Hi @Athanaseus, I think I've managed to figure out (and solve) the postprocessing loading of modules, now also seems to work in the pypi environment, and in the process fixed another few small bits that were broken/prevented the loaded module to actually load

Athanaseus · 2021-08-18T14:45:36Z

Hi @haavee, this is great news, thank you.
Looking at the updates, it looks good to me.

haavee · 2021-08-19T12:31:12Z

I'd like for people local to try this branch too before considering merging the PR.
Another issue is that there's a few dev branches in the original repo that haven't been merged into master, or, what the intended new master is, python23compat that this fork is based on.
Given that this PR basically relocates all files in the project, I wonder how to merge the existing development branch(es) easily. Do you know if git handles that?

Athanaseus · 2021-08-19T12:39:18Z

I'd like for people local to try this branch too before considering merging the PR.

That's a great idea.

Another issue is that there's a few dev branches in the original repo that haven't been merged into master, or, what the intended new master is, python23compat that this fork is based on.

I think we can start by resolving the conflicts with python23compat. This can be done by Resolve conflict, which will show us any clashing code on the two branches and you can remove any code that will not be required anymore.

Given that this PR basically relocates all files in the project, I wonder how to merge the existing development branch(es) easily. Do you know if git handles that?

Also if these other branches are ready to merge with the master they can be merged, and then update this PR with all new commits from master then also resolve any conflicts that might arise.

haavee · 2021-08-19T13:07:54Z

OK, the plan was that python23compat would become the new master. So how about starting to work towards that first, again with local testers to see if the new master is fine and then it makes sense to start working on making the changeover to pypi. How's that sound?

Athanaseus · 2021-08-19T13:17:10Z

Sure that sounds good to me.
I'm happy to help with the testing as well.

It bugged the **** out of me that symbol scaling didn't work (e.g. "ptsz 8" didn't do nothing for the data points). After carefully reading PGPLOT documentation and inspection of the plot commands: the code used symbol "-2" for plotting data points. According to docs https://sites.astro.caltech.edu/~tjp/pgplot/subroutines.html#PGPT this symbol scales with *linewidth* in stead of character height. After testing turned out that PGPLOT 5.2.2 (on our Linux 18.04 server) still doesn't honour this. Nor does Giza for that matter (tested). Solution: support setting symbol numbers for the different data categories: Unflagged, Flagged, Marked, Markedflagged. This allows user to set a scaling symbol (e.g. #17) for the data points, which does honour "ptsz" setting. Introduced "symbol" command for this. Found that markers were drawn not honouring "marksz" setting:code set different line width in stead of character height. (fixed)

Athanaseus added 5 commits May 18, 2021 12:42

Add setup file

dcc6764

Add requirements file

a76024c

Move files and add github actions

6f1b4f2

UPdate setup file

109dff0

Import Jiveplot moduls

a03afae

Athanaseus mentioned this pull request May 19, 2021

package jiveplot kernsuite/packaging#198

Open

Remove req file

0f96b47

Don't enforce py3 and add __init__ file

bc76dec

move the postprocessing module

16226ea

haavee added 2 commits August 18, 2021 16:23

some leftover Py2->2/3 compat issues

335f5e3

Found some instances that were not excercides during the Py2 -> Py2/3 compatibility yet

Fix postprocessing dynamic loading of module

83a6588

Finally figured out how to properly do it, so even from within pypi it is now possible to say: jcli> postprocess postprocessing/phatime.phaserate and expect it to work

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pypi release #17

Pypi release #17

Athanaseus commented May 19, 2021

haavee commented May 19, 2021 •

edited

haavee commented May 19, 2021 •

edited

Athanaseus commented May 19, 2021

haavee commented May 19, 2021

haavee commented May 19, 2021

Athanaseus commented May 20, 2021

haavee commented May 20, 2021

Athanaseus commented May 20, 2021

haavee commented May 27, 2021

Athanaseus commented Jun 1, 2021

haavee commented Jun 1, 2021

Athanaseus commented Jun 2, 2021

haavee commented Aug 18, 2021

Athanaseus commented Aug 18, 2021

haavee commented Aug 19, 2021

Athanaseus commented Aug 19, 2021

haavee commented Aug 19, 2021

Athanaseus commented Aug 19, 2021

Pypi release #17

Are you sure you want to change the base?

Pypi release #17

Conversation

Athanaseus commented May 19, 2021

haavee commented May 19, 2021 • edited

haavee commented May 19, 2021 • edited

Athanaseus commented May 19, 2021

haavee commented May 19, 2021

haavee commented May 19, 2021

Athanaseus commented May 20, 2021

haavee commented May 20, 2021

Athanaseus commented May 20, 2021

haavee commented May 27, 2021

Athanaseus commented Jun 1, 2021

haavee commented Jun 1, 2021

Athanaseus commented Jun 2, 2021

haavee commented Aug 18, 2021

Athanaseus commented Aug 18, 2021

haavee commented Aug 19, 2021

Athanaseus commented Aug 19, 2021

haavee commented Aug 19, 2021

Athanaseus commented Aug 19, 2021

haavee commented May 19, 2021 •

edited

haavee commented May 19, 2021 •

edited