Fast C implementation of units #231

ejeffrey · 2016-03-23T00:10:32Z

This implements a fast (50X to 100X faster than before) version of the units code in C. It is structured somewhat differently, but is almost entirely compatible with the old implementation. It currently passes its own tests, and most pylabrad tests with the following caveats.

The special behavior of '0.0' which could compare to any value is not included. I liked this, but other people didn't, and it would require extra effort to include
DimensionlessFloat, DimensionlessComplex, and DimensionlessArray types do not subclass float, complex, and ndarray. This means isinstance(4*ns * 1 * GHz, float) is false. However, these types have float, complex and array methods which should allow them to be used in almost any situation that expects a raw float/complex/ndarray
New unit types are no longer automatically created on demand. If you try Value(3, 'secomd'), it will fail. You must explicitly create new unit types using e.g., addNonSI().
There are still some non-standard units and constants missing.

This code is in a new top level package called fastunits. pylabrad.units now tries to import that, but falls back on the python implementation (renamed units_compat) if it can't import fastunits.

@maffoo @zchen088

* Make pickling work * implement isangle property * addNonSI function * add more units / constants * construct arrays from lists: ValueArray([1,2,3], 'ns') * Add array properties ndim, shape, and dtype

rename value to _value

zchen088 · 2016-03-29T18:42:14Z

labrad/units.py

+    DimensionlessArray = ValueArray
+    DimensionlessFloat = Value
+    DimensionlessComplex = Complex
+    for k, v in constants.items():


Getting NameError: name 'constants' is not defined here.

zchen088 · 2016-03-30T00:53:08Z

Getting this error when trying to run setup.py:

WindowsError                              Traceback (most recent call last)
C:\Users\Jimmy\Desktop\pylabrad\setup.py in <module>()
     52     scripts=[],
     53     ext_modules=[Extension("fastunits.unitarray",
---> 54                              sources = ["fastunits/unitarray.c"])]
     55 )

C:\Python27\lib\distutils\core.pyc in setup(**attrs)
    109     # (ie. everything except distclass) to initialize it
    110     try:
--> 111         _setup_distribution = dist = klass(attrs)
    112     except DistutilsSetupError, msg:
    113         if 'name' in attrs:

C:\Python27\lib\site-packages\setuptools\dist.pyc in __init__(self, attrs)
    270         for ep in pkg_resources.iter_entry_points('distutils.setup_keywo
rds'):
    271             vars(self).setdefault(ep.name, None)
--> 272         _Distribution.__init__(self,attrs)
    273         if isinstance(self.metadata.version, numbers.Number):
    274             # Some people apparently take "version number" too literally
 :)

C:\Python27\lib\distutils\dist.pyc in __init__(self, attrs)
    285                     break
    286
--> 287         self.finalize_options()
    288
    289     def get_option_dict(self, command):

C:\Python27\lib\site-packages\setuptools\dist.pyc in finalize_options(self)
    325             if value is not None:
    326                 ep.require(installer=self.fetch_build_egg)
--> 327                 ep.load()(self, ep.name, value)
    328         if getattr(self, 'convert_2to3_doctests', None):
    329             # XXX may convert to set here when we can rely on set being
builtin

C:\Python27\lib\site-packages\setuptools_scm\integration.pyc in version_keyword(
dist, keyword, value)
     17     if getattr(value, '__call__', None):
     18         value = value()
---> 19     dist.metadata.version = get_version(**value)
     20
     21

C:\Python27\lib\site-packages\setuptools_scm\__init__.pyc in get_version(root, v
ersion_scheme, local_scheme, write_to, write_to_template, relative_to, parse)
    100     trace('root', repr(root))
    101
--> 102     version = _do_parse(root, parse)
    103
    104     if version:

C:\Python27\lib\site-packages\setuptools_scm\__init__.pyc in _do_parse(root, par
se)
     67     else:
     68         # include fallbacks after dropping them from the main entrypoint

---> 69         version = version_from_scm(root)
     70
     71     if version:

C:\Python27\lib\site-packages\setuptools_scm\__init__.pyc in version_from_scm(ro
ot)
     28
     29 def version_from_scm(root):
---> 30     return _version_from_entrypoint(root, 'setuptools_scm.parse_scm')
     31
     32

C:\Python27\lib\site-packages\setuptools_scm\__init__.pyc in _version_from_entry
point(root, entrypoint)
     34     ep = find_matching_entrypoint(root, entrypoint)
     35     if ep:
---> 36         return ep.load()(root)
     37
     38

C:\Python27\lib\site-packages\setuptools_scm\git.pyc in parse(root, describe_com
mand)
      9
     10 def parse(root, describe_command=DEFAULT_DESCRIBE):
---> 11     real_root, _, ret = do_ex('git rev-parse --show-toplevel', root)
     12     if ret:
     13         return

C:\Python27\lib\site-packages\setuptools_scm\utils.pyc in do_ex(cmd, cwd)
     56             LC_ALL='C',
     57             LANGUAGE='',
---> 58             HGPLAIN='1',
     59         ))
     60     )

C:\Python27\lib\subprocess.pyc in __init__(self, args, bufsize, executable, stdi
n, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, s
tartupinfo, creationflags)
    708                                 p2cread, p2cwrite,
    709                                 c2pread, c2pwrite,
--> 710                                 errread, errwrite)
    711         except Exception:
    712             # Preserve original exception in case os.close raises.

C:\Python27\lib\subprocess.pyc in _execute_child(self, args, executable, preexec
_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell,
 to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite)
    956                                          env,
    957                                          cwd,
--> 958                                          startupinfo)
    959             except pywintypes.error, e:
    960                 # Translate pywintypes.error to WindowsError, which is

WindowsError: [Error 2] The system cannot find the file specified

zchen088 · 2016-03-31T04:40:42Z

Ok got it to compile with help from @maffoo . Two things I've found so far.

The string representation for values seems to be:
fastunits.Value(1.0, "GHz")
Don't remember seeing that in your older versions, is that intentional?
Values are missing an isCompatible method

ejeffrey · 2016-03-31T04:44:48Z

Hmm... I have to check on the string representation. I had to change the
class name in order to get pickle to work properly -- without that it
doesn't know how to find it when unpickling. I will see if there is a way
to change the string representation.

isCompatible should be a pretty easy fix.

On Wed, Mar 30, 2016 at 9:40 PM, zchen088 notifications@github.com wrote:

Ok got it to compile with help from @maffoo https://github.com/maffoo .
Two things I've found so far.

The string representation for values seems to be:
fastunits.Value(1.0, "GHz")
Don't remember seeing that in your older versions, is that intentional?
2.

Values are missing an isCompatible method

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#231 (comment)

* Fix unit __repr__ to print only the class name without the module

ejeffrey · 2016-03-31T05:16:54Z

Both the string repr and the isCompatible method should be fixed

On Wed, Mar 30, 2016 at 9:44 PM, Evan Jeffrey ejeffrey@gmail.com wrote:

Hmm... I have to check on the string representation. I had to change the
class name in order to get pickle to work properly -- without that it
doesn't know how to find it when unpickling. I will see if there is a way
to change the string representation.

isCompatible should be a pretty easy fix.

On Wed, Mar 30, 2016 at 9:40 PM, zchen088 notifications@github.com
wrote:

Ok got it to compile with help from @maffoo https://github.com/maffoo
. Two things I've found so far.

The string representation for values seems to be:
fastunits.Value(1.0, "GHz")
Don't remember seeing that in your older versions, is that
intentional?
2.

Values are missing an isCompatible method

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#231 (comment)

zchen088 · 2016-03-31T06:11:35Z

Can't iterate over a ValueArray:

In [15]: x
Out[15]: ValueArray([ 1.  2.  3.  4.  5.], "GHz")

In [16]: iter(x)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-16-128770259dbe> in <module>()
----> 1 iter(x)

TypeError: 'fastunits.ValueArray' object is not iterable
> <ipython-input-16-128770259dbe>(1)<module>()
----> 1 iter(x)

ejeffrey · 2016-03-31T15:30:32Z

Thanks. I added tests for these. Will try to figure out why iteration
doesn't work.

On Wed, Mar 30, 2016 at 11:11 PM, zchen088 notifications@github.com wrote:

Can't iterate over a ValueArray:

In [15]: x
Out[15]: ValueArray([ 1. 2. 3. 4. 5.], "GHz")

In [16]: iter(x)---------------------------------------------------------------------------TypeError Traceback (most recent call last) in ()----> 1 iter(x)
TypeError: 'fastunits.ValueArray' object is not iterable> (1)()----> 1 iter(x)

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#231 (comment)

* Allow iteration over unit arrays by implementing sequence len and index * Fix test string comparison for 1/x vs x^-1 * Fix cases where we could have a non-canonical fractional power (negative denominator)

ejeffrey · 2016-03-31T18:41:47Z

OK, sequence methods added. I also fixed a canonicalization problem with
fractional units where you could have Hz^(1/-2).

On Thu, Mar 31, 2016 at 8:30 AM, Evan Jeffrey ejeffrey@gmail.com wrote:

Thanks. I added tests for these. Will try to figure out why iteration
doesn't work.

On Wed, Mar 30, 2016 at 11:11 PM, zchen088 notifications@github.com
wrote:

Can't iterate over a ValueArray:

In [15]: x
Out[15]: ValueArray([ 1. 2. 3. 4. 5.], "GHz")

In [16]: iter(x)---------------------------------------------------------------------------TypeError Traceback (most recent call last) in ()----> 1 iter(x)
TypeError: 'fastunits.ValueArray' object is not iterable> (1)()----> 1 iter(x)

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#231 (comment)

zchen088 · 2016-04-01T01:59:34Z

ValueArrays lack a "size" attribute.

In general it seems like ValueArrays are no longer inheriting attributes from numpy arrays like they were in previous builds.

ejeffrey · 2016-04-01T02:49:36Z

regular valuearrays don't have a size field either (just tested). Neither
one of them implements the full ndarray interface.

One difference that might be causing your problem is that python
ValueArrays automatically convert to a numpy.ndarray when they are
dimensionless. C ValueArrays do not, they just implement a array
method so that when you call a function that expects an array it can
automatically convert.

I guess I will have to either implement that or all of the ndarray methods.

On Thu, Mar 31, 2016 at 6:59 PM, zchen088 notifications@github.com wrote:

ValueArrays lack a "size" attribute.

In general it seems like ValueArrays are no longer inheriting attributes
from numpy arrays like they were in previous builds.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#231 (comment)

ejeffrey · 2016-04-01T07:11:32Z

OK. I just added almost the entire numpy api to ValueArray. There are a few methods that aren't exactly cookie-cutter and are not implemented, namely: clip, fill, flat, var, and dot.

Also, anything that has to do with data layout or subtypes is not implemented.

What is now implemented: transpose, cumprod, cumsum, compress, diagonal, flatten, max, min, mean, std, prod, sum, trace, conj, conjugate, choose, reshape, resize, real, imag, T, all, any, argmin, argmax, argpartition, argsort, nonzero

zchen088 · 2016-04-01T17:42:52Z

Thanks @ejeffrey, can you push?

ejeffrey · 2016-04-01T18:08:40Z

Also added dot product method

zchen088 · 2016-04-12T20:56:20Z

I'm an APPCRASH when I try to import labrad, seems to have been introduced by 0584a21.

ejeffrey · 2016-04-12T21:09:05Z

I think the problem was the ValueArray_methods array wasn't zero terminated. Probably it happens to be followed by zeros on my computer and not yours.

ejeffrey · 2016-04-12T21:11:29Z

And, I went a little overboard adding zeros that didn't belong there, fixed now.

ejeffrey · 2016-04-12T21:13:05Z

Oh, and thanks for bisecting that -- knowing the exact change and that it
crashes on import made it super easy to find (assuming this fixes the
problem)

On Tue, Apr 12, 2016 at 1:56 PM, zchen088 notifications@github.com wrote:

I'm an APPCRASH when I try to import labrad, seems to have been introduced
by 0584a21
0584a21
.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#231 (comment)

zchen088 · 2016-04-12T21:14:31Z

Yes, works now. Fixing some zero comparisons in pyle so we don't have to worry about those.

zchen088 · 2016-04-12T21:18:32Z

Getting the following flattening error:

In [9]: sq.rabihigh(s, 'q1')
---------------------------------------------------------------------------
FlatteningError                           Traceback (most recent call last)
<ipython-input-9-3d21a39cd9bd> in <module>()
----> 1 sq.rabihigh(s, 'q1')

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\singleQubit.pyc in rabihigh(Sample, me
asure, name, amps, alpha, state, numPulses, stats, IQ, mode, correct, save, coll
ect, noisy, update, manual)
   1596             returnValue(np.squeeze(prob))
   1597
-> 1598     data = sweeps.grid(func, axes, dataset=dataset, save=save, noisy=Fal
se)
   1599
   1600     if update and manual:

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\sweeps.pyc in grid(func, axes, dataset
, **kw)
    321         dataset=dataset,
    322         abortPrefix=[1],
--> 323         **kw
    324     )
    325

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\sweeps.pyc in _run(func, sweep, save,
dataset, abortable, abortPrefix, collect, noisy, pipesize)
    220         params = dataset.params
    221         name = dataset.name
--> 222         data = np.vstack(it)
    223         kwargs = {'number': dataset.num, 'path': dataset.path} if save e
lse {}
    224         return pyle.datavault.LocalDataSet(data, independents, dependent
s,

C:\Python27\lib\site-packages\numpy\core\shape_base.pyc in vstack(tup)
    228
    229     """
--> 230     return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
    231
    232 def hstack(tup):

C:\Users\Jimmy\Desktop\pyle\pyle\datasaver.pyc in capture(self, iterable)
    146         """
    147         with self as dataset:
--> 148             for data in iterable:
    149                 dataset.add(data)
    150                 yield data

C:\Users\Jimmy\Desktop\pyle\pyle\pipeline.pyc in gen()
     94                 runner = _FutureRunner(result)
     95             elif is_generator(result):
---> 96                 runner = _GeneratorRunner(result)
     97             else:
     98                 runner = _ValueRunner(result)

C:\Users\Jimmy\Desktop\pyle\pyle\pipeline.pyc in __init__(self, gen)
    118         self.gen = _run(gen)
    119         self.done = False
--> 120         self.advance()
    121
    122     def advance(self):

C:\Users\Jimmy\Desktop\pyle\pyle\pipeline.pyc in advance(self)
    123         if not self.done:
    124             try:
--> 125                 self.gen.next()
    126             except _DefGen_Return, e:
    127                 self.done = True

C:\Users\Jimmy\Desktop\pyle\pyle\pipeline.pyc in _run(gen)
    173             elif is_generator(result):
    174                 try:
--> 175                     for _ in _run(result):
    176                         yield
    177                     result = None

C:\Users\Jimmy\Desktop\pyle\pyle\pipeline.pyc in _run(gen)
    179                     result = e.value
    180             else:
--> 181                 result = gen.send(result)
    182         except _DefGen_Return, e:
    183             returnValue(e.value)

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\singleQubit.pyc in func(amp)
   1588
   1589         if IQ:
-> 1590             data = yield Sample.run(alg, stats, dataFormat='iqRaw')
   1591             I,Q = readout.averageIQSingleDemod(data)
   1592             returnValue([I,Q])

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\qubitsequencer.pyc in __call__(self, a
lg, stats, **kw)
     66     def __call__(self, alg, stats, **kw):
     67         return fpgaseqTransmon.runQubits(self.server, alg, stats,
---> 68                                          jump_table=self.jump_table, **k
w)
     69
     70

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\fpgaseqTransmon.pyc in runQubits(serve
r, alg, stats, dataFormat, debug, jump_table)
     56
     57     p = runQubitsGetPacket(server, alg, stats, jump_table=jump_table)
---> 58     return sendPacket(p, debug, dataFormat)
     59
     60

C:\Users\Jimmy\Desktop\pyle\pyle\dataking\fpgaseqTransmon.pyc in sendPacket(p, d
ebug, dataFormat)
    306
    307     # send the request
--> 308     req = p.send(wait=False)
    309
    310     if debug:

C:\Users\Jimmy\Desktop\pyle\pyle\util\labradtools.pyc in send(self, wait, **kwar
gs)
     95
     96     def send(self, wait=False, **kwargs):
---> 97         req = self._packet.send(wait=wait, **kwargs)
     98         if not wait:
     99             self._manager._start_request(req)

C:\Python27\lib\site-packages\pylabrad-0.95.1.dev11+ng25c79c2-py2.7-win32.egg\la
brad\client.pyc in send(self, wait, **kw)
    326                           "Use packet.send_future(...) instead.")
    327         f = self.send_future(**kw)
--> 328         f.result()
    329         return f.result() if wait else f
    330

C:\Python27\lib\site-packages\pylabrad-0.95.1.dev11+ng25c79c2-py2.7-win32.egg\la
brad\concurrent.pyc in result(self, timeout)
     98     def result(self, timeout=None):
     99         if not self._done:
--> 100             self._result = super(MutableFuture, self).result(timeout)
    101             self._done = True
    102         if self._error is not None:

C:\Python27\lib\site-packages\concurrent\futures\_base.pyc in result(self, timeo
ut)
    403                 raise CancelledError()
    404             elif self._state == FINISHED:
--> 405                 return self.__get_result()
    406             else:
    407                 raise TimeoutError()

C:\Python27\lib\site-packages\concurrent\futures\_base.pyc in __get_result(self)

    355     def __get_result(self):
    356         if self._exception:
--> 357             raise type(self._exception), self._exception, self._tracebac
k
    358         else:
    359             return self._result

FlatteningError: (12345) Could not flatten FlatData(bytes='\x00\x00\x00\x02q1\x0
0\x00\x00\x05uwave@\x14\xb8Q\xeb\x85\x1e\xb8@$\x00\x00\x00\x00\x00\x00', tag=LRC
luster(LRCluster(LRStr(), LRStr()), LRValue('GHz'), LRValue('dBm'))) to ((ss)v[G
Hz]v[dBm]).
Setting ID 210. [payload=None]

zchen088 · 2016-04-12T21:43:25Z

Previously, diving two values with the same units would return a float, and some parts of pyle are expecting this behavior (e.g. to cast to int). Is this something you can implement?

ejeffrey · 2016-04-12T21:49:47Z

Hmm,, OK. I didn't realize that wouldn't work.

What I did was implement a __float__ and __complex__ method, so that:

float(5_ns/(1_ns)) == 5.0 does what you expect. I sort of assumed int()
would also work. It is easy to implement __int__ the same way, not sure
if this will fix everything.

FWIW, the more "approved" way to do this now is:

ratio = 5_ns // (1_ns) --> gives 5.0

or even better:

q, r = divmod(5_ns, 2_ns)

which returns (2.0, 1*ns)

On Tue, Apr 12, 2016 at 2:43 PM, zchen088 notifications@github.com wrote:

Previously, diving two values with the same units would return a float,
and some parts of pyle are expecting this behavior (e.g. to cast to int).
Is this something you can implement?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#231 (comment)

zchen088 · 2016-04-12T23:35:34Z

np.alen on a single Value does not work, but it's kind of misused in pyle anyways.

ejeffrey · 2016-04-13T01:00:39Z

It's not really possible to fix this without breaking something else. np.alen does the following:

def alen(x):
    try:
        return len(x)
    except:
        return len(array(x))

In the python units, array(x) makes an array of dtype object, which is bad, and we have always tried to kill anything that makes those. Now, array(Value/ValueArray) only works if the value is dimensionless.

The other way to make this work is to implement len on Value, but that will make Value iterable, which I would rather not have.

There is one final option, which is for array(Value) to make a ValueArray. This is could be a resonable idea, except that I think that will cause a lot of numpy functions to fail in "intersting" ways because they call array() and expect the result to be an ndarray. I think it is better if these fail with a unit mismatch error.

Given that most of the np.xxx functions don't work on ValueArrays, I don't think this is worth making something else bad.

ejeffrey added 3 commits March 22, 2016 11:01

Initial commit of fastunits code

bf532c8

A bunch of updates to fix existing unit tests:

13ab09b

* Make pickling work * implement isangle property * addNonSI function * add more units / constants * construct arrays from lists: ValueArray([1,2,3], 'ns') * Add array properties ndim, shape, and dtype

Implement allclose

d87044c

rename value to _value

zchen088 reviewed Mar 29, 2016
View reviewed changes

Add numpy includes to setup.py

91284e5

* Add isCompatible method

201aea8

* Fix unit __repr__ to print only the class name without the module

Add test for iteration and isCompatible

57e46b7

Several more fixes:

f17c00d

* Allow iteration over unit arrays by implementing sequence len and index * Fix test string comparison for 1/x vs x^-1 * Fix cases where we could have a non-canonical fractional power (negative denominator)

Add a bunch of numpy methods to unitarray

0584a21

Add dot product

87b4327

ejeffrey mentioned this pull request Apr 11, 2016

Faster units #213

Open

zero terminate arrays

5006a52

Removed excess zero initializer

25c79c2

isCompatible returns value

040bda9

Evan Jeffrey added 3 commits April 12, 2016 15:41

Add conversion for int and long on dimensionless values

88ce8cc

Fix tests

7d688cb

Change working directory in test to avoid import problem

35f7a3a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fast C implementation of units #231

Fast C implementation of units #231

ejeffrey commented Mar 23, 2016

zchen088 Mar 29, 2016

zchen088 commented Mar 30, 2016

zchen088 commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

zchen088 commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

zchen088 commented Apr 1, 2016

ejeffrey commented Apr 1, 2016

ejeffrey commented Apr 1, 2016

zchen088 commented Apr 1, 2016

ejeffrey commented Apr 1, 2016

zchen088 commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

zchen088 commented Apr 12, 2016

zchen088 commented Apr 12, 2016

zchen088 commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

zchen088 commented Apr 12, 2016

ejeffrey commented Apr 13, 2016

Fast C implementation of units #231

Are you sure you want to change the base?

Fast C implementation of units #231

Conversation

ejeffrey commented Mar 23, 2016

zchen088 Mar 29, 2016

Choose a reason for hiding this comment

zchen088 commented Mar 30, 2016

zchen088 commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

zchen088 commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

ejeffrey commented Mar 31, 2016

zchen088 commented Apr 1, 2016

ejeffrey commented Apr 1, 2016

ejeffrey commented Apr 1, 2016

zchen088 commented Apr 1, 2016

ejeffrey commented Apr 1, 2016

zchen088 commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

zchen088 commented Apr 12, 2016

zchen088 commented Apr 12, 2016

zchen088 commented Apr 12, 2016

ejeffrey commented Apr 12, 2016

zchen088 commented Apr 12, 2016

ejeffrey commented Apr 13, 2016