Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NEURON v9 python syntax does not allow use of template instance name #2860

Open
nrnhines opened this issue Apr 28, 2024 · 12 comments · May be fixed by #2862
Open

NEURON v9 python syntax does not allow use of template instance name #2860

nrnhines opened this issue Apr 28, 2024 · 12 comments · May be fixed by #2862
Assignees

Comments

@nrnhines
Copy link
Member

Version 8.2 and before allows

$ python
>>> from neuron import h
>>> v = h.Vector()
>>> v
Vector[0]
>>> h.Vector[0]
Vector[0]

But with the current master

$ python
>>> from neuron import h
>>> v = h.Vector()
>>> v
Vector[0]
>>> h.Vector[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type 'hoc.Vector' is not subscriptable
@nrnhines nrnhines self-assigned this Apr 28, 2024
@nrnhines
Copy link
Member Author

@ramcdougal I'm stuck on this and could use your help. From what I see in nrnpython/nrnpy_hoc.h the slots are properly filled

   {Py_mp_subscript, (void*) hocobj_slice_getitem},
   {Py_sq_item, (void*) hocobj_getitem},

but the error message generated by h.Vector[0] is coming from Python's Objects/abstract.c in that

PyObject *
PyObject_GetItem(PyObject *o, PyObject *key)
{

and I don't understand why we aren't getting into either hocobj_getitem or hocobj_slice_getitem .

Note that if I add a debugging statement in nrnpython/nrnpy_hoc.cpp

--- a/src/nrnpython/nrnpy_hoc.cpp
+++ b/src/nrnpython/nrnpy_hoc.cpp
@@ -1011,6 +1011,9 @@ static PyObject* hocobj_getattr(PyObject* subself, PyObject* pyname) {
     auto location = sym_to_type_map.find(sym);
     if (location != sym_to_type_map.end()) {
         Py_INCREF(location->second);
+printf("RRR return  %p hocobj_getitem=%p\n", location->second, hocobj_getitem);
+        PyObject* pp = (PyObject*)location->second;
+PyObject_Print(pp, stdout, 0);
         return (PyObject*) location->second;
     }

and with gdb set a breakpoint at the return, then it looks ok with

p *(*(PyTypeObject*)pp).tp_as_sequence
sq_item = 0x7ffff796a1bb <hocobj_getitem(PyObject*, Py_ssize_t)>,
p *(*(PyTypeObject*)pp).tp_as_mapping
mp_subscript = 0x7ffff796acb7 <hocobj_slice_getitem(PyObject*, PyObject*)>,

I guess my next debug strategy will be to build a -g version of python and follow what happens on return from hocobj_getattr.
But if you have an idea about what is going on, let me know.

@ramcdougal
Copy link
Member

I wonder if this broke with the meaningful types commit... Prior to that, there was no hoc.Vector object.

@nrnhines
Copy link
Member Author

I believe it did. But it is not directly due to hoc.Vector per se. It happens with any hoc template, e,g hoc.Graph.
I do have a -g compiled version of python3.12.3 now and I see that the first puzzle is in Objects/abstract.c PyObject_GetItem(PyObject *o, PyObject *key) where the o arg is

(gdb) p *o
$4 = {{ob_refcnt = 44, ob_refcnt_split = {44, 0}}, 
  ob_type = 0x7ffff61735e0 <PyType_Type>}

and

(gdb) p *(PyTypeObject*)o
$5 = {ob_base = {ob_base = {{ob_refcnt = 44, ob_refcnt_split = {44, 0}}, 
...
  tp_name = 0x7ffff29782f0 "hoc.Vector", tp_basicsize = 96, tp_itemsize = 0, 
...
  tp_as_number = 0x555555704a70, tp_as_sequence = 0x555555704ba8,
  tp_as_mapping = 0x555555704b90,
...

but both lines in the code

   PyMappingMethods *m = Py_TYPE(o)->tp_as_mapping; 
and
   PySequenceMethods *ms = Py_TYPE(o)->tp_as_sequence;

ended up as

(gdb) p m
$8 = (PyMappingMethods *) 0x0
(gdb) p ms
$9 = (PySequenceMethods *) 0x0

So I guess I need to look at the Py_TYPE(o) which (because it is a macro?) cannot be run from gdb

(gdb) p Py_TYPE(o)
Couldn't write extended state status: Bad address.

@nrnhines
Copy link
Member Author

It's just

static inline PyTypeObject* Py_TYPE(PyObject *ob) {
    return ob->ob_type;
}

@ramcdougal
Copy link
Member

All these types are created dynamically, I think... So maybe they just need to get a copy of the hoc.HocObject pointers

@nrnhines
Copy link
Member Author

nrnhines commented Apr 29, 2024

I think you are right. From the above casting with *(PyTypeObject*)o has non-null tp_as_sequence and tp_as_mapping.
But from

(gdb) p *o->ob_type
$12 = {ob_base = {ob_base = {{ob_refcnt = 4294967295, ob_refcnt_split = {
          4294967295, 0}}, ob_type = 0x7ffff61735e0 <PyType_Type>}, 
    ob_size = 0}, tp_name = 0x7ffff5fa48f1 "type", tp_basicsize = 920, 
  tp_itemsize = 40, tp_dealloc = 0x7ffff5de8800 <type_dealloc>, 
  tp_vectorcall_offset = 400, tp_getattr = 0x0, tp_setattr = 0x0, 
  tp_as_async = 0x0, tp_repr = 0x7ffff5dea9d0 <type_repr>, 
  tp_as_number = 0x7ffff6173780 <type_as_number>, tp_as_sequence = 0x0, 
  tp_as_mapping = 0x0, tp_hash = 0x7ffff5edaf20 <_Py_HashPointer>, 
  tp_call = 0x7ffff5deaf20 <type_call>, tp_str = 0x7ffff5de4480 <object_str>, 
  tp_getattro = 0x7ffff5dfc9f0 <_Py_type_getattro>, 
  tp_setattro = 0x7ffff5df7910 <type_setattro>, tp_as_buffer = 0x0, 
  tp_flags = 2156420354, 
  tp_doc = 0x7ffff5fa94c0 <type_doc> "type(object) -> the object's type\ntype(name, bases, dict, **kwds) -> a new type", 
  tp_traverse = 0x7ffff5de4e80 <type_traverse>, 
  tp_clear = 0x7ffff5dea5c0 <type_clear>, 
  tp_richcompare = 0x7ffff5de5120 <object_richcompare>, 
  tp_weaklistoffset = 368, tp_iter = 0x0, tp_iternext = 0x0, 
  tp_methods = 0x7ffff6173b20 <type_methods>, 
  tp_members = 0x7ffff6173f40 <type_members>, 
  tp_getset = 0x7ffff6173d60 <type_getsets>, 
  tp_base = 0x7ffff6173440 <PyBaseObject_Type>, tp_dict = 0x0, 
  tp_descr_get = 0x0, tp_descr_set = 0x0, tp_dictoffset = 264, 
--Type <RET> for more, q to quit, c to continue without paging--

those are null.
Not clear to me what object it gets added into. I guess I need to determine if o->ob_type is unique to (PyTypeObject*)o

@nrnhines
Copy link
Member Author

@ramcdougal Do you think it makes sense to do a self reference, i.e. pto->ob_type = pto for all the

pto = (PyTypeObject*) PyType_FromSpecWithBases(&spec, bases);

I guess, I'll check what pto->ob_type starts out as for all the items in

        sym_to_type_map[hoc_lookup(name)] = pto;

@nrnhines
Copy link
Member Author

nrnhines commented Apr 29, 2024

The following results puzzle me since the pto->ob_type seems generic and should not be changed?
I tested with

--- a/src/nrnpython/nrnpy_hoc.cpp
+++ b/src/nrnpython/nrnpy_hoc.cpp
@@ -3319,6 +3319,12 @@ PyObject* nrnpy_hoc() {
         exposed_py_type_names.push_back(std::string("hoc.") + name);
         spec = obj_spec_from_name(exposed_py_type_names.back().c_str());
         pto = (PyTypeObject*) PyType_FromSpecWithBases(&spec, bases);
+auto z = (PyObject*)pto;
+printf("%s %p %p\n  ", name, z, z->ob_type);
+PyObject_Print(z, stdout, 0);
+printf("  ");
+PyObject_Print((PyObject*)z->ob_type, stdout, 0);
+printf("\n");
         sym_to_type_map[hoc_lookup(name)] = pto;

and the first few lines of output are

SectionList 0x60f7b4778670 0x77aa34b735e0
  <class 'hoc.SectionList'>  <class 'type'>
SectionRef 0x60f7b4779cb0 0x77aa34b735e0
  <class 'hoc.SectionRef'>  <class 'type'>
Graph 0x60f7b477a3e0 0x77aa34b735e0
  <class 'hoc.Graph'>  <class 'type'>
HBox 0x60f7b4780330 0x77aa34b735e0
  <class 'hoc.HBox'>  <class 'type'>

@nrnhines
Copy link
Member Author

I'm starting to think that the ob_type of h.Vector should be <class 'hoc.HocObject'>

@nrnhines
Copy link
Member Author

nrnhines commented Apr 30, 2024

Types of user define hoc templates are working as in the past. i.e.

from neuron import h

h(
    """
begintemplate A
proc init() {
  x = 2
}
endtemplate A
"""
)
a = h.A()
print(a)
print(type(a))
print(type(type(a)))
print(h.A[0])

v = h.Vector(3)
print(v)
print(type(v))
print(type(type(v)))
print(h.Vector[0])

prints the output

A[0]
<class 'hoc.HocObject'>
<class 'type'>
A[0]
Vector[0]
<class 'hoc.Vector'>
<class 'type'>
Traceback (most recent call last):
  File "/home/hines/neuron/vecinst/src/nrnpython/temp.py", line 22, in <module>
    print(h.Vector[0])
          ~~~~~~~~^^^
TypeError: type 'hoc.Vector' is not subscriptable

@nrnhines
Copy link
Member Author

nrnhines commented May 5, 2024

After a very long conversation with chatgpt 3.5, we came up with a pure python example of what I wanted, I.e.

% cat temp2.py
class CustomMeta(type):
    def __new__(cls, name, bases, dct):
        return super().__new__(cls, name, bases, dct)

    def __getitem__(cls, index):
        print("inside CustomMeta.__getitem__")
        return f"{cls}[{index}]"

class Foo(metaclass=CustomMeta):
    def __init__(self):
        print("Foo.__init__ ", self)

    def __getitem__(self, index):
        print("inside Foo.__getitem__")
        return f"{self}[{index}]"

print("type(Foo) is", type(Foo))
print("Foo[3] is ", Foo[3])
print("Foo()[5] is ", Foo()[5])

% python3 temp2.py
type(Foo) is <class '__main__.CustomMeta'>
inside CustomMeta.__getitem__
Foo[3] is  <class '__main__.Foo'>[3]
Foo.__init__  <__main__.Foo object at 0x1001bfe60>
inside Foo.__getitem__
Foo()[5] is  <__main__.Foo object at 0x1001bfe60>[5]

@nrnhines
Copy link
Member Author

nrnhines commented May 5, 2024

Here's the remainder of the conversation where I asked for a c-api version in the context of the nrnpy_hoc.cpp code. It will likely require another long conversation for me to get it working in our context.

Now, how would you get that behavior with my original fragment where I
started with
 for (auto name: py_exposed_classes) {

ChatGPT
To achieve similar behavior in your original C++ fragment, where you're creating classes using PyType_FromSpecWithBases, you can define a custom metaclass and include a __getitem__ method in that metaclass.

Here's how you can modify your original C++ code to achieve the desired behavior:

// Define the custom metaclass
static PyObject *CustomMeta_getitem(PyObject *cls, PyObject *index) {
    const char *cls_name = PyUnicode_AsUTF8(PyUnicode_FromObject(cls));
    const char *index_str = PyUnicode_AsUTF8(PyUnicode_FromObject(index));
    printf("%s[%s]\n", cls_name, index_str);
    Py_RETURN_NONE;
}

static PyMethodDef CustomMeta_methods[] = {
    {"__getitem__", CustomMeta_getitem, METH_O, "Get item"},
    {NULL, NULL, 0, NULL} /* Sentinel */
};

static PyTypeObject CustomMeta_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom_module.CustomMeta",
    .tp_basicsize = sizeof(PyTypeObject),
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_methods = CustomMeta_methods
};

// Class creation loop
for (auto name: py_exposed_classes) {
    // Create the type object for the class
    PyTypeObject* pto = (PyTypeObject*) PyType_FromSpecWithBases(&spec, bases);
    if (!pto) {
        // Handle error
    }

    // Set the metaclass to CustomMeta
    Py_INCREF(&CustomMeta_Type);
    pto->tp_base = &CustomMeta_Type;

    // Ensure type finalization
    if (PyType_Ready(pto) < 0) {
        // Handle error
    }

    // Add the class type to the module
    PyModule_AddObject(m, name, (PyObject*)pto);
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants