[cython-users] Accessing Python long type?

[Jeroen Demeyer]

Is there any way in Cython to access the Python "long" type?

Essentially, what I'm trying to do is


cdef extern from "longintrepr.h":
cdef long PyLong_SHIFT
ctypedef unsigned int digit
ctypedef class __builtin__.long [object PyLongObject]:
cdef digit* digits

cdef extern from "Python.h":
cdef _PyObject_NewVar(type t, Py_ssize_t size)

def foo(s):
cdef long L = _PyObject_NewVar(long, s)


The problem is that Cython always thinks that "long" means C long, while
I really want Python long. Is there any solution for this?

[Stefan Behnel]

Post by Jeroen Demeyer
Is there any way in Cython to access the Python "long" type?
Essentially, what I'm trying to do is
cdef long PyLong_SHIFT
ctypedef unsigned int digit
cdef digit* digits
cdef _PyObject_NewVar(type t, Py_ssize_t size)
cdef long L = _PyObject_NewVar(long, s)
The problem is that Cython always thinks that "long" means C long, while I
really want Python long. Is there any solution for this?

Well - what do you need it for?

Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

Stefan

[Sturla Molden]

Post by Stefan Behnel
Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

It seems the question is how to cdef a PyLong, since "cdef long" will
cdef a C long. It is similar to how to cdef a PyInt or PyFloat. I must
say I have never thought about it. But if Cython can optimize operations
on these elementary Python types there must be a way to do this.

Sturla

[Stefan Behnel]

Post by Stefan Behnel
Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

It seems the question is how to cdef a PyLong, since "cdef long" will cdef
a C long. It is similar to how to cdef a PyInt or PyFloat. I must say I
have never thought about it. But if Cython can optimize operations on these
elementary Python types there must be a way to do this.

It applies this special casing internally. No need to declare them.

Stefan

[Robert Bradshaw]

Post by Stefan Behnel
Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

It seems the question is how to cdef a PyLong, since "cdef long" will cdef a
C long. It is similar to how to cdef a PyInt or PyFloat. I must say I have
never thought about it. But if Cython can optimize operations on these
elementary Python types there must be a way to do this.

It's a bit of a pain, because the parser has special code to deal with
these non-whitespace-delimited types (e.g. long long int). To get
around this you can make the declarations in a pxd file and use fully
qualified cimported names, e.g. put

cdef extern from "longintrepr.h":
cdef long PyLong_SHIFT
ctypedef unsigned int digit
ctypedef class __builtin__.long [object PyLongObject]:
cdef digit* digits

in pylong.pxd and then do

cimport pylong
cdef pylong.long foo = ...

Unfortunately, it looks like there's still other issues with dealing
with variable sized types...

[Jeroen Demeyer]

Post by Stefan Behnel
Well - what do you need it for?

I need to manually create such objects using _PyLong_New() and then
manually fill in the ob_digit field. Currently, I use
(<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
elegant as it could be.

Post by Stefan Behnel
Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

I'm using the constants defined in the Python headers (like
sizeof(digit) and PyLong_SHIFT). The Cython-generated code is compiled
using the Python header, so I don't see the problem here.

[Stefan Behnel]

Post by Jeroen Demeyer

Post by Stefan Behnel
Well - what do you need it for?

I need to manually create such objects using _PyLong_New() and then
manually fill in the ob_digit field. Currently, I use
(<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
elegant as it could be.

And would you mind sharing with us why you need to create the objects in
this unusual way?

Post by Jeroen Demeyer

Post by Stefan Behnel
Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

I'm using the constants defined in the Python headers (like sizeof(digit)
and PyLong_SHIFT). The Cython-generated code is compiled using the Python
header, so I don't see the problem here.

The problem is that the size of "digit" can either be specified by
configure, or determined at C compile time by the header files. I've seen
cases where the latter yielded a different size for me than what CPython
was really using internally, thus leading to crashes and/or data corruption.

Basically, if PYLONG_BITS_IN_DIGIT is defined by pyconfig.h, then you're on
the safe side. If it's not (and that's the normal case), then there's a
relatively high chance of being lucky, and a certain risk of being unlucky.

Stefan

[Jeroen Demeyer]

Post by Stefan Behnel

Post by Jeroen Demeyer

Post by Stefan Behnel
Well - what do you need it for?

I need to manually create such objects using _PyLong_New() and then
manually fill in the ob_digit field. Currently, I use
(<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
elegant as it could be.

And would you mind sharing with us why you need to create the objects in
this unusual way?

To implement a conversion between GMP mpz_t and Python longs:
http://trac.sagemath.org/ticket/17853

Post by Stefan Behnel
The problem is that the size of "digit" can either be specified by
configure, or determined at C compile time by the header files.

That shouldn't matter. What matters is that the size of "digit" is a
deterministic function of the Python header files.

Post by Stefan Behnel
I've seen
cases where the latter yielded a different size for me than what CPython
was really using internally, thus leading to crashes and/or data corruption.

How can this happen? Doesn't CPython use its own header files??? Perhaps
this is a bug which got fixed?

[Stefan Behnel]

Post by Jeroen Demeyer

Post by Stefan Behnel

Post by Jeroen Demeyer

Post by Stefan Behnel
Well - what do you need it for?

I need to manually create such objects using _PyLong_New() and then
manually fill in the ob_digit field. Currently, I use
(<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
elegant as it could be.

And would you mind sharing with us why you need to create the objects in
this unusual way?

http://trac.sagemath.org/ticket/17853

Ok, got it. That's a very special use case that might call for some very
special measures.

Would _PyLong_FromByteArray() be an option?

Post by Jeroen Demeyer

Post by Stefan Behnel
The problem is that the size of "digit" can either be specified by
configure, or determined at C compile time by the header files.

That shouldn't matter. What matters is that the size of "digit" is a
deterministic function of the Python header files.

Post by Stefan Behnel
I've seen
cases where the latter yielded a different size for me than what CPython
was really using internally, thus leading to crashes and/or data corruption.

How can this happen? Doesn't CPython use its own header files??? Perhaps
this is a bug which got fixed?

https://bugs.python.org/issue22313

As to why this happens, I can't say. Would need some deeper digging than I
wanted to invest at the time.

Stefan

[Jeroen Demeyer]

Post by Stefan Behnel
Would _PyLong_FromByteArray() be an option?

I think so, but that would be less efficient since there would be a
double conversion
PyLong <-> unsigned char* <-> mpz_t
instead of a direct conversion
PyLong <-> mpz_t

[Stefan Behnel]

Post by Stefan Behnel

Post by Jeroen Demeyer

Post by Stefan Behnel

Post by Jeroen Demeyer
I need to manually create such objects using _PyLong_New() and then
manually fill in the ob_digit field. Currently, I use
(<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
elegant as it could be.

The problem is that the size of "digit" can either be specified by
configure, or determined at C compile time by the header files.

That shouldn't matter. What matters is that the size of "digit" is a
deterministic function of the Python header files.

Post by Stefan Behnel
I've seen
cases where the latter yielded a different size for me than what CPython
was really using internally, thus leading to crashes and/or data corruption.

How can this happen? Doesn't CPython use its own header files??? Perhaps
this is a bug which got fixed?

https://bugs.python.org/issue22313
As to why this happens, I can't say. Would need some deeper digging than I
wanted to invest at the time.

I tried to reproduce this problem now and failed. So I'll just assume it
generally works and enabled the PyLong optimisations for all installations,
also for Py2.7 (it was previously limited to Py3.x). While not helping in
your use case, it should speed up arithmetic/binary operations with C long
sized Python integers in Cython, as long as one of them is a constant (so
that Cython can guess what the other one will also be a number).

Stefan