Add python3 support and improve ctypes wrapper by arichardson · Pull Request #10 · tmetsch/python-dtrace

arichardson · 2020-12-29T13:32:34Z

This pull requests allows building for python3 and add function
signatures to the ctypes wrapper.

Fixes #3
Fixes #4

We can just declare the enumerators and then the generated C code will use the values from dtrace.h

This commit changes the print statement to use a function and adds the python3 integer division operator. The str/bytes changes will be done in a follow-up change.

It's not currently used but can be useful to print information on which probe fired.

Explicitly set language_level=2 until the port to python3 is complete.

We have to open libdtrace.dylib not libdtrace.so

Also update tests since the callback function now receives a bytes object and not a string.

arichardson · 2020-12-29T13:33:42Z

This is a rather large diff, I can split it up into multiple pull requests if needed. Otherwise I'd recommend looking at each of the individual commits rather than the full diff.

Buffered output hanlding appears to be broken on macOS. The -B flag is disabled in dtrace.c () and compiling dtrace.c with that flag enabled results in the same crash that I see with python-dtrace: ``` * thread tmetsch#1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7fb100000000) * frame #0: 0x00007fff6f83de52 libsystem_platform.dylib`_platform_strlen + 18 frame tmetsch#1: 0x00007fff6f6d9891 libsystem_c.dylib`__vfprintf + 5379 frame tmetsch#2: 0x00007fff6f6ffad3 libsystem_c.dylib`__v2printf + 475 frame tmetsch#3: 0x00007fff6f6e5ee7 libsystem_c.dylib`_vsnprintf + 417 frame tmetsch#4: 0x00007fff6f6e5f90 libsystem_c.dylib`vsnprintf + 68 frame tmetsch#5: 0x00007fff6d3650e3 libdtrace.dylib`dt_printf + 524 frame tmetsch#6: 0x00007fff6d33da7f libdtrace.dylib`dt_consume_cpu + 2536 frame tmetsch#7: 0x00007fff6d33c9e0 libdtrace.dylib`dtrace_consume + 1090 frame tmetsch#8: 0x00007fff6d366534 libdtrace.dylib`dtrace_work + 116 frame tmetsch#9: 0x000000010d6be9ff dtrace2`main(argc=4, argv=0x00007ffee2547250) at dtrace2.c:1834:17 frame tmetsch#10: 0x00007fff6f647cc9 libdyld.dylib`start + 1 ``` To work around this crash, output to stderr for now.

tmetsch · 2020-12-30T10:55:57Z

Thanks for this - this is great! Thanks for putting in the time to refactor this. Will need a bit of time to test this (by any chance did you check this on Solaris & FreeBSD & Mac?), but this certainly warrants a new release imho.

Thanks again, and as usual - drinks on me once when we meet someday :-)

arichardson · 2020-12-30T11:24:59Z

Thanks for this - this is great! Thanks for putting in the time to refactor this. Will need a bit of time to test this (by any chance did you check this on Solaris & FreeBSD & Mac?), but this certainly warrants a new release imho.

Thanks again, and as usual - drinks on me once when we meet someday :-)

I have some follow-up fixes for freebsd+macos that I will post as a PR after some minor cleanups. I don't have access to Solaris so I can only test FreeBSD+macOS (the latter appears to be broken right now).

I've pushed the full set of changes to https://github.com/Cambridge-AdvancedOS/l41-python-dtrace/tree/python3 in case you'd like to have a look at them before I open the pr.

tmetsch · 2020-12-30T11:38:51Z

Ok, that is great to know - I actually do not have access to a current Mac system. Will test this on Solaris so (y)

tmetsch · 2020-12-30T17:33:07Z

Hey, quick Q - have you seen this error when running the simple example for the ctypes examples:

$ pfexec python3 syscall_count_own_walk.py 
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 232, in 'calling callback function'
  File "syscall_count_own_walk.py", line 29, in my_walk
    print('{0:4d} > {1:60s}'.format(instance, name))
TypeError: unsupported format string passed to bytes.__format__

arichardson · 2020-12-30T17:41:10Z

I didn't run that example, but I can try running them all with py2+3

tmetsch · 2020-12-30T17:50:19Z

Seems to be an issue with how the data is provided to the callback. Looks like it is a byte array now, proper formatting fixes it.

tmetsch · 2020-12-30T19:40:30Z

This is a "preview" branch of this PR - did some minor code style changes here and there. This seems to work on Solaris (OmniOS) atm with python 3. Need to check python 2 - once it is backward compatible will merge with the master branch or decide to make python 3 a requirement (it is EoY 2020 after all).

https://github.com/tmetsch/python-dtrace/tree/arichardson-python3-support

arichardson · 2020-12-30T20:16:47Z

@tmetsch Thanks! I checked the hello world example with python2, but I can also make sure the other works for one last py2 release if you'd like?

tmetsch · 2020-12-30T20:23:30Z

This looks all good - I did merge it into the master now. All examples did run with python 2 & 3 on OmniOS. So did the unittests etc. Thanks so much for this - this is awesome!!

arichardson added 21 commits December 29, 2020 12:21

cython: Fix compiler warning about missing const in function signature

71e814f

cython: fix -Wshorten-64-to-32 warning

726f40a

cython: stop re-defining enum values

c7018a1

We can just declare the enumerators and then the generated C code will use the values from dtrace.h

cython: remove unncessary semicolon

e912e8e

cython: prepare for python3

0cf1f7a

This commit changes the print statement to use a function and adds the python3 integer division operator. The str/bytes changes will be done in a follow-up change.

cython: Make a few types match the headers more closely

a7b5dd5

cython: expose dtrace_probedesc_t

975655d

It's not currently used but can be useful to print information on which probe fired.

setup.py: Update for cython 3.0a6

828c07c

Explicitly set language_level=2 until the port to python3 is complete.

cython: import INT_MIN/INT_MAX from libc.stdint

cde343b

ctypes: Fix library name for macOS

d82cb9d

We have to open libdtrace.dylib not libdtrace.so

ctypes: Add function signatures for dtrace functions

62c01cb

ctypes: Reduce code duplication for dtrace_open call

9965abf

ctypes: handle python3 strings

9bccf41

Use python3 print function

2063499

ctypes: Remove unnecessary POINTER() in callback types

c1cfffb

Add dtrace_cython/consumer.c to .gitignore

6802730

Check return value of dtrace_work()

f596ce4

ctypes: Update record definitions to match dtrace.h

a91c497

Support python3 strings

0c13e0c

Also update tests since the callback function now receives a bytes object and not a string.

Fix PyCharm type checker errors in amqp example

0f3497c

Set Cython language_level to 3 when compiling for python3

08f633c

This was referenced Dec 30, 2020

Ctypes on Mac OS X #4

Closed

Can't install on Mac OS X 10.10 #3

Closed

tmetsch merged commit 5c34ac9 into tmetsch:master Dec 30, 2020

arichardson deleted the python3-support branch January 27, 2023 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add python3 support and improve ctypes wrapper#10

Add python3 support and improve ctypes wrapper#10
tmetsch merged 21 commits intotmetsch:masterfrom
arichardson:python3-support

arichardson commented Dec 29, 2020 •

edited

Loading

Uh oh!

arichardson commented Dec 29, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

arichardson commented Dec 30, 2020 •

edited

Loading

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

arichardson commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

arichardson commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

arichardson commented Dec 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arichardson commented Dec 29, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

arichardson commented Dec 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

arichardson commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

arichardson commented Dec 30, 2020

Uh oh!

tmetsch commented Dec 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arichardson commented Dec 29, 2020 •

edited

Loading

arichardson commented Dec 30, 2020 •

edited

Loading