cassandra/query: use timezone specific API to avoid deprecated warning#1213
cassandra/query: use timezone specific API to avoid deprecated warning#1213absurdfarce merged 1 commit intoapache:masterfrom
Conversation
|
Thanks for the PR @tchaikov ! Have you signed the Contributor License Agreement for contributions to DataStax open source projects? If not you can find it at https://cla.datastax.com/. Thanks! |
|
@absurdfarce yeah, i did. |
|
Excellent, thank you @tchaikov! I like this change; it looks like the deprecation came in with Python 3.12 and we're doing a lot of work now to try and stabilize our story on that Python version. If we're going to make this change, though, I think we should make it everywhere datetime.datetime.utcfromtimestamp() is used throughout the code base. That means we're looking at the following instances: Are you able to update this PR to include fixes for all these occurrences? If not I can certainly create a PR to do so. Just let me know! Thanks again! |
bd03409 to
3b6bcf3
Compare
|
@absurdfarce hi Bret, thanks for your review. $ rg utcfromtimestamp || echo done
done |
ee56bd8 to
b7ce43c
Compare
absurdfarce
left a comment
There was a problem hiding this comment.
Looking pretty good @tchaikov! It looks like there are a few places where the presence or absence of timezone information in the datetime object needs to be a bit more consistent but overall I think we're pretty close here.
| return datetime(*(value.timetuple()[:6])) | ||
|
|
||
| return datetime.utcfromtimestamp(value) | ||
| return datetime.fromtimestamp(value, tz=timezone.utc).replace(tzinfo=None) |
There was a problem hiding this comment.
Straight replace of utcfromtimestamp() here so a naive datetime object seems right
| def __init__(self, description, timeuuid, source, source_elapsed, thread_name): | ||
| self.description = description | ||
| self.datetime = datetime.utcfromtimestamp(unix_time_from_uuid1(timeuuid)) | ||
| self.datetime = datetime.fromtimestamp(unix_time_from_uuid1(timeuuid), tz=timezone.utc) |
There was a problem hiding this comment.
Creation time for trace objects should absolutely be a fixed point in time regardless of time zone so maintaining UTC here seems correct
| DATETIME_EPOC = datetime.datetime(1970, 1, 1) | ||
| UTC_DATETIME_EPOC = datetime.datetime.utcfromtimestamp(0) | ||
| DATETIME_EPOC = datetime.datetime(1970, 1, 1, tzinfo=datetime.timezone.utc).replace(tzinfo=None) | ||
| UTC_DATETIME_EPOC = datetime.datetime.fromtimestamp(0, tz=datetime.timezone.utc).replace(tzinfo=None) |
There was a problem hiding this comment.
Another outright replace of utcfromtimestamp() so conversion to a naive object makes sense here as well
|
|
||
| DATETIME_EPOC = datetime.datetime(1970, 1, 1) | ||
| UTC_DATETIME_EPOC = datetime.datetime.utcfromtimestamp(0) | ||
| DATETIME_EPOC = datetime.datetime(1970, 1, 1, tzinfo=datetime.timezone.utc).replace(tzinfo=None) |
There was a problem hiding this comment.
Nit: I don't think you actually want to use tzinfo=utc here. This value is contrasted with UTC_DATETIME_EPOC so I understand it as corresponding to the local time version of the epoch.
| self.DatetimeTest.objects.create(test_id=5, created_at=dt_value) | ||
| dt2 = self.DatetimeTest.objects(test_id=5).first() | ||
| self.assertEqual(dt2.created_at, datetime.utcfromtimestamp(dt_value)) | ||
| self.assertEqual(dt2.created_at, datetime.fromtimestamp(dt_value, tz=timezone.utc)) |
There was a problem hiding this comment.
This one needs to have the timezone removed. You're testing against what cqlengine creates and in the changes to cassandra.cqlengine.columns above the resulting Python type is now datetime instance with no timezone specified. Failing to remove the timezone here means that we're comparing a naive (i.e. having no timezone info) datetime instance to an aware (i.e. being aware of it's timezone) instance:
$ python3 -V
Python 3.8.16
$ CASSANDRA_VERSION="4.0.2" with_ssl pynose tests/integration/cqlengine/columns/test_validation.py
…
======================================================================
ERROR: test_conversion_specific_date (tests.integration.cqlengine.columns.test_validation.TestTimeUUIDFromDatetime)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/work/git/python-driver/tests/integration/cqlengine/columns/test_validation.py", line 812, in test_conversion_specific_date
new_dt = datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
AttributeError: type object 'datetime.datetime' has no attribute 'timezone'
======================================================================
FAIL: test_datetime_timestamp (tests.integration.cqlengine.columns.test_validation.TestDatetime)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/work/git/python-driver/tests/integration/cqlengine/columns/test_validation.py", line 100, in test_datetime_timestamp
self.assertEqual(dt2.created_at, datetime.fromtimestamp(dt_value, tz=timezone.utc))
AssertionError: datetime.datetime(2016, 2, 3, 17, 29, 14) != datetime.datetime(2016, 2, 3, 17, 29, 14, tzinfo=datetime.timezone.utc)
----------------------------------------------------------------------
Ran 49 tests in 16.571s
FAILED (errors=1, failures=1)
|
|
||
| ts = (uuid.time - 0x01b21dd213814000) / 1e7 # back to a timestamp | ||
| new_dt = datetime.utcfromtimestamp(ts) | ||
| new_dt = datetime.fromtimestamp(ts, tz=datetime.timezone.utc) |
There was a problem hiding this comment.
This fails because "datetime" in this test means "datetime.datetime" so you're trying to reference "datetime.datetime.timezone" here... which doesn't exist.
======================================================================
ERROR: test_conversion_specific_date (tests.integration.cqlengine.columns.test_validation.TestTimeUUIDFromDatetime)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/work/git/python-driver/tests/integration/cqlengine/columns/test_validation.py", line 812, in test_conversion_specific_date
new_dt = datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
AttributeError: type object 'datetime.datetime' has no attribute 'timezone'
You can fix that by just using "tz=timezone.utc" here (since we already imported "timezone" in the same import statement referenced above) but that leaves you with a new problem:
======================================================================
FAIL: test_conversion_specific_date (tests.integration.cqlengine.columns.test_validation.TestTimeUUIDFromDatetime)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/work/git/python-driver/tests/integration/cqlengine/columns/test_validation.py", line 815, in test_conversion_specific_date
assert new_dt == dt
AssertionError
Problem here is that you're comparing an aware datetime instance "new_dt" to a reference value "dt" which is naive. You can fix that by doing a "replace(tzinfo=None)" on new_dt here to make it naive as well.
|
|
||
| AllDatatypesModel.create(id=0, a='ascii', b=2 ** 63 - 1, c=bytearray(b'hello world'), d=True, | ||
| e=datetime.utcfromtimestamp(872835240), f=Decimal('12.3E+7'), g=2.39, | ||
| e=datetime.fromtimestamp(872835240, tz=timezone.utc), f=Decimal('12.3E+7'), g=2.39, |
There was a problem hiding this comment.
Same problem here. This test also fails now because cassandra.cqlengine.columns.DateTime.to_python() returns a datetime instance without timezones (due to the changes above). That causes this test to fail unless the target timezone also doesn't contain a timezone:
Traceback (most recent call last):
File "/home/jenkins/.pyenv/versions/3.12.0/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
yield
File "/home/jenkins/.pyenv/versions/3.12.0/lib/python3.12/unittest/case.py", line 634, in run
self._callTestMethod(testMethod)
File "/home/jenkins/.pyenv/versions/3.12.0/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
if method() is not None:
^^^^^^^^
File "/home/jenkins/workspace/drivers_python_oss_PR-1213/tests/integration/cqlengine/model/test_model_io.py", line 216, in test_can_insert_model_with_all_column_types
self.assertEqual(input[i], output[chr(i_char)])
File "/home/jenkins/.pyenv/versions/3.12.0/lib/python3.12/unittest/case.py", line 885, in assertEqual
assertion_func(first, second, msg=msg)
File "/home/jenkins/.pyenv/versions/3.12.0/lib/python3.12/unittest/case.py", line 878, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: datetime.datetime(1997, 8, 29, 6, 14, tzinfo=datetime.timezone.utc) != datetime.datetime(1997, 8, 29, 6, 14)
|
|
||
| input = AllDatatypes(a='ascii', b=2 ** 63 - 1, c=bytearray(b'hello world'), d=True, | ||
| e=datetime.utcfromtimestamp(872835240), f=Decimal('12.3E+7'), g=2.39, | ||
| e=datetime.fromtimestamp(872835240, tz=timezone.utc), f=Decimal('12.3E+7'), g=2.39, |
There was a problem hiding this comment.
Same here. Just adding ".replace(tzinfo=None)" onto the value for the keyword "e" here causes the test to pass (for reasons identical to what is outlined above).
9629f84 to
dc55556
Compare
|
v3:
|
|
@absurdfarce hi Bret, thank you for the detailed and constructive review. it's very helpful. i rely on the CI to identify the test failures. am i able to access the jenins job you referenced in the comment as an external contributor? |
|
Hey @tchaikov, thanks for all the work to address the various test issues! Unfortunately I can't make the Jenkins results public but I'm happy to try another run with the various changes you've made here. That's been kicked off; I'll report back here when it completes and we can see whether all the outstanding test issues have been addressed! |
|
Hey @tchaikov, Jenkins run just finished and it looks like we still have a problem. I observed the following on all Python + Cassandra combinations we test against: I can repro this error by running the test locally, so that's a good thing. Looks like that test explicitly compare the "input" sequence to whatever it gets back from the cqlengine model. And since elsewhere in this PR we explicitly change the model to return a naive datetime we need to compare against one here as well. So "input" needs to contain a naive datetime which means this change will fix the issue: diff --git a/tests/integration/cqlengine/model/test_model_io.py b/tests/integration/cqlengine/model/test_model_io.py
index c4880d4f..9414d83d 100644
--- a/tests/integration/cqlengine/model/test_model_io.py
+++ b/tests/integration/cqlengine/model/test_model_io.py
@@ -197,7 +197,7 @@ class TestModelIO(BaseCassEngTestCase):
sync_table(AllDatatypesModel)
- input = ['ascii', 2 ** 63 - 1, bytearray(b'hello world'), True, datetime.fromtimestamp(872835240, tz=timezone.utc),
+ input = ['ascii', 2 ** 63 - 1, bytearray(b'hello world'), True, datetime.fromtimestamp(872835240, tz=timezone.utc).replace(tzinfo=None),
Decimal('12.3E+7'), 2.39, 3.4028234663852886e+38, '123.123.123.123', 2147483647, 'text',
UUID('FE2B4360-28C6-11E2-81C1-0800200C9A66'), UUID('067e6162-3b6f-4ae2-a171-2470b63dff00'),
int(str(2147483647) + '000')]It's also worth noting that the referenced change to cassandra.cqlengine.columns will always return a naive datetime regardless of input. That means we can supply a datetime with or without time zone info to our model creation call. That in turn means that this change also passes the test without issue: diff --git a/tests/integration/cqlengine/model/test_model_io.py b/tests/integration/cqlengine/model/test_model_io.py
index 9414d83d..ce43422e 100644
--- a/tests/integration/cqlengine/model/test_model_io.py
+++ b/tests/integration/cqlengine/model/test_model_io.py
@@ -203,7 +203,7 @@ class TestModelIO(BaseCassEngTestCase):
int(str(2147483647) + '000')]
AllDatatypesModel.create(id=0, a='ascii', b=2 ** 63 - 1, c=bytearray(b'hello world'), d=True,
- e=datetime.fromtimestamp(872835240, tz=timezone.utc).replace(tzinfo=None), f=Decimal('12.3E+7'), g=2.39,
+ e=datetime.fromtimestamp(872835240, tz=timezone.utc), f=Decimal('12.3E+7'), g=2.39,
h=3.4028234663852886e+38, i='123.123.123.123', j=2147483647, k='text',
l=UUID('FE2B4360-28C6-11E2-81C1-0800200C9A66'),
m=UUID('067e6162-3b6f-4ae2-a171-2470b63dff00'), n=int(str(2147483647) + '000'),Note that the first change above is required to make the test pass. The second change is entirely optional. |
|
v2:
|
absurdfarce
left a comment
There was a problem hiding this comment.
Good news is that with your last round of changes @tchaikov we did get rid of some of the test failures in the Jenkins run. Bad news is that when I looked originally I missed a few test failures that were hiding in amongst the failures we already discussed. The new failures look like this:
Build and Test / 3.11, py3.8.16, Cython / tests.unit.cython.test_types.TypesTest.test_datetype (from nosetests)
Error Message
datetime.datetime(1970, 1, 1, 0, 0) != datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
Stacktrace
Traceback (most recent call last):
File "/home/jenkins/.pyenv/versions/3.8.16/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/home/jenkins/.pyenv/versions/3.8.16/lib/python3.8/unittest/case.py", line 676, in run
self._callTestMethod(testMethod)
File "/home/jenkins/.pyenv/versions/3.8.16/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
method()
File "/home/jenkins/workspace/drivers_python_oss_PR-1213/tests/unit/cython/test_types.py", line 25, in test_datetype
types_testhelper.test_datetype(self.assertEqual)
File "tests/unit/cython/types_testhelper.pyx", line 55, in tests.unit.cython.types_testhelper.test_datetype
assert_equal(deserialize(expected), datetime.datetime.fromtimestamp(expected, tz=datetime.timezone.utc))
File "/home/jenkins/.pyenv/versions/3.8.16/lib/python3.8/unittest/case.py", line 912, in assertEqual
assertion_func(first, second, msg=msg)
File "/home/jenkins/.pyenv/versions/3.8.16/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: datetime.datetime(1970, 1, 1, 0, 0) != datetime.datetime(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
I tracked down the root cause of these failures and added some comments to types_testhelper.pyx which should address them.
| cdef Buffer buf | ||
|
|
||
| dt = datetime.datetime.utcfromtimestamp(timestamp) | ||
| dt = datetime.datetime.fromtimestamp(timestamp, tz=datetime.timezone.utc) |
There was a problem hiding this comment.
The original impl (using utcfromtimestamp()) returns a naive datetime and this version doesn't. But after looking into this a bit it doesn't actually matter. This datetime is immediately serialized and then deserialized in order to test the end-to-end process. All that really matters here are the time values; any timezone info is lost in that process.
| # beyond 32b | ||
| expected = 2 ** 33 | ||
| assert_equal(deserialize(expected), datetime.datetime(2242, 3, 16, 12, 56, 32)) | ||
| assert_equal(deserialize(expected), datetime.datetime(2242, 3, 16, 12, 56, 32, tz=datetime.timezone.utc)) |
There was a problem hiding this comment.
Note that we're dealing with a datetime constructor here rather than a utcfromtimestamp() invocation. We don't need to replace this in order to handle any deprecation concerns.
More to the point: the datetime coming out of deserialize() is naive so this actually introduces a test failure. I believe this should stay as assert_equal(deserialize(expected), datetime.datetime(2242, 3, 16, 12, 56, 32))
There was a problem hiding this comment.
yeah, silly me! reverted this change.
| # epoc | ||
| expected = 0 | ||
| assert_equal(deserialize(expected), datetime.datetime.utcfromtimestamp(expected)) | ||
| assert_equal(deserialize(expected), datetime.datetime.fromtimestamp(expected, tz=datetime.timezone.utc)) |
There was a problem hiding this comment.
This one took some digging to sort out but I believe I have it now. deserialize() returns a naive datetime based on the UTC values so you're correct to continue to resolve the datetime against UTC... but we shouldn't have the timezone in place for what we return. As a result this needs to become assert_equal(deserialize(expected), datetime.datetime.fromtimestamp(expected, tz=datetime.timezone.utc).replace(tzinfo=None))
There was a problem hiding this comment.
yeah, i missed it in previous fixes.
before this change, when testing with cqlsh, we have warnings like: ``` <frozen importlib._bootstrap>:488: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). ```` in this change, we replace the deprecated API with timezone-aware API, to avoid this warning. to keep the backward compatibility, `DateTime.to_python()` still returns an offset-naive timestamp. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
|
v3:
|
|
@absurdfarce hi Bret, thank you very much! was tied up yesterday. just sent another revision. hopefully it's better now. |
|
Last CI run looked great, so I think we're ready to go here! Thanks for the PR (and your perseverance) @tchaikov! |
|
Thank you, Bret! |
before this change, when testing with cqlsh, we have warnings like: ``` <frozen importlib._bootstrap>:488: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). ```` in this change, we replace the deprecated API with timezone-aware API, to avoid this warning. to keep the backward compatibility, `DateTime.to_python()` still returns an offset-naive timestamp. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
before this change, when testing with cqlsh, we have warnings like: ``` <frozen importlib._bootstrap>:488: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). ```` in this change, we replace the deprecated API with timezone-aware API, to avoid this warning. to keep the backward compatibility, `DateTime.to_python()` still returns an offset-naive timestamp. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
before this change, when testing with cqlsh, we have warnings like: ``` <frozen importlib._bootstrap>:488: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). ```` in this change, we replace the deprecated API with timezone-aware API, to avoid this warning. to keep the backward compatibility, `DateTime.to_python()` still returns an offset-naive timestamp. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
before this change, when testing with cqlsh, we have warnings like: ``` <frozen importlib._bootstrap>:488: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). ```` in this change, we replace the deprecated API with timezone-aware API, to avoid this warning. to keep the backward compatibility, `DateTime.to_python()` still returns an offset-naive timestamp. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
before this change, when testing with cqlsh, we have warnings like: ``` <frozen importlib._bootstrap>:488: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). ```` in this change, we replace the deprecated API with timezone-aware API, to avoid this warning. to keep the backward compatibility, `DateTime.to_python()` still returns an offset-naive timestamp. Signed-off-by: Kefu Chai <kefu.chai@scylladb.com>
before this change, when testing with cqlsh, we have warnings like:
in this change, we replace the deprecated API with timezone-aware API, to avoid this warning.