[pull] main from tobymao:main#830
Open
pull[bot] wants to merge 7878 commits into
Open
Conversation
5882a13 to
7a70164
Compare
32ef521 to
bbeb881
Compare
5068bc7 to
14f9e93
Compare
2f19d1d to
f1c4fa0
Compare
84af655 to
04002ae
Compare
ff7370c to
aa14297
Compare
cc48178 to
e18a24e
Compare
Build the TokenizerCore fresh in Tokenizer.__init__ instead of caching it in a process-lifetime thread-local. The cache stored a stateful core keyed by tokenizer type; building fresh keeps the core scoped to its Tokenizer instance and removes the shared mutable thread-local.
Signed-off-by: c-tonneslan <cst0520@gmail.com>
…res (#7855) * fix(bigquery): transpile single projection GENERATE_SERIES from postgres * ref * ref impl * ref 2 * ref 3 * ref f
Co-authored-by: Giorgos Michas <geomichas96@gmail.com>
Co-authored-by: Giorgos Michas <geomichas96@gmail.com>
…REGR_SYY (#7851) * feat(typing): add databricks type inference for REGR_SXX, REGR_SXY, REGR_SYY * fix(typing): correct DISTINCT arg semantics for REGR_SXX and REGR_SYY in Databricks * refactor(typing): extract shared _regr_sql helper in DatabricksGenerator * fix: update databricks generator to have entries in transform
…iter, preserve escapes in raw strings (#7870) * Fix(tokenizer)!: handle escaped quote before closing multi-char delimiter, preserve escapes in raw strings * Sync w/ integration tests
* feat(typing): add databricks type inference for RINT, ROUND * fix(optimizer)!: revert Round pass-through annotator for databricks [CLAUDE]
* fix(spark): parse month, day without leading zeros * fix(spark): also fix generation of `exp.StrToTime` * chore(spark): narrow type to fix mypy attr-defined * fix(spark): parse strict month/day via distinct canonical format Spark 3+ parses MM/dd strictly (single-digit months/days don't parse), unlike the lax %m/%d most dialects produce. Map Spark's MM/dd to a distinct canonical token (%mstrict/%dstrict) only when parsing, so the strict format roundtrips while lax %m/%d still becomes the lenient M/d. Formatting keeps the padded %m/%d -> MM/dd. The internal tokens degrade to %m/%d for every other dialect via the metaclass inverse fallback and the generic strtotime_sql. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(spark): narrow generator dialect type via TYPE_CHECKING Replace the per-call local import and `assert isinstance(self.dialect, Spark)` in SparkGenerator.format_time with a TYPE_CHECKING-guarded `dialect: Spark` class annotation, removing the runtime overhead while keeping mypy happy. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(bigquery): degrade strict time token in FORMAT clause The strict->lax canonical fallback was applied only to INVERSE_TIME_MAPPING, so BigQuery's `FORMAT '...'` clause (which uses INVERSE_FORMAT_MAPPING) leaked the internal token, e.g. Spark `TO_DATE(x, 'MM/dd/yyyy')` -> BigQuery `... FORMAT 'MMstrict/DDstrict/YYYY'`. Apply the same fallback to INVERSE_FORMAT_MAPPING via a shared helper so it degrades to 'MM/DD/YYYY'. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(spark): share parse-time expression set, clarify strtotime CLAUDE Reuse dialect.STRICT_PARSE_TIME_EXPRESSIONS in SparkGenerator.format_time instead of a duplicate LENIENT_TIME_EXPRESSIONS tuple, removing the cross-module constant that had to be kept in sync by hand. Also document why the base strtotime_sql degrades only the strict tokens rather than routing the whole format through self.format_time(). Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(spark): host lenient time mapping on the base Dialect CLAUDE Declare LENIENT_INVERSE_TIME_MAPPING/_TRIE on the base Dialect (mirroring STRICT_TIME_MAPPING and INVERSE_TIME_MAPPING) and build the trie in the metaclass. This drops the Spark-only attribute that forced a `dialect: Spark` type-narrowing annotation in SparkGenerator plus its TYPE_CHECKING import, and lets spark.py stop hand-building the trie. No behavior change. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(spark): use `or` fallback in format_time, matching the base CLAUDE Replace the `inverse_time_mapping is None and ...` guard with the same `inverse_time_mapping or ...` fallback idiom the base Generator.format_time uses. Same semantics (an explicitly-passed mapping still wins), but consistent with the base and without the extra clause. No behavior change. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * docs(spark): trim over-explaining comments on lenient time mapping CLAUDE Drop the "used by the generator's format_time override / trie built by the metaclass" note on Spark.LENIENT_INVERSE_TIME_MAPPING (no other mapping documents where it's consumed or that the metaclass builds its trie), and reword the base docstring's awkward "Lets e.g. Spark" to the parenthetical "(e.g. Spark)" form already used by STRICT_TIME_MAPPING. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(hive, spark): model strict MM/dd parsing via TIME_MAPPING + a flag CLAUDE Applies the maintainer's feedback on #7773 and extends strict date parsing to the whole Hive family. Instead of patch-1's STRICT_TIME_MAPPING + LENIENT_INVERSE_TIME_MAPPING (and the `strict` flag threaded through Dialect.format_time / build_formatted_time), fold MM -> %mstrict into the regular TIME_MAPPING and decide the parse/format role in the generator. The _Dialect metaclass keeps degrading %mstrict -> %m for non-strict dialects. Strictness (modern Hive, Spark 3+, Databricks are strict; Spark 2 is lenient) is a single dialect flag STRICT_TIME_PARSING; a lenient canonical %m is rendered as the non-padded M for parse expressions in HiveGenerator.format_time, so single-digit sources stay parseable while %mstrict round-trips to MM. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(dialects): derive strict-time inverse in the metaclass CLAUDE Hive, Spark, and Databricks each restated the same `INVERSE_TIME_MAPPING = {"%m": "MM", "%d": "dd"}` to restore the padded formatting inverse that MM/dd -> %mstrict/%dstrict displaces. Fold that into `_with_strict_time_fallback`: for a strict dialect (whose auto-derived inverse already contains %mstrict), default the lax %m/%d to the strict form's inverse. This is the maintainer's suggestion on #7773 - set the strict entries once in the _Dialect metaclass instead of repeating them per dialect. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(dialects): drive strict-time tokens from STRICT_TIME_PARSING CLAUDE Strictness was declared twice per dialect: the STRICT_TIME_PARSING flag and the hand-written MM/dd -> %mstrict/%dstrict entries in TIME_MAPPING, which had to be kept in sync. Make the flag the single source of truth and derive the tokens in the _Dialect metaclass: promote lax %m/%d to the strict tokens for strict dialects, strip them back otherwise (needed when a strict base is inherited, e.g. Spark 2 <- Hive). Dialects now declare only the flag - Spark and Spark2 drop their TIME_MAPPING overrides entirely, and the %mstrict/%dstrict tokens no longer appear in any dialect file. Rename the inverse helper to _with_strict_time_inverse so it pairs with the new _with_strict_time_mapping. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * refactor(hive): render parse formats in one pass via a derived mapping CLAUDE The strict-dialect format_time override ran two passes: first rewriting the lax canonical %m/%d to %-m/%-d (with identity entries just to keep the trie from partial-matching the %m prefix inside %mstrict), then converting through INVERSE_TIME_MAPPING. Instead, derive a PARSE_INVERSE_TIME_MAPPING in the _Dialect metaclass - INVERSE_TIME_MAPPING with %m/%d overridden to render non-padded - and pass it to the base format_time in a single call. This drops the LENIENT_PARSE_TIME_MAPPING identity-entry hack, halves the format traversals, and makes the parse-side rendering an explicit named mapping. Lenient dialects alias the plain inverse mapping, so no extra tries are built for them. Assisted-by: Claude Fable 5 <noreply@anthropic.com> * fix(hive, spark): extend lenient parse formats to StrToUnix CLAUDE UNIX_TIMESTAMP parses its string input, but exp.StrToUnix was missing from PARSE_TIME_EXPRESSIONS, so a lax foreign %m/%d still rendered as the strict MM/dd - e.g. STR_TO_UNIX(x, '%Y-%m-%d') became UNIX_TIMESTAMP(x, 'yyyy-MM-dd'), which silently returns NULL for single-digit months/days in Spark 3+ and modern Hive. Include it so lax sources render as M/d, matching the StrToTime/StrToDate/TsOrDsToDate handling; Spark's own strict MM/dd still roundtrips via %mstrict/%dstrict. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
* fix for projection collision * updates from PR feedback * merge issues * add schema, update merged_sources, search deeper into group by * fix(optimizer): revert unrelated qualify_columns star-dedup change (CLAUDE) This half of the previous commit was the query-star-collision fix and belongs on optimizer/bug-query-star-collision instead. Keeps this branch scoped to the merge_subqueries.py literal GROUP BY fix. * added test and handling for empty schema
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )