Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on Apr 17, 2026

• [SPARK-56256][PYTHON][DOCS][FOLLOWUP] Document SparkSession.emptyDataFrame in API reference

  Show description for 4fab58a
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  4fab58a

  Copy full SHA for 4fab58a

• [SPARK-56518][PYTHON] Add current_path to PySpark functions

  Show description for 8312182
  

  srielau

  authored and

  HyukjinKwon

  committed
  8312182

  Copy full SHA for 8312182

Commits on Apr 16, 2026

• [SPARK-56507][ML][PYTHON][TESTS] Reduce maxIter in CrossValidator test to speed up CI

  Show description for 26a86f9
  zhengruifeng

  committed
  26a86f9

  Copy full SHA for 26a86f9

Commits on Apr 15, 2026

• [SPARK-56399][PYTHON][TESTS] Add SimpleWorkerTests for non-daemon worker path

  Show description for f61cb45
  

  anblanco

  authored and

  dongjoon-hyun

  committed
  f61cb45

  Copy full SHA for f61cb45

• [SPARK-56351][PYTHON][DOCS] Add dedicated documentation page for Arrow Python UDFs

  Show description for da37644
  

  Yicong-Huang

  authored and

  zhengruifeng

  committed
  da37644

  Copy full SHA for da37644

• [SPARK-56424][PYTHON][TESTS] Add ASV benchmark for SQL_SCALAR_PANDAS_UDF

  Show description for a578296
  

  Yicong-Huang

  authored and

  zhengruifeng

  committed
  a578296

  Copy full SHA for a578296

Commits on Apr 14, 2026

• [SPARK-56463][PYTHON] Disallow unpickling UDT

  Show description for 6e55b20
  

  gaogaotiantian

  authored and

  HyukjinKwon

  committed
  6e55b20

  Copy full SHA for 6e55b20

• [SPARK-54731][SQL] Support Reverse expression to handle BinaryType by performing raw byte-level reversal

  Show description for 3bcfa1e
  

  xiaoxuandev

  authored and

  HyukjinKwon

  committed
  3bcfa1e

  Copy full SHA for 3bcfa1e

• [SPARK-56254][PYTHON][CONNECT] Make spark.read.xml accept DataFrame input

  Show description for 190e3b8
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  190e3b8

  Copy full SHA for 190e3b8

• [SPARK-56387][PYTHON] Refactor ProfileResults to a dict of dict

  Show description for 0bcb175
  

  gaogaotiantian

  authored and

  HyukjinKwon

  committed
  0bcb175

  Copy full SHA for 0bcb175

Commits on Apr 13, 2026

• [SPARK-56255][PYTHON][CONNECT] Make spark.read.csv accept DataFrame input

  Show description for 64f9344
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  64f9344

  Copy full SHA for 64f9344

• [SPARK-56452][PYTHON] Fix pip install failure for prerelease versions

  Show description for 790cad7
  

  kiyeonjeon21

  authored and

  HyukjinKwon

  committed
  790cad7

  Copy full SHA for 790cad7

Commits on Apr 12, 2026

• [SPARK-56253][PYTHON][CONNECT][FOLLOW-UP] Reject multi-column DataFrame input in spark.read.json

  Show description for 638ca41
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  638ca41

  Copy full SHA for 638ca41

Commits on Apr 10, 2026

• [SPARK-56350][SQL] Skip ColumnarToRow for Arrow-backed input to Python UDFs

  Show description for 2d5c430
  

  viirya

  authored and

  dongjoon-hyun

  committed
  2d5c430

  Copy full SHA for 2d5c430

• [SPARK-56430][PYTHON] Remove unnecessary .keys() in dict iterations

  Show description for 2b55507
  

  zhengruifeng

  authored and

  dongjoon-hyun

  committed
  2b55507

  Copy full SHA for 2b55507

• [SPARK-56381][PYTHON][TEST] Add ASV microbenchmark for SQL_COGROUPED_MAP_ARROW_UDF

  Show description for f00ea04
  

  Yicong-Huang

  authored and

  zhengruifeng

  committed
  f00ea04

  Copy full SHA for f00ea04

• [SPARK-56428][PYTHON] Move eval type specific data to eval_conf for UDTF

  Show description for d97ac82
  

  gaogaotiantian

  authored and

  zhengruifeng

  committed
  d97ac82

  Copy full SHA for d97ac82

• [SPARK-56403] Refactor kafka test so it's skipped when dependency is not available

  Show description for 6243211
  

  gaogaotiantian

  authored and

  zhengruifeng

  committed
  6243211

  Copy full SHA for 6243211

Commits on Apr 9, 2026

• [SPARK-56407][BUILD][TESTS] Remove pre-built class files and JARs used in artifact transfer tests

  Show description for 6423cb0
  

  sarutak

  authored and

  HyukjinKwon

  committed
  6423cb0

  Copy full SHA for 6423cb0

• [SPARK-56377][PYTHON] Add type hint for shuffle.py

  Show description for 528386c
  

  gaogaotiantian

  authored and

  HyukjinKwon

  committed
  528386c

  Copy full SHA for 528386c

• [SPARK-56253][PYTHON][CONNECT] Make spark.read.json accept DataFrame input

  Show description for 21ada68
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  21ada68

  Copy full SHA for 21ada68

Commits on Apr 8, 2026

• [SPARK-56221][SQL][PYTHON][FOLLOWUP] Rename TablePartition and remove SHOW CACHED TABLES / listCachedTables

  Show description for a3930d3
  cloud-fan

  committed
  a3930d3

  Copy full SHA for a3930d3

• [SPARK-55306] Add ability to run Kafka tests in Python

  Show description for aa6154b
  

  jerrypeng

  authored and

  viirya

  committed
  aa6154b

  Copy full SHA for aa6154b

Commits on Apr 7, 2026

• [SPARK-55450][SS][PYTHON][DOCS] Document admission control in PySpark streaming data sources

  Show description for 98cdaee
  

  jiteshsoni

  authored and

  HeartSaVioR

  committed
  98cdaee

  Copy full SHA for 98cdaee

• [SPARK-56340][PYTHON] Move input_type schema to eval conf

  Show description for 9dbe381
  

  gaogaotiantian

  authored and

  zhengruifeng

  committed
  9dbe381

  Copy full SHA for 9dbe381

Commits on Apr 6, 2026

• [SPARK-56342][PYTHON] Tighten type hints for refactored eval type functions in worker.py

  Show description for 975b299
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  975b299

  Copy full SHA for 975b299

• [SPARK-56248][PYTHON][SS] Optimize python stateful processor serialization to skip unnecessary list/dict/row construction

  Show description for ae7f6e3
  

  jiateoh

  authored and

  HeartSaVioR

  committed
  ae7f6e3

  Copy full SHA for ae7f6e3

• [SPARK-56359][PYTHON] Remove unused ArrowStreamArrowUDFSerializer

  Show description for e42a561
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  e42a561

  Copy full SHA for e42a561

Commits on Apr 5, 2026

• [SPARK-56349][PYTHON] Remove unused ArrowStreamAggArrowUDFSerializer

  Show description for 08b1390
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  08b1390

  Copy full SHA for 08b1390

• [SPARK-56348][PYTHON] Remove unused ArrowBatchUDFSerializer

  Show description for 561b7b9
  

  Yicong-Huang

  authored and

  HyukjinKwon

  committed
  561b7b9

  Copy full SHA for 561b7b9

• [MINOR][PYTHON] Fix PySparkException failing when messageParameters is omitted

  Show description for c6a198e
  

  fangchenli

  authored and

  HyukjinKwon

  committed
  c6a198e

  Copy full SHA for c6a198e

• [SPARK-49793][PYTHON][TESTS][FOLLOW-UP] Fix test_caching in connect mode

  Show description for c1dd15c
  

  gaogaotiantian

  authored and

  HyukjinKwon

  committed
  c1dd15c

  Copy full SHA for c1dd15c

• [SPARK-54938][PYTHON][TEST][FOLLOW-UP] Fix inferred time unit for pandas >= 3

  Show description for 842eb7b
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  842eb7b

  Copy full SHA for 842eb7b

Commits on Apr 3, 2026

• [SPARK-56322][CONNECT][PYTHON] Fix TypeError when self-joining observed DataFrames

  Show description for d9c8eda
  

  2 peopleauthored and

  ueshin

  committed
  d9c8eda

  Copy full SHA for d9c8eda

• [SPARK-56329][PYTHON] Fix all E721 type comparison violations

  Show description for 5f5fc89
  zhengruifeng

  committed
  5f5fc89

  Copy full SHA for 5f5fc89