PEP 249 defaults a database cursor to one row per .fetchmany() call. PySpark defaults an Arrow record batch to 10,000 records. Same question in both documents — how many rows move per crossing — and four orders of magnitude between the answers.
Python sits between the model and the warehouse because almost none of the work happens in Python. The libraries that matter are C and CUDA with a Python surface, so the language is where three runtimes’ type systems meet. Syntax isn’t the reason. Knowing what each crossing costs is the scarce part.
Why Python sits between the model and the warehouse
Python sits there because of a memory-sharing interface nobody brings up: the buffer protocol, PEP 3118, status Final, shipped with Python 3.0. It lets any extension module “create objects that share memory … manipulate raw memory from arbitrary objects that export the interface”.
A NumPy array is shaped for that handover. The NumPy internals page splits one into “the raw array data” and “the information about the raw array data”: shape, dtype, strides. (A stride “does not have to be a multiple of the element size”, it says — hold on to that, it comes back under object dtype.)
So a call into C converts nothing. The data stays put; a pointer and a few integers of metadata cross instead. The handle moves, not the array.
A data and AI stack crosses four runtime boundaries, and each has a price
Four boundaries, and only two publish a batch size: Python to C over PEP 3118, Python to the JVM over Py4J and Arrow, Python to the GPU over the CUDA Array Interface or DLPack, Python to the database over PEP 249’s cursor.
The table gives each boundary’s published default and what it governs. Two cells hold no number, and that is the finding.
| Boundary | Carries the data | Published default, and what it governs |
|---|---|---|
| Python → C | PEP 3118 buffer protocol | none; the array crosses as one handle |
| Python → JVM | Arrow batches to a forked worker | 10,000 records per batch |
| Python → GPU | CUDA Array Interface v3 | none; a device pointer, optional stream |
| Python → database | PEP 249 cursor, .arraysize |
1 row per .fetchmany() |
PySpark’s debugging page for 4.2.0 is flat about the JVM boundary: “On the driver side, PySpark communicates with the driver on JVM by using Py4J”, and executor-side workers are “lazily launched only when Python native functions or data have to be handled”. Data counts too, so an RDD call with no user function in it still starts one. Apache Arrow in PySpark says what crosses: Arrow is “used in Spark to efficiently transfer data between JVM and Python processes”.
Processes. Plural, separate, named.
And that plural bounds what Arrow can promise. The Arrow C data interface specification, the in-process ABI for one address space, claims “Zero-copy by design”; the IPC format’s listed advantage is that it “Works across processes and machines”. A forked worker gets the second, not the first. Written, then read, whatever the flag says.
The GPU boundary is thinnest of all. Version 3 of the CUDA Array Interface hands over a device pointer; the cost hides in the optional stream, where somebody must synchronise “at the point of consumption”.
Does the global interpreter lock slow down NumPy, pandas and PySpark?
Mostly no, and CPython’s own documentation says why. Start with what the lock protects. Section 3.1 of the Python Language Reference, Objects, values and types, carries this note:
CPython currently uses a reference-counting scheme with (optional) delayed detection of cyclically linked garbage, which collects most objects as soon as they become unreachable
Reference counts are what the lock keeps correct, and the C API page on thread states is blunt: “only a thread that holds the GIL may operate on Python objects or invoke Python’s C API”. Work touching no Python object needs no lock, and the glossary entry adds that “the GIL is always released when doing I/O”. None of it is new; it’s been the answer in numerical Python for a decade.
PySpark’s case is stronger, because no shared lock exists to contend for. Executor-side Python runs in processes forked from pyspark.daemon, each with its own interpreter and lock. Contention is structurally impossible. The cost is actually the pipe, not the lock.
But the ceiling is published too. PEP 703, in its Alternatives section under “Releasing the GIL in C-API Extensions”, grants that releasing the lock buys “some degree of parallelism”, then says the cost of acquiring and releasing it “typically prevents this from scaling efficiently beyond a few threads”. NumPy maintainer Ralf Gommers, quoted in the same PEP, says it shorter: “NumPy does release the GIL in its inner loops (which do the heavy lifting), but that is not nearly enough.”
Where the GIL actually costs you: object dtype, small arrays, the loader and the server
Four places, all documented: object dtype, loops too small for NumPy to drop the lock, the data input pipeline that feeds a GPU, and model serving.
The second carries a number almost nobody quotes. In the Group 1 macro list near the foot of the NumPy C-API array reference, under Threading support, NPY_BEGIN_THREADS_THRESHOLDED(int loop_size) is “useful to release the GIL only if loop_size exceeds a minimum threshold, currently set to 500”. Exceeds, so at or below 500 elements the lock stays on. Release and reacquire cost more than the loop.
Object dtype is the first place, and it’s where the stride aside stops working. A slice or a transpose still crosses as a handle, because a stride can say anything and the C side walks what it is told. An object-dtype array is a block of pointers instead — every one a reference count the interpreter has to hold. NumPy’s thread safety page says it outright: “operations on arrays with dtype=np.object_ do not release the GIL”, and those “might be better served with multiprocessing”. A pandas column of Python strings is that case.
The last two sit at the edges of a GPU job. PEP 703’s motivation section says keeping a GPU fed means spreading the data input pipeline across CPU cores, and quotes PyTorch’s Zachary DeVito: “We often end up with 72 processes in place of one because of the GIL.” Another subsection is titled “The GIL Makes Deploying Python AI Models Difficult”. The matrix multiply was never the problem. The loader and the server are.
Check the rows per crossing before you tune the query
At the database boundary the published default is one row per .fetchmany() call, and a 50,000,000-row read taken that way spends 100 minutes on round trips before a byte is decoded. Both inputs are chosen, not measured: 50,000,000 rows, 120 microseconds per crossing.
PEP 249 puts .arraysize in its Cursor methods list, between .nextset() and .setinputsizes(). The sentence everybody quotes is “It defaults to 1 meaning to fetch a single row at a time”; the two beside it matter more. The attribute “specifies the number of rows to fetch at a time with .fetchmany()”, and implementations “must observe this value with respect to the .fetchmany() method, but are free to interact with the database a single row at a time”. So the default binds one method and raising it binds nobody: a driver handing .fetchmany() 10,000 rows might still have gone to the database one row at a time. The specification guarantees a floor and promises no ceiling.
50,000,000 rows / 1 row per crossing = 50,000,000 crossings. At 120 µs each that is 6,000,000,000 µs = 6,000,000 ms = 6,000 s = 100 minutes. Set .arraysize to 10,000, PySpark’s published batch size, and the read becomes 50,000,000 / 10,000 = 5,000 crossings, so 5,000 × 120 µs = 600,000 µs = 600 ms = 0.6 s. A second route reaches 100 minutes from capacity and counts no crossings: one row per crossing at 120 µs is a ceiling of 1 / 0.000120 = 8,333 rows a second, and 50,000,000 / 8,333 = 6,000 s to the nearest second.
A round trip to a database and a write to a pipe are different costs, and my 120 µs belongs to the first, so I won’t set those wall times side by side. The 10,000x in the title is a ratio between published defaults, not a benchmark.
The argument about the lock never reaches the cursor.
So where’s the floor? Set a tolerance first, boundary overhead at or under 1% of job wall time, and call it the crossing budget. No published number survives being copied: they spread across four orders of magnitude, 1 row for the cursor, 500 elements for NumPy’s threshold, 10,000 records for an Arrow batch. For a job with 600 seconds of useful work, also chosen, the budget is 0.01 × 600 = 6 s. At 120 µs that allows 6,000,000 µs / 120 µs = 50,000 crossings, so 50,000,000 rows / 50,000 crossings = 1,000 rows per crossing. Below roughly 1,000 rows per crossing the boundary is the job — query tuning cannot touch it.
Derate the cost fourfold, to 30 µs, and the floor moves: 6,000,000 µs / 30 µs = 200,000 crossings, and 50,000,000 / 200,000 = 250 rows per crossing. A quarter of the figure I just defended, and below NumPy’s 500. The threshold is the sensitive number; the ratio between the defaults is not, because no cost ever went into it. Double the volume and the floor doubles: 100,000,000 rows against 50,000 crossings is 2,000 rows per crossing. We don’t price a Spark or warehouse tuning engagement, the kind on our data engineering page, before counting the crossings.
What spark.sql.execution.arrow.pyspark.enabled defaults to in Spark 3.5, 4.0, 4.1 and 4.2
Off in 3.5.7, off in 4.0.0, off in 4.1.0, on in 4.2.0. The flip lands three minor releases into Spark 4, not at the major boundary where everybody guesses.
The 3.5.7 Arrow page tells you to set the flag true, then says “This is disabled by default”, and the 4.0.0 and 4.1.0 pages carry that sentence in the same slot; the 4.2.0 page says “This is enabled by default”. The page moved too, from user_guide/sql/ to tutorial/sql/, so an old bookmark 404s rather than showing a stale default. Four tabs settle it in under a minute.
The obvious response is to raise maxRecordsPerBatch above 10,000, and the Arrow page pushes back in a footnote of a sentence: “If the number of columns is large, the value should be adjusted accordingly.” Downwards. A batch is rows times columns, so at a chosen 1,200 columns, ordinary in a reporting layer, 10,000 rows is 10,000 × 1,200 = 12,000,000 cells. Put a chosen 8 bytes on a cell and that is 96,000,000 bytes, so 96,000,000 / 1,000,000 = 96 MB a batch per worker, and a chosen eight workers hold 8 × 96 = 768 MB live. The same page warns that Arrow conversion “can temporarily lead to high memory usage in the JVM”.
The 8 bytes is where the obvious estimate breaks, in the expensive direction. A fixed-width column really is 8 bytes a cell, but an Arrow string column is an offset buffer plus a variable-width one, so a mostly-text table runs well above 8 and the 768 MB is a floor. Raising the batch size turns a slow job into a failed one.
Free-threaded Python is officially supported in 3.14, and I would not move this workload to it
Free threading is supported and no longer experimental as of Python 3.14, and I’d still not move a data or AI workload onto it in 2026. What’s New In Python 3.14 puts it plainly: “This is the start of phase II where free-threaded Python is officially supported but still optional”.
My objection is packaging rather than speed. Every compiled dependency needs a second wheel, and the C API extensions HOWTO says free-threaded “wheels, shared libraries, and binaries are indicated by a t suffix”, on a build that “does not currently support the Limited C API or the stable ABI”. One unprepared extension turns the lock back on at import, since the free-threading HOWTO says some packages “may not be ready for use in a free-threaded build, and will re-enable the GIL”. You find out afterwards, by calling sys._is_gil_enabled(). If your bottleneck is a pure-Python loop over a million objects and you own every extension in your tree, it might be worth it. A Spark pipeline is neither.
Which overhead figure applies depends on your platform and your C compiler, and python.org hands you four. PEP 703 reports 5–6% on pyperformance 1.0.6, against a prototype build from early 2023. PEP 779 says “currently around 10% (except on macOS, where it’s more like 3%)”, and What’s New In Python 3.14 says “roughly 5-10%, depending on the platform and C compiler used”. The free-threading HOWTO, which tracks main, says “about 1% on macOS aarch64 to 8% on x86-64 Linux systems”. Those are not four readings of one artefact — they are three years of an implementation getting faster, so any figure older than your release is stale and none was taken on your compiler. I’d measure it on the build I was about to ship.
Is Python still worth learning for a data engineering career?
Yes, and on its own that is worth close to nothing, because Python is the cheapest part of this job to produce now. Fluency is the entry fee. What gets paid for is the four crossings and their prices: which dtype drags the interpreter back into the loop at the C boundary, what the batch size is on each side of the JVM boundary and which release’s default you are on, who synchronises the stream at the GPU boundary, and what .arraysize is at the database boundary. None of that is syntax, and none of it is in a Python course.
The uncomfortable half is that a model writes the Python faster than you do and always will, while the list price of that writing falls and the bill goes up — its own argument, about how token prices fell 73% and the same answer costs 2.7 times more. What a model is bad at today is knowing that the default flipped at Spark 4.2 and not at Spark 4.0, and that a 1,200-column table makes the obvious fix worse than the problem. I won’t pretend that gap is permanent. It closes the day retrieval against your pinned versions and your real config is routine, and the scarce skill moves again, to whoever can say which number was worth measuring. The argument about the syntax will still be running.