MDEV-40642 WIP: stop re-parsing JSON the server itself produced - #5535
Draft
arcivanov wants to merge 26 commits into
Draft
MDEV-40642 WIP: stop re-parsing JSON the server itself produced#5535arcivanov wants to merge 26 commits into
arcivanov wants to merge 26 commits into
Conversation
Nine new test files capture the current, unchanged behavior of the whole
JSON surface so that any later change to it shows up as an explicit
result diff rather than as a judgement call.
- `func_json_format` - the exact output bytes of every producer, in
several input formattings, including the chained calls where one JSON
function feeds another, and `JSON_EXTRACT`'s dependence on whether it
is evaluated in string or numeric context.
- `func_json_invalid` - input that is not valid JSON, is NULL, is not a
string, or carries bytes JSON cannot hold unescaped; the recorded
observable is the whole trichotomy of value, warning and error code.
- `func_json_depth` - where the nesting limit sits for each container
mix, every function at and one past it, path depth, and the composed
depth reached by splicing an acceptable document into another one.
- `func_json_agg_limits` - `JSON_ARRAYAGG`'s length cut walked byte by
byte across the interesting offsets, for string and for non-string
members, next to `JSON_OBJECTAGG`.
- `func_json_charset` - the character set and collation every function
reports, the same bytes read under different labels, character sets
in which JSON punctuation is a letter, wide character sets, arguments
of mixed character sets meeting in one call, and what `CONVERT` and
`CAST` each make of a value that is already a document.
- `func_json_columns` - a JSON column and its check constraint in every
shape the constraint can take, including what the functions make of a
column whose stored bytes no longer satisfy it, the same column
reached through a subquery and through merged and materialised views,
a user-created temporary table, stores that truncate or convert on the
way in, tables rebuilt by `ALTER`, generated columns, and the
temporary tables the server makes for grouping, derived tables, union
branches and window frames.
- `func_json_sp` - JSON values in stored program variables: how a
variable declared JSON differs from a column declared JSON, what the
declared type does to a value on assignment, and a variable that is
both source and target of the same statement.
- `func_json_table` - what `JSON_TABLE` output columns are treated as
when fed back into JSON functions, a document read from a column whose
stored bytes are not valid, and a `DEFAULT` clause supplying a value
for a column declared JSON.
- `func_json_mislabel` - the same stored bytes read under a different
character set label, reached by relabelling a column through
`VARBINARY`. Converting a column repairs or rejects the bytes first,
so this is the only way to put a document in front of the parser that
is not valid in the character set it claims to be in.
Two of these turn on something easy to get wrong about the shape of the
test itself, so they are built to show it.
A stored program runs under the `sql_mode` it was CREATED with and not
the one in force at the call, so setting the mode around a `CALL`
decides nothing. `func_json_sp` therefore gives each mode a procedure
of its own and calls each under both modes, which puts that invariance
on the record beside the behaviour. The non-strict one truncates an
over-long assignment silently - no error and no warning - leaving
`{"a": 9,` in a `VARCHAR(8)`.
A document spliced into a document of another character set is written
in the character set of the answer, and nothing refuses it.
`func_json_charset` records the two ways that set is chosen: a mutator
takes it from its document argument and converts the fragment into it,
writing a latin1 fragment back out as utf8mb4, while a constructor
aggregates over all of its arguments, so the fragment can decide the
set and its bytes then go across unchanged.
Output that is not printable ASCII is recorded through `HEX()` so the
result files stay readable and diffable. `--echo` text is part of the
recorded output, so the prose in these files is a gated expectation
rather than a comment.
One combination is deliberately left out: `JSON_OVERLAPS` crashes the
server when an argument is an array nested past the scanner limit, so
`func_json_columns` excludes that one row from that one statement, with
a comment saying why. Every other function in the file takes it.
Tests only; no server code is touched.
The JSON functions splice one document into another without reading the result back, so bytes that are not a document, or that mean something else in the character set they land in, pass through in silence. This makes each of those a note, without altering a single answer the server already gives. Every diagnostic added here is a **note**, not a warning. A warning is escalated to an error inside a statement running under strict mode, so a warning would stop statements that finish today. `json_valid()` already reports at this level for the same condition. **What is said now that was not said before** - `JSON_ARRAY`, `JSON_OBJECT`, the aggregates and the mutators scan what they splice, in the character set the result will be read in, and say which argument did not parse. The bytes are still spliced, exactly as they were. - Composed nesting is charged to the argument that caused it: a document that is inside the limit on its own but over it once wrapped is now named, rather than being noticed by whoever reads the result. - A value that cannot be written into a document at all - a character with no JSON spelling - is named where it was previously dropped without a word, and so is a key with the same character in it. The two go through different routines and only one of them used to say anything, so a document could be given up on over a key while the value beside it was named. - `JSON_QUOTE` says it too. It is one of five callers of `st_append_escaped()` and the only one folding that call into a `||` chain, which cannot tell a character it may not write from a buffer that would not grow: on a cp1250 row `JSON_ARRAY(c)` answers NULL naming argument 1 while `JSON_QUOTE(c)` answered NULL with nothing at all. The chain is split so that the two are told apart and the note reported. The answer does not move; only the silence does. **What was wrong, and is now right** - A path key was spliced in the character set of the *path*, into a document in a different one. An ASCII key was enough to make a ucs2 or utf16 document unreadable, so `JSON_SET` and `JSON_INSERT` could not add any key at all to one. The key is converted when the document has room for it, and goes in as it arrived when it has not. Whether the result reads back is then decided where it always was, on the way out. - The same mislabelling made `JSON_REMOVE` match nothing, and so remove nothing, in those documents. - A value spliced by a mutator kept its own character set, because the mutators take the character set of the document and aggregate nothing. A utf8mb4 value landing in a latin1 document arrived as twice as many characters as it was, and latin1 admits every byte, so nothing caught it. A conversion that loses nothing is now made; one that would lose something is not made at all, and the original bytes stay. - `JSON_UNQUOTE` declares `utf8mb4_bin` before it has read anything, then handed the argument straight back in whatever character set it arrived in. Everything downstream read those bytes as utf8mb4. They are converted now, except where converting them would change them, and the conversion is made from the binary character set itself, whose `mb_wc` hands the byte back. That is what makes the two ways out of the function name a byte the same character: `json_unescape()` on the string way out maps each byte to the character of that number, and latin1 - which in MariaDB is cp1252 - answers something else at 27 of the 32 positions in 0x80-0x9F. Read through latin1, `X'228022'` came back `C280` and `X'5B2280225D'` came back carrying the euro sign. - `json_nice()` ignored eight of its own writes, and `JSON_OBJECTAGG` ignored four, either of which lets a function report success over a half written document. **Nothing here returns NULL where it did not before, and no result changes bytes except where one of the conversions above is made.** That conversion is done with `copy_and_convert()` and not with `String::copy()`, which for a binary source does not convert at all: `needs_conversion()` answers false whenever the length divides the destination's `mbminlen`, and utf8mb4's is 1, so the bytes would go across untouched and be relabelled - 0x80 would leave as a lone continuation byte, which is not a character of the set the result says it is in. `st_append_escaped()` now returns which of the two ways it failed, since a buffer that will not grow has already stopped the statement while a character that cannot be written has not, and the two were previously indistinguishable to callers. It also reported success when it failed to reserve room for an empty value. It says so in its return type, as do the two other functions that write escaped text into a document, rather than in an `int` that happens to carry the same values; and the one place that answered with a bare success-or-failure of an append names which of the two it means, that append having failed for the one reason the type has a name for. The parameter block over `append_json_value()` says what the three parameters threaded through it do: the depth is consulted before the type is looked at rather than only for a value typed JSON, the function name and argument number reach both arms - the non-JSON one names the function in the bad-character note - and every function that edits a document passes the depth the path reached, taken off the scanner, never 0. `my_realloc()` gains a debug keyword of its own for failing a buffer that is being grown. The existing keyword fails every allocation including the first, which cannot reach the code that appends to a buffer already held - and that is where the unchecked writes are. Failing a buffer as it grows still cannot reach a write into a buffer that was sized before it was written, and every write that puts a value into a document is one of those - the room is asked for up front, at the widest the answer could come to. Each of those writes therefore names a keyword of its own, so that a test can have that one write report no room while everything else in the statement goes through. Where the debug support is off a keyword is a constant the compiler folds away, and the condition is the one it always was. Four new test files: `func_json_embed` for what is spliced and what is said about it, `func_json_pathkey` and `func_json_unquote` for the two character set faults, and `func_json_oom` for the invariant that a function either errors or returns something that parses, never a half written document. `func_json_embed` section 10 puts the cp1250 fixture through `JSON_QUOTE` beside the constructor that always named the argument, and `func_json_unquote` section 6 reads 0x80 and 0x9F both ways out, plus 0x81, one of the five positions where the two readings agree anyway. `func_json_oom` carries both instruments. Its first sections fail the buffer as it grows; the ones after them fail one write at a time - the quotes round a value, the room the escaping asks for, the quote that opens a key, the two quotes `JSON_QUOTE` is made of, both roads by which `JSON_UNQUOTE` hands an argument back, a value converted into the character set of the document it is joining, and a path key converted the same way. The recorded baselines that move do so by gaining notes, except where one of the conversions above is made. `func_json_charset` gains values where a mutator could not add any key to a ucs2 document at all, and moves one where a constructor spliced a ucs2 fragment with its bytes relabelled rather than converted; `json_no_table` moves the one line where `JSON_UNQUOTE` is handed the bytes of a geometry value. No other recorded value moves.
`JSON_OBJECTAGG` wrote its opening brace in the constructor, which runs
while the expression is still being parsed - before `fix_fields()` has
resolved the character set the result will be in. The brace therefore
went down as a single byte. The closing brace is written when the
result is asked for, by which time the character set is settled, so that
one went down at full width.
In `ucs2`, `utf16` and `utf32` the two ends did not match and everything
between them sat a byte out of step. `JSON_VALID()` answered 0, and
converting the result to `utf8mb4` raised
Warning 1300 Invalid ucs2 character string: '}'
which is the server refusing to read a document it had just written.
Two further places measured that brace rather than writing it, and both
are right only where a character is one byte: `clear()` kept byte 0 with
`result.length(1)`, and `add()` decided whether a pair needed a
separator in front of it by asking whether `result.length() > 1`.
The brace is now written in `clear()`, once per group, where the result
already carries the character set it will be read in, so
`String::append()` puts it down at the width that character set asks for
and the two ends agree. `add()` asks `null_value` whether a pair has
been written already instead of measuring the buffer, and `cleanup()`
empties the buffer rather than keeping one byte of what may be a wider
character.
A result computed where one byte is one character does not move,
`latin1` and `utf8mb4` included.
New test `func_json_agg_charset`, covering both aggregates because they
are asked the same question and answer it differently. Section 7 is
this change, recorded before it: the three wide character sets; one
pair, where no separator is ever written, and several, where one is; a
group each, so that the opening brace is written more than once; and a
`WITH ROLLUP` copy, which is made without fixing the item again and so
has to settle its own character set.
Sections 1 to 6 are the sibling aggregate, which is in the same
difficulty for an unrelated reason and is recorded here as it stands.
`swe7` is the vehicle for them: having no code point for the JSON
punctuation, it keeps every recorded byte ASCII while still showing
which results were converted on the way out and which were not. A later
commit puts that half right.
The scanner refuses to open a structure nested deeper than it can keep
track of, and records that it has, but it moved its nesting counter
before working out whether there was room. A refusal therefore left the
counter one past the end of the stack it indexes.
`stack_p` is declared immediately after `stack`, and `stack` is exactly
`JSON_DEPTH_LIMIT` long, so `j->stack[j->stack_p]` at that point reads
the counter itself and yields `JSON_DEPTH_LIMIT` as though it were a
scanner state. `json_scan_next()` then indexes a table of
`NR_JSON_STATES` rows with it and calls what it finds there:
json_actions[32][t_next](j) /* the table has 10 rows */
Getting that far needs a caller that keeps scanning after being told the
document is too deep, and `json_compare_arr_and_obj()` is one:
`json_skip_level()`'s result is not looked at, and the loop around it
asks only whether the next scan step succeeded. So
SELECT JSON_OVERLAPS(CONCAT(REPEAT('[',32),'1',REPEAT(']',32)),
'{"a":1}');
killed the server, in either argument position and whatever the other
argument held.
The counter is now moved only once the room for it is known to be there,
so a refusal leaves it where it was and every read of
`stack[stack_p]` stays inside the stack. Making that a property of the
scanner rather than of its callers matters here: six further places in
this family also skip levels without looking at the result, and each of
them was one ignored refusal away from the same ending.
**A walk that has been refused is over, and stays refused.** That is
the other half of making it the scanner's property, and three things
rest on it.
Every error is reported by subtracting the scanner's read pointer from
the start of the document, and `json_scan_next()` moves that pointer
before it looks at anything, so each further step taken after a refusal
moves the place the refusal will be said to have happened. For one
document and four different second arguments:
SELECT JSON_OVERLAPS(@d,'{"a":1}'), JSON_OVERLAPS(@d,'"s"'),
JSON_OVERLAPS(@d,'[[1]]'), JSON_OVERLAPS(@d,'[1,2]');
at position 34
at position 34
at position 34
at position 33
while `JSON_DEPTH(@d)` says 32 for the same bytes, and so do
`JSON_CONTAINS`, `JSON_EQUALS` and `JSON_TYPE` four lines away in the
same recorded file. A failed engine no longer moves, so the pointer
stays where the refusal was made.
Both comparison walks remember where an engine stood and put it back to
try the next candidate, and the snapshot predates the failure, so a
restore rolled the error away with it. `json_resume_scan()` states the
rule once and the five restore sites go through it; without it
`JSON_OVERLAPS('[1,2]', <32 deep>)` answers 0 and says nothing at all.
`compare_nested_object()` measured its two fragments from where the skip
stopped and normalized them. After a refusal that is a piece of a
document, so normalizing found it unterminated and said so, and
SELECT JSON_OVERLAPS(CONCAT('[',REPEAT('{"a":',31),'1',
REPEAT('}',31),']'), '{"a":1}');
drew `4037 Unexpected end of JSON text` for a document that is complete
and balanced and whose only fault is depth - beside the pure-object case
one screen above, which says 4040 and where. The refusal is honoured
instead, and the engine's own reason survives to the caller: 4040 at
position 152, which is what `JSON_DEPTH` says of the same bytes.
`JSON_OVERLAPS` now answers 0 with warning 4040 for an over-deep
argument, which is what it has always answered when the same depth is
reached through objects instead of arrays - the crash was particular to
arrays. `Item_func_json_overlaps::val_bool()` was already reporting the
engine's error once the comparison returned; it simply never got to.
Present since `JSON_OVERLAPS` was added, so 10.9 onwards.
**Not one answer moves; what moves is where a complaint is placed, and
whether it is made at all.** Against a released server:
JSON_OVERLAPS('{"b":9}', '{"a":1 "b":2}') 9 -> 8
JSON_OVERLAPS('{"z":0}', '{"a":1, "b":2 "c":3}') 16 -> 15
JSON_OVERLAPS('{"a":{"p":1}}', '{"a":{"p":1 "q":2}}')
15 -> 13
JSON_OVERLAPS('[{"z":9},{"b":2 "c":3}]', '[{"a":1}]')
18 -> 17
JSON_OVERLAPS('[9]', '[1,2,3 4]') silent -> 8
JSON_OVERLAPS('[{"a":1}]', '[{"z":9},{"b":2 "c":3}]')
silent -> 17
The new positions are the right ones, and the tests say so by something
other than themselves: `JSON_VALID` and `JSON_EXTRACT` read the same
bytes with no comparison around them, and they give 8, 15, 13, 17 and 8
for these documents. Both readings are recorded side by side, so a
later change that moves one without the other fails there.
Nothing in the suite stood on any of this. The only `json_overlaps`
syntax-error warning recorded anywhere is a whole document rejected at
position 1, which never reaches the walk at all. The two silent cases
could have come back in silence and the four positions could have
drifted with nothing to say so.
Two new tests. `func_json_overlaps_depth`: both argument positions, the
other argument of each kind, both arguments too deep, the same depth
reached through objects and through a mix of the two, one container
inside the limit where the comparison does run, what the neighbouring
comparisons answer on the same input, and a table of such rows so that
one engine is reused across evaluations; its section 6 puts one first
argument against four second arguments in a single statement, so that
four positions stand in one block and any drift between them is visible,
with `JSON_DEPTH` of the same value beneath it, and the mixed chain in
section 2 gains the same anchor. `func_json_overlaps_diag` takes the
break in either argument, at the top level and below it, on the road
where every candidate is tried and put back, and out of a table in both
argument orders so no one statement settles another; whole documents and
a document refused before the walk begins stand beside them, unmoved.
Three recorded lines move outside the new tests, all of them a
diagnostic position and none of them an answer. Each moves onto the
number the same file already records for the same bytes:
`func_json_columns`, 34 to 32; and `func_json_depth`, `JSON_KEYS` 160 to
156 where `JSON_DEPTH(@o32)` in that file says 156, and `JSON_OVERLAPS`
33 to 32 where `JSON_DEPTH(@D32)` says 32.
The recorded baseline `func_json_columns` no longer holds its over-deep
row back from the `JSON_OVERLAPS` statement. The comment there said the
row was excluded until this was fixed; the row now answers like every
other function in that statement.
`JSON_ARRAYAGG` puts the brackets around its elements by building them
in a scratch `String` and exchanging that with the result. The scratch
one was left at its default character set, which is `my_charset_bin`,
and `String::swap()` carries the character set across along with the
bytes, so the result was handed back saying it was bytes rather than
text.
That cost twice over:
1. **The brackets were written one byte wide** whatever width a
character of the result takes. In `ucs2`, `utf16` and `utf32` that
left everything between them a byte out of step and the result read
as an entirely different set of characters: `JSON_VALID()` answered
0, and one recorded value came back as six CJK ideographs in place of
`[["x"]]`. Only the two brackets were written this way; the elements
between them, and the separators, always had the result's own
character set.
2. **The result was not converted on its way to the client.**
`Protocol::needs_conversion()` reads a binary label as needing no
conversion, so `JSON_ARRAYAGG` alone among these functions ignored
`character_set_results`. Under `character_set_results = swe7`, which
has no code point for a bracket, every other JSON function arrived
with those characters replaced and this one arrived untouched.
One line settles both: the scratch buffer is given the character set the
expression was resolved to, so the exchange hands that to the result,
and `String::append(char)` writes a bracket at the width that character
set asks for.
A result computed in `latin1` or `utf8mb4` is unchanged to the byte.
Stored copies are unaffected either way, `Item::save_str_in_field()`
passing the item's declared collation to `Field::store()` rather than
the returned value's own label.
`func_json_agg_charset` was written one commit earlier and records this
half, so the change shows there as a move rather than as a new file.
Its header states the two questions the file asks: the delivery
conversion, which `swe7` makes visible because it has no punctuation to
convert into and `latin1` hides because it has, and the character set
the result is computed in, which the later sections vary while leaving
the client alone. Sections 1 and 5 stop arriving unconverted under
`character_set_results = swe7`; section 6 puts the brackets at full
width and `JSON_VALID()` answers 1 where it answered 0. There is no
unconverted delivery left anywhere in the file: what section 3 records
under `latin1` is both aggregates arriving intact, which is the
conversion having happened and having had nothing to change.
One statement is added there, reading the `ucs2` result back as
`utf8mb4`. It could not be recorded before the change: the value being
handed over unconverted, that statement answered with the raw bytes and
the NULs between them, which is not text and cannot be written down in a
result file.
Two recorded baselines move as well, both repairs:
- `func_json_charset`, one line: the delivery is now converted.
- `func_json_embed`, seven lines: `ucs2` brackets now at full width.
One answer changes for a result the expression resolves to `swe7`, and
that one is not a repair of a spelling but a different answer to a
question:
SELECT JSON_VALID(JSON_ARRAYAGG(CONVERT(a USING swe7))) FROM t1;
Before, 1; now, 0. The bytes are the same either way,
`5B226161222C226262222C226363222C226464225D`. What moved is the label
they are read under: they used to arrive as `binary`, where a byte is a
character and `5B` is a bracket, and they now arrive as `swe7`, where
`5B` is a national letter. Neither `swe7` nor any other character set
lacking the JSON punctuation can hold a document, so 0 is the answer
that matches what is there; the 1 was an artifact of the wrong label,
and removing that label is this change.
The sibling aggregate already answered 0 for the same reason and does
not move. Section 8 of `func_json_agg_charset` records both, `swe7`
being what the expression resolves to there rather than what the client
asked for, which is what every earlier section uses it for.
Both JSON aggregates put the last of their punctuation on in the call
that hands the result over. Nothing says how often a statement asks for
the result of a group, and the buffer being written into belongs to the
item and outlives the asking, so a statement that asks twice gets a
value closed twice.
`JSON_OBJECTAGG` returns a document with an extra closing brace, which
is not JSON at all. `JSON_ARRAYAGG` returns one wrapped in an extra
pair of brackets, which is valid JSON of the wrong value - the quieter
of the two, since nothing downstream has any reason to reject it.
Neither raises a warning or an error.
`HAVING` on the alias is the shortest statement that asks twice, and the
count follows the number of conditions:
SELECT JSON_OBJECTAGG(a,a) AS v FROM t1 HAVING v LIKE '%';
{"x":"x", "y":"y"}}
SELECT JSON_OBJECTAGG(a,a) AS v FROM t1
HAVING v LIKE '%' AND v LIKE '{%';
{"x":"x", "y":"y"}}}
Every group of a `GROUP BY` is affected, not only the first.
The parent of one of the two has the same exposure and guards against
it: `GROUP_CONCAT` keeps a flag saying the result has been produced, and
a second call hands back the buffer without rebuilding it. That flag
protects the elements, which is why the elements are not duplicated
above. It does not reach the brackets or the brace, those being written
by the derived classes after the parent has returned.
Each aggregate now keeps the same kind of flag for the punctuation it
writes itself, reset per group in `clear()` next to the flags already
there. `JSON_OBJECTAGG` skips the closing brace when the result is
already closed; `JSON_ARRAYAGG` hands back what the parent gave it,
already inside its brackets.
Both flags are initialised in the copy constructor as well as in the
first one, for the symmetry rather than for a state a reader could
reach: the first row of every group - the copy `WITH ROLLUP` evaluates
included - goes through `Item_sum::reset_and_add()`, which calls
`clear()`, and both `clear()` bodies set the flag before anything looks
at it. Given `m_closed(true)` in both copy constructors, which is the
value that would spoil the answer if it were ever read, every recorded
answer stays where it is.
New test `func_json_agg_reread`: asked once, twice and three times;
every group of a `GROUP BY`; a group of one row, where no separator is
ever written; the `WITH ROLLUP` copy; and a group with no rows, which is
a NULL and stays one however often it is asked for.
The rollup section asks through `HAVING` like the rest of the file,
because the condition is what makes it a test of the copy rather than of
the copy being made. With no condition each level is asked exactly once
and is closed once whether the flag is kept or not, and those statements
answer `{"x":"x", "y":"y"}` and `["x","y"]` at every level against a
server carrying neither guard. With `HAVING v LIKE '%'` on them that
server stops at the first level, on the self-check that says a value
marked valid has to read as a document. The two-condition form asks a
third time, and the unconditional pair stays at the end, where it is the
statement that asking once must not move either.
Both defects date from the commit that added the two aggregates,
`ba8e5e689c8` (2019-10-14), first released in 10.5.0. No recorded
result in the suite moves, no test there having asked for either
aggregate more than once in a statement, which is why neither has shown
up.
`Item_copy::copied_in` says whether `copy()` has run, and twenty `DBUG_ASSERT`s across the copy classes read it before trusting the cached value. Nothing else reads it, and it is declared - and the three places that write it are guarded - `#ifndef DBUG_OFF`. That guard names the wrong set. `DBUG_ASSERT_AS_PRINTF` turns an assertion into a printed complaint with a stack trace instead of an abort, and it does so by setting `DBUG_OFF` while leaving the assertion's expression standing (`include/my_dbug.h`). So in that shape the twenty reads survive, the member does not, and the build stops on all twenty. The server otherwise supports the shape: `mysqld.cc` installs `mariadb_dbug_assert_failed` for it. `DBUG_ASSERT_EXISTS` is defined in both branches that leave an assertion's expression to be compiled, so it names exactly the set that reads this member. The five sites - the declaration, the clearing in `Item_copy::Item_copy()`, and the setting in `Item_copy_string::copy()`, `Item_copy_real::copy()` and `Item_copy_timestamp::copy()` - move to it together, since a member kept without the writes that maintain it would be read unset. No shipped binary changes: `DBUG_ASSERT_EXISTS` is defined in every build that defines `DBUG_OFF` alone, and a build defining neither is a debug build, which already had all five.
Every JSON function that is handed a document reads it to find out
whether it is one, and reads it again if it is spliced into something
else. The function that produced it knew the answer and had nowhere to
put it: `val_str()` hands back a pointer, a length and a character set,
and there is no room in that for anything else.
This adds the room. Nothing reads it yet - no scan is skipped here and
no answer moves - so what this commit is, on its own, is a set of
promises and a way of catching one that is not kept.
That way of catching is two functions read by a `DBUG_ASSERT` and by
nothing else, so they are compiled wherever a `DBUG_ASSERT` is rather
than wherever a debug build is. Those are not the same set:
`DBUG_ASSERT_AS_PRINTF` turns the assertion into a printed complaint
and leaves its expression standing, while setting `DBUG_OFF`, so a
guard written the other way round leaves that expression with nothing
to call and the build stops compiling.
**The two questions**
`Item::is_valid_json()` asks whether the value just handed back is a
document. `Item::is_nice_json()` asks whether it is spelled the way
`json_nice()` spells one in its loose form, which is the spelling these
functions produce. Both are about the last evaluation and stand until
the item is evaluated again. `Item::is_valid_json_static()` asks
instead whether *every* evaluation gives back a document or nothing at
all - a property of the class, so it can be asked before anything has
run, which is what a caller working at create time needs.
All three answer FALSE by default, and FALSE does not say the value is
not a document. It says nothing answers for it, so a caller who needs
to know must read it - which is what every caller does today. The
asymmetry is the whole design: answering TRUE for a value that is not
guaranteed is the one answer that must never be given, while answering
FALSE for one that is costs a reading and nothing else. Where the
cheap answer was not obviously safe, FALSE was chosen.
**Catching a promise that is not kept**
The marks appear in no result, so a broken one would go unnoticed until
something acted on it. They are therefore not two bare bools but a
`Json_result_marks`, whose setter is handed the value being answered
for, and in a debug build reads it back:
DBUG_ASSERT(!valid || json_value_reads_as_document(str));
DBUG_ASSERT(!nice || json_value_is_nice(str));
Whether a value reads as a document is the question `json_valid()` is,
so the first of those asks it rather than asking it again in a loop of
its own. The second runs `json_nice()` in its loose form and compares
the two runs of bytes with the one function there is for comparing two.
This turns every JSON statement in the existing suite into a detector
for an overclaim, and it earned that during development: it caught a
character set in which a document cannot be written at all, it caught
`JSON_ARRAY` calling a value nicely spelled that was not a document, and
it caught the path below that is written in a wide set and yet carries
one part of itself in single bytes. Each was proven to fire by
deliberately introducing a false claim and watching the server stop.
The class lives in `sql/item.h` rather than with the JSON functions
because a class that is not one of them can carry a value through - see
`Item_func_conv_charset` below - and going through this member is what
puts such a class under that check. A copy of it says nothing: a copy
is a new item that has not been evaluated, and what it was copied from
speaks about bytes that it still owns and may write over.
**What each function says**
Six editing functions - `JSON_INSERT`/`SET`/`REPLACE`, `JSON_REMOVE`,
`JSON_ARRAY_APPEND`, `JSON_ARRAY_INSERT`, `JSON_MERGE` and
`JSON_MERGE_PATCH` - answer the static question, their epilogue reading
the whole result before handing it over. So do `JSON_EXTRACT`,
`JSON_QUERY`, `JSON_SEARCH` and `JSON_NORMALIZE`, each of which hands
back a span cut from something already read through. Of those,
`JSON_NORMALIZE` is not nicely spelled - it emits the compact form - and
neither is `JSON_QUERY`, which keeps the spacing it was given.
`JSON_ARRAY`, `JSON_OBJECT` and the two aggregates build their result
rather than reading it, so they answer per evaluation, from what they
learned putting it together. A `Json_splice_marks` is carried through
the splice helpers for that: it starts out saying both things and is
only ever cleared, a conjunction over the values that went in, since an
array is a document only if every element was. A value that did not
parse, one holding a character that cannot be written, and one that came
in by a route with no opinion about spelling each clear what they have
to. Niceness is asked of the item rather than measured.
An aggregate says nothing about a group cut to `group_concat_max_len`.
The cut lands on whatever byte the limit falls on, so what stands
between the brackets can be half an element - or, where the cut is just
past an opening quote and that quote is written back to close the
string, an element no row of the group ever held, which reads as data
and is not any. Nor does it say the group is nicely spelled: what goes
between the elements is the separator inherited from `GROUP_CONCAT`, a
comma with nothing after it where the loose spelling puts a space as
well, and there is no asking for another one - the grammar takes no
`SEPARATOR` in this production and writes that comma itself.
`JSON_FORMAT` forwards: its `val_str()` answers `(true, fmt == LOOSE)`,
and its `val_json()` answers exactly what its argument answers.
Everything else says nothing, `JSON_KEYS` and `JSON_QUOTE` included.
**Values that arrive by some other route**
`Item_func_conv_charset` is not a JSON function but carries one's result
through. A conversion that substituted nothing and stopped on nothing
kept the sequence of characters it was given, and JSON is a grammar over
characters, so a document that went in comes out one, spelled the same
way. It therefore passes the argument's answer on. Conversion to
`binary` is refused: it is not a conversion but a relabelling, keeping
the bytes and making each one a character of its own, so a document two
bytes to the character comes out as twice as many characters and is not
one. The copier cannot report that, `binary` having no byte it rejects.
`Item_direct_ref` forwards all three, its `val_str()` handing back what
the referenced item returned byte for byte. `Item_ref` does not: it
reads through `str_result()`, which for a field is a copy sitting in a
record buffer rather than the value the producing item made.
`Item_aggregate_ref` forwards for a reason of its own, given at the
site. `Item_direct_view_ref` declines on a row an outer join filled in
with NULLs, where its `val_XXX` return without evaluating the referenced
item at all and what that item can say is about some earlier row.
**A document cannot be written in every character set**
A function that writes its own punctuation writes it into the result's
character set, and `String::append(char)` puts a raw byte where a
character can be a single byte. In swe7 the bytes that spell brackets
and braces elsewhere spell national letters, so what is written is not a
document and cannot be read as one - which is what the server has always
handed back, and is not for this to change. What it is for is to keep
that result from being taken for a document later, so
`json_charset_can_spell()` gates the claim wherever punctuation is
written without the result being read back. It admits ucs2, utf16 and
utf32, which are wide enough that `String::append()` converts and the
bracket goes down as a bracket.
That the set can spell a document is not the same as the value having
been spelled in it, and `JSON_SEARCH` is where the two part company. It
builds a path out of punctuation it writes, keys it copies from the
document, and the index of an array step. The punctuation goes in
through `String::append()`, which converts it, so in `ucs2` a bracket is
`005B`; the index does not go that way, `String::append_ulonglong()`
writing the digits as themselves whatever the set. One stray byte
leaves every character after it read from the wrong place - a path 39
bytes long where every character is meant to be two or four. So the
path says for itself whether it was spelled, and one that was not is
handed on as the bytes it is rather than as a document. Both shapes in
one statement, over a table:
id spliced
1 0 -- the path carries an index
2 1 -- the path is keys all the way down
Released 10.11 answers 0 to both, refusing the second wrongly; that half
is this branch's.
Two constants, `json_loose_comma` and `json_loose_colon`, are hoisted to
file scope and used both by `json_nice()` and by the functions that
write that spelling themselves, so a claim of niceness rests on a shared
fact rather than on two pieces of code happening to agree.
`JSON_SEARCH` writes the separator between several paths from that
constant for the same reason: it claims the loose spelling over a result
it never reads back, so the claim has to rest on the one fact that
spelling is defined by. With the constant temporarily made comma-tab,
`JSON_SEARCH('{"a":"x","b":"x"}', 'all', 'x')` answers its two paths
separated by a comma and a tab; with the separator written out by hand
instead, a debug build stops on the niceness half of the self-check.
**Nothing moves**
No existing recorded result changes, which is this commit's gate and is
checkable with `git status`. The new `func_json_marks` proves the same
for statements that did not exist before: it was recorded on this build,
then `sql/` was restored to the previous commit, rebuilt, and
re-recorded, and the two files are identical byte for byte.
That test walks every function past the point where it makes a promise,
in a build that reads the value back. Its sections are: building from
good values; building from a column holding text that is not a document;
the six editing functions; the reading functions; the two aggregates,
including a group cut to fit the length limit, `WITH ROLLUP`, and a
broken column; character sets, including swe7, sjis and the three wide
ones; values arriving through a conversion, including the binary
relabelling; values arriving through a reference, including an outer
join over a merged view; and items that are copied, by condition
pushdown and by copying a table definition that holds a JSON expression.
The cut group recorded there is `["a","b",""]`, which is not half of
anything: the limit fell just past an opening quote and the quote
written back to close the string left a third element that no row of
the group ever held - valid JSON, so nothing downstream has any reason
to reject it, and one row of it invented. Warning 1260 says a row was
cut, not that one was manufactured. The group built out of a broken
column alongside it is complete too, and is not a document at all.
`func_json_search_wide_charset` carries the path shapes: two, four and
one byte to the character, an index path and a keys-only path, read back
and spliced into `JSON_ARRAY` and `JSON_MERGE_PRESERVE`, and out of a
table so no one statement settles another.
In a character set that spends more than one byte on a character,
`JSON_EXTRACT()` returned NULL for documents it had just read
successfully, and complained about a character the document did not
contain:
SET collation_connection='utf16_bin';
SELECT JSON_EXTRACT('{"a":1,"b":2}','$.a');
NULL
Warning 4036 Character disallowed in JSON in argument 1 to
function 'json_extract' at position 2
`JSON_ARRAY_APPEND()` did the same, but only when the path led to
something that was not an array and had to be wrapped in one first.
Appending to an array that was already there worked - the same
function, in the same statement, in the same character set.
**What was happening**
`String::append(const char *, size_t)` converts what it is given from
`latin1` whenever the string being appended to is one a character
cannot fit into a single byte. That is right for text that really is
`latin1`, and it is what puts a proper wide bracket into a `ucs2`
result when a bracket is written. It is wrong for a run of bytes cut
out of the document, because those bytes are already in the document's
character set, which is the one the result is being built in. They
were being encoded a second time.
The single character `1` is `0031` in `utf16`. Its two bytes were
read as the `latin1` characters U+0000 and U+0031 and written out
again as `00000031` - a NUL, and then the digit. Both functions read
their whole result back before returning it, met the NUL, and gave up.
So what the caller saw was not the corruption but a complaint naming
their own argument and a position inside it. The character it named
had been put there by the server a few lines earlier, which is why
this survived being looked at: MDEV-34143 rewrote the loop around the
`JSON_EXTRACT` call while fixing a crash in `chop()` under exactly
these character sets, carried the call across unchanged, and recorded
the NULL as the expected answer.
**What is done instead**
The three places that copy a piece of document now say so, with
`append_simple()`, which puts the bytes down as they stand. The
punctuation these functions write themselves - brackets, commas - is
still appended the converting way, because it is written rather than
copied and does have to be converted. `JSON_ARRAY_APPEND`'s other
arm has always copied its two pieces of document correctly, with
`q_append()`; the two arms now agree.
Nothing changes in a character set of one byte, where the conversion
was a copy anyway.
**Not covered here**
`Item_func_json_object_to_array`, which does not exist on this branch,
copies both the key and the value of every pair the same way and wants
the same treatment where it exists.
The recorded answer from MDEV-34143 moves from NULL to the value, and
`func_json_wide_charset` covers reading and editing at two and at four
bytes to the character, keeps a one-byte character set alongside as a
control, and checks that a document which is genuinely malformed is
still refused and still has the fault located in what the caller sent.
`JSON_MERGE_PATCH()` returned NULL, and blamed an argument holding no
JSON text at all, for a statement with nothing wrong in it:
SELECT JSON_MERGE_PATCH(NULL, LOWER('[1]'),
LOWER('{"AAAAAAAAAAAAAAAA":1}'));
NULL
Warning 4038 Syntax error in JSON text in argument 1 to
function 'json_merge_patch' at position 1
Argument 1 is the four characters `NULL`. Writing the same two
documents out as literals instead of building them gave the answer
they ask for, `{"aaaaaaaaaaaaaaaa": 1}`.
**What was happening**
When everything before it was NULL, the function takes a whole
document over rather than merging into one - unless it is an object,
which is merged. It took it over by pointing at the buffer it came
in, and that buffer is the one every argument is read into. The next
argument was then read into it, a buffer that has to grow to hold one
is not the buffer it was, and the document taken over was read from
where it used to be. It had been released by then.
Which is why the spelling of the argument decided the answer.
Whether an argument is read into the caller's buffer at all is a
property of the function that produced it: `LOWER`, `UPPER`, `CONCAT`
and `REVERSE` build an answer and write into it; `TRIM` and
`SUBSTRING` hand back a piece of what they were given and leave it
alone; a plain column reference and a literal never touch it. So the
function was right or wrong according to how its arguments happened
to be written, and every test in the tree writes them as literals.
**What is done instead**
The document is copied. A copy that cannot be made is a failure like
any other and takes the failure path.
`func_json_merge_patch_adopt` covers taking over a document of each
kind that is not an object, arguments written both ways, room that has
to be found rather than grown into, and further arguments after the
one taken over. Two controls sit alongside: the same second argument
reached without a take-over, and a second argument no longer than the
first.
`JSON_MERGE()` and `JSON_MERGE_PATCH()` took a document argument
holding text that is no part of any document, and said nothing:
SELECT JSON_MERGE_PATCH('{"a":1} rubbish', '{"b":2}');
{"a": 1, "b": 2}
`JSON_VALID()` returns 0 about those same characters, and
`JSON_INSERT()`, `JSON_REMOVE()`, `JSON_EXTRACT()` and `JSON_TYPE()`
all return NULL with a warning. So one column value was a document
to two functions and not a document to the rest, and nothing said
which of the two answers was being given. What stands there can be
a whole second document, dropped just as quietly:
SELECT JSON_MERGE_PATCH('{"a":1} {"zzz":9}', '{"b":2}');
{"a": 1, "b": 2}
**Why these two and not the others**
Both read a document argument only as far as its first value, and
compose the answer out of that value alone. The other functions
that edit a document copy the rest of it into the result untouched,
so the text lands there and the reading back at the end meets it.
These two never copy it, so there is nothing for that reading to
meet: it reads what was composed, and what was composed holds only
what was scanned.
**What is done instead**
The reading is carried on to the end of the argument, and what it
finds is said. It is carried on from where the merging left off
rather than started again, so an argument with nothing after its
value pays one step and no more, and it is carried on over a copy of
the engine, so that a reading done to find out whether to say
something cannot become the reason anything else is said.
It is not carried on at all where the argument has already answered
for itself. Being a document is a statement about the whole of a
value, so one with anything standing after it is not one: where
`is_valid_json()` is true - the same question this code asks a few
lines above, to decide whether to splice - the reading could only come
back saying there is nothing there. The taken-over argument is the
one that would otherwise pay a full walk of a whole document on every
row, that walk being the only reading this path has ever made.
Said before it is settled whether the argument goes into the answer at
all. Every shape of argument merged onto SQL NULL is taken over,
except an object, which contributes nothing and is dropped; being
dropped makes what stands after it no more a document than it was, and
the very same characters are spoken for in every other argument
position. So the sentence stands above the early return that handles
the object and both arms reach it, while the marks stay below it, they
being about a document that was taken over and an object is not.
A note, not a warning. A warning becomes an error under a strict
mode and would take away an answer that has always been given back.
`JSON_VALID()` already says the same thing the same way, and for the
same reason.
Only an argument can be named. After the first turn of the loop the
document on the left is what the function itself composed, so the
left is only spoken about while it is still argument 1. What comes
of that is a position that is a real offset into the text the caller
wrote, and it is the position every other function names for the same
bytes: `JSON_EXTRACT` says 8 for `{"a":1 "b":2}`, and so does the
merge.
The block over the reporting helper says both of the things it is for:
text standing after a value, and an argument the merging broke off
inside, the merging composing out of what it had and handing the
engine over still carrying the refusal.
**What is deliberately left alone**
The key loops still swallow an error their condition hides, so a
broken argument does not fail the merge. A released server answers
SELECT JSON_MERGE('{}','{"a":1 "b":2}');
{"a": 1}
and failing the merge would answer NULL instead, which is the one
transition nothing here may make. Measured against a released server,
every answer and every mark given for a broken merge argument is
already identical to it; the notes are the whole of the difference.
`func_json_merge_trailing` covers text after the value of either
argument and of both, three arguments with the middle one carrying
it, a second whole document standing there, and the taken-over case
in each of its two outcomes. The object arm has all its shapes -
merged over, empty, followed by a whole document of its own, twice in
one statement, and as the last word; without the sentence above the
early return all five of those answer with no diagnostic at all, and
with it each names its argument and its position, the two-argument one
naming both 2 and 3. A section compares positions with
`JSON_EXTRACT`, pinning that agreement rather than proving it, and
pins the shapes that simply stop as well. Alongside them sit the
controls that have to stay silent: trailing whitespace, well formed
arguments, arguments that are not documents from their first
character, and NULL. One more holds the note to being a note - the
same statement inside a `STRICT_ALL_TABLES` insert, which says it and
still stores the row.
`func_json_columns` gains the two notes its row 12 has always had
coming.
Seven JSON functions ended by reading their own answer back:
`json_scan_start()` followed by `json_nice(&je, str, LOOSE)`. That
epilogue does **two** jobs - it spells the answer the loose way, and it
refuses it if what was composed does not read as a document. Neither
job is free: the whole answer is walked a second time, every time.
The second walk is only worth its cost when something about the
composition is unknown. This drops it where nothing is:
1. **The document was a document.** Every function that edits one
copies the parts it is not changing out of an argument that answered
`is_valid`, at offsets its own walk computed rather than guessed.
2. **The document was already spelled that way.** The argument answered
`is_nice`, and the punctuation written here was spelled to match,
`colon_len` being settled before anything was composed.
3. **The punctuation can be spelled at all.** A character set that puts
national letters at the ASCII code points writes a bracket that is
not a bracket, so `json_charset_can_spell()` sends those answers
through the reading back exactly as before.
Where any of the three fails, the epilogue stays where it was and the
answer is byte for byte what it has always been.
The three are one question and are asked in one place, rather than
written out again at each of the five functions that ask them. What
each of them then DOES with the answer stays where it is; that part is
about the function and not about the argument. Patching a document
asks the question too: the colon it writes is three characters where
the answer is handed back and two where it is read again, so the width
follows from the answer rather than standing in for it.
`Item` gained two per-evaluation questions, `is_valid_json()` and
`is_nice_json()`, each true or false and asked after the argument has
been worked out. They are separate because a value can be a document
and still not be written the loose way, which is the ordinary case for
anything cut out of a document that was written some other way.
`Item_field` overrides neither, so a column is never `is_valid`: the
bytes in a row are whatever was put there.
`Json_splice_marks` carries the same two for the values spliced in, and
a function hands its answer over only when the document, the punctuation
and every splice all say yes.
Handing it over is one method the six functions that do it call, rather
than the same seven lines written out six times. What they differ in
is where they report a buffer that would not grow - some have a
document engine to complain through and some do not - so the failure is
handed back to them. The debug hook that stands in for that failure is
one site as well, reached from whichever of the six is running.
A function that claims the loose spelling over an answer it does not
read back has to write that spelling from the fact it is defined by, and
not from a literal that happens to agree. `JSON_EXTRACT` writes both of
its separators from `json_loose_comma` for that reason - between two
values, and between the repeats of a value that answered more than one
of the paths asked for. Nothing is wrong until the two disagree, so it
is measured that way: with the constant temporarily made comma-tab, both
`JSON_EXTRACT('{"a":1,"b":2}', '$.a', '$.b')` and the same document read
through `'$.a'` twice answer with a comma and a tab between, where a
hand-written separator stops a debug build on the niceness half of the
self-check.
The loose form is written in one place. Reading an answer back to
settle its spelling walked it a second time through a copy of the same
state machine, and the two were required to go on agreeing byte for
byte with nothing making them do it: what says whether a value is
spelled the loose way is a check that measures it by writing it
through one of the two, so the day they drifted apart, the drift is
what would have been approved.
How deep a walk over a value went is wanted by every caller there is,
so the walk takes somewhere to put that answer rather than a pointer
that may be nothing - and the tests against nothing, made in the
middle of the walk, go with it.
Nothing is taken on faith. `Json_result_marks::set()` reads the answer
back in a **debug** build and stops the server if `is_valid` was claimed
over something that does not read as a document, or `is_nice` over
something that is not spelled that way. It is the same reading, kept
exactly where it is worth its cost and nowhere else. It found eleven
defects while this was being written.
That reading is held off the kill a test arranges partway through a
walk. A debug build makes one walk more than a release build - the
one that writes a value out again to find out whether it was already
written that way - and a kill raised in THAT walk would be a debug
build ending a statement a release build finishes. The same reading
is held off the count of readings, and for the same reason.
Where the reading back does still happen, what it reads is text this
function composed, and a position it complains about is an offset into
that text. So the text has to be composed the way a released server
composed it, character for character, even in the parts the reading is
about to write over.
`JSON_REMOVE` is the one that turns on this. It writes its single comma
at the width a released server writes it, and the space standing behind
that comma is asked the same question: skipped on both roads, the text
read back is one character shorter than a released server's and every
position complained about in it moves. Nothing that comes out of the
function moves either way - the reading back writes the answer afresh
and drops the space itself, which is how a released server comes to hand
back `[2]` while composing `[ 2]`. It shows only where that reading
fails, which is where the position is all there is:
SELECT JSON_REMOVE('[1, 2] x', '$[0]');
names position 6, which is the position a released server names.
A composition can be deeper than a document is allowed to be even when
both sides of it fit, so each site now accounts for the levels it adds:
- reading a container **pushes** the scanner's stack, so
`JSON_ARRAY_APPEND` and `JSON_ARRAY_INSERT` were counting the target
twice, and refused documents that fit;
- an array or object is **popped before** the state that says it ended
is set, so `JSON_INSERT` and `JSON_SET` were counting one level too
few at the two arms that reach the end without finding the path;
- where a level is added over text that was copied rather than read -
the bracket `JSON_EXTRACT` puts round more than one match, the array a
merge wraps a conflict in, the array an autowrap keeps a value inside
- how deep that text already went is not known here, and the answer is
read back rather than guessed at.
`append_json_value()` writes the depth down before it looks at the
value, because the value is going there whatever it turns out to be, and
it asks whether that depth is past the limit in the same place - in
front of every arm rather than inside the one that reads. A value put
where the limit has already been reached is past it whatever it holds,
and only the arm that reads can say how much further down it goes; a
value written out as a string, a bare word and a null never reach that
arm at all.
The flag saying the marks were given up over the depth is kept only
where the assertion that reads it is kept, nothing else ever looking
at it. That is not the same set of builds as the debug one: an
assertion survives into a build with `DBUG_OFF` set where it was asked
to print rather than to stop, and the expression it is given is
compiled there. The marks get a constructor of their own with it, a
member that is not in every build not being one a brace-initialiser
can name.
The one way to be handed a depth past the limit is a function that
WRAPS: it puts a new array where a value already stood, and the value
going in beside it is counted from inside that new array. Appending
gets there because its path names the value and the array it adds needs
no step of its own, so a path of the full 31 steps still leaves room to
add a 32nd structure. The two that wrap while INSERTING name the place
they insert at, which takes the step the path cannot spare, and the path
refuses them first. So
SET @w31= CONCAT(REPEAT('[', 31), '1', REPEAT(']', 31));
SET @wp31= CONCAT('$', REPEAT('[0]', 31));
SELECT JSON_ARRAY_APPEND(JSON_EXTRACT(@w31, '$'), @wp31, 2);
composes 32 structures and, the value having been quoted rather than
read, would otherwise be handed over as an answer nobody had read and
marked as one. Every function that can be handed such a depth composes
into an answer it hands over only while these marks stand, so taking
them away puts it back on the reading a released server always did - and
that reading gives the answer that server gives, in the words it has
always used.
Punctuation written here arrives as ASCII and is converted into the set
being built in. Bytes copied out of a document are already in it, and
putting them through a converting append writes them a second time -
which in a set that spends more than one byte on a character produces
something that is not a document at all. `do_merge()` was doing that to
both of its array spans, and `do_merge_patch()` was looking at a single
byte to decide whether it was standing on a space, which it never is in
`ucs2`, `utf16` or `utf32`. Both now copy, and a new
`json_skip_space()` reads one character and asks what it is. A
character is taken off a string through the one spelling the character
set offers, where two other spellings of the same call had crept in.
A value spliced into a document being built in another character set is
converted first, the result being read in the set it is built in. A
conversion that lost nothing kept the characters it was given, and
being a document is a property of characters, so what was answered
about the value still holds of what came out of it. The three
questions above are asked after that conversion rather than before,
where they could only ever be answered for a value that needed none.
A conversion that put a question mark in place of a character it could
not spell has changed the value, so the bytes that arrived go in
unchanged - as they always have - and are read.
Measured, a document answered for in one character set and spliced into
another falls from three readings to two, and two such values from five
to three.
`Json_scans` counts every reading of a value. It is counted in
`json_scan_start()`, which is the one door they all go through:
counting at the callers would mean keeping a list of them, and a list
like that is wrong the day something is added to it.
The readings this file makes only to check what it claimed go through
that door as well, so each is counted like every other and taken back
off again - by `json_scan_start_unbilled()` where the reading starts
here, and by the same two steps by hand where it is borrowed from
`json_valid()`.
Measured over two rows, editing a document handed on from another JSON
function falls from six readings to four, and a chain of three edits
from ten to six.
Nothing here owns the document. A function takes raw pointers into it,
works out the rest of what it was given, and reads it afterwards. What
working those out can write is not this function's to know: the
document can be a table's row, a routine's variable or another
statement's user variable. That the bytes stay where they are is true
today because of the shapes an expression can take, and not because
anything makes it so, which is exactly the kind of thing that stops
being true quietly.
So a debug build holds a copy from before the rest is worked out and
says whether the document still matches when it is read, comparing the
two runs of bytes with the one function there is for comparing two. A
release
build keeps nothing and asks nothing, this being a check on the code
rather than on the data. It costs a wrong answer nothing and it costs
a right one nothing either.
It is taken wherever a document outlives the working out of anything
else, which is every function that takes one and something besides.
Which argument does the working out does not matter: an argument is
worked out no earlier for being a path than for being a value, so
`JSON_REMOVE`, which has no value argument at all, holds its document
across every path it is handed, and so does the one that says whether
to match one key or all of them.
The four that compose in place hand what they wrote back as the
document for the next path, so what the document IS changes between
one pass and the next, on purpose. Theirs is taken again each time
round rather than once at the start.
Thirteen files, of which `func_json_trusted` and
`func_json_wide_compose` are the ones that matter most: everything
else in the suite writes its documents as string literals, which
answer `is_valid` false, so the epilogue still runs and the road this
commit adds is never taken. Their answers were recorded by running
them against a released server, so what they hold is what that server
says rather than what this one does. Where a released server is not a
fair judge - the wide character sets, already put right below this -
each answer is instead compared against the same composition in
`utf8`, which recording cannot quietly agree with.
`func_json_trusted` records the two documents that lose a space to that
skip beside three that have none to lose: one written without spacing,
one where what goes is not the first piece, and one where it is a key.
All five are the positions a released 10.11 server gives.
`func_json_splice_depth` carries the wrapping cases: refused at 32 on
both roads, answering at 31 on both, the same refusal reached through a
value that is a document of its own, and the two inserting functions
stopped by the path length instead.
`func_json_splice_oom` fails the writing of a spliced value, which until
now no injection could reach. It also fails the writes this commit
leaves no reading back to cover: handing over a document that was
composed in place, which is a copy and can run short of room like any
other write; the comma between the values a reading function found and
the copy of each value itself; the key a path names that the document
did not have; and the colon that joins a key to its value, which is
written in the spelling settled here.
Every argument in the hand-over cases is built by a producer that
answers for what it hands over. A document written as a string literal
answers is_valid false, so a statement built out of literals never
composes an answer at all and would say nothing about any of them -
however the literal is respelled first, respelling being unable to
answer for what nothing answered for.
`func_json_walk_oom` fails the writing of the loose spelling itself,
which is the other half of the same injection. A mutator asks for the
room the document it was handed takes and a little over, so the loose
form, which spends a character on every comma and every colon, has a
fixed amount of room and runs out at a place that depends on what is
being written. The room is made to run out by the walk itself, where
the writing is, rather than by a wrapper standing in front of it: a
wrapper is a second door, and a caller going through the other one
would have had the one writing of a value that no injection could
fail, a writing whose handling of failure nobody has checked.
The cases sweep a key length and a run of pairs past that place, so
the failure lands on each of the appends the walk makes in turn, and
every sweep is run both as a bare statement and as an assignment, the
two landing it in different appends. Wherever it lands the answer
either reads back as a document or is refused; nothing half written is
handed back.
`func_json_emit_killed` arms a kill partway through a walk, and what
sorts its statements is whether a walk happens at all rather than
which function was asked. An answer composed from a document nobody
answered for is read back before it is handed over; that reading
writes as it goes, so it is a walk like any other and a killed query
is let go of in it - as it is on a released server, which makes the
same reading over an engine armed to hear a kill.
`func_json_search_typed` says that `JSON_SEARCH` hands back a document
whatever it was asked. Every path it produces is built out of pieces
of a document it has just read through and the punctuation round them
is its own, so a column filled from it is a column of documents and a
later reader is spared finding that out. It has both answer shapes,
the branches of a union agreeing about it, a list of values written
out in the statement, a table the server built for itself, a column
the statement named, and the readings counted with the column's answer
taken away beside the same query with it left in.
`func_json_unfinished` reads a path counted from the end of an array
in a document that stops in the middle. Such a step is not known
until the array it is counted in has been counted, so the document is
read through twice, and the counting pass can run off the end of the
text; what it finds there is carried out of the loop rather than left
to the answering pass, which would otherwise answer about an array it
never finished counting. Every unfinished document is recorded beside
the same document finished, which is what its answer is read against.
A column declared JSON carries a check constraint, json_valid(col),
which reads every value stored into it to find out whether it is a
document. Where the value arrived from something that has already
answered for it, and the store put those same characters down, that
reading finds out nothing that was not already known. This leaves
it undone.
## What has to hold
Five things, all of them asked of the store as it happens:
1. **The Item answered `is_valid_json()`.** A column never does:
the bytes in a row are whatever was put there, and a row can
hold bytes no check ever saw - a replica applies rows
unchecked, a session can turn the checks off, a tablespace can
be imported.
2. **The store answered 0**, so every character mapped across with
nothing put in its place.
3. **The field puts down the characters it is given.** `CHAR` and
`BINARY` pad, `ENUM` and `SET` keep the member they matched
rather than the text that matched it, geometry and the
compressed types store something else - and every one of them
answers 0 while doing it. `Field::store_keeps_characters()`
asks this of `real_type()` rather than by a virtual, so that a
field type has to be NAMED to be trusted: `Field_geom` derives
from `Field_blob` and answers `MYSQL_TYPE_GEOMETRY`, so naming
the blobs does not name it. A switch can only refuse what says
something, though, and the compressed classes say nothing
there: they derive from a named type and go on answering its
`real_type()`, saying what they are through `binlog_type()`,
which nothing here consults. They are therefore asked about
their compression ahead of the names, that being the one thing
they do say. A virtual is safe THAT way round: it is asked in
order to REFUSE, so a field type that has never heard of it is
left where it started, and the inheritance that costs the trust
hands none out. Without that question,
j LONGTEXT COMPRESSED CHECK (JSON_VALID(j))
is marked as holding a document over a record that holds an
image of one behind a header of its own. Nothing comes out
wrong by it and nothing was going to, every reader going
through `val_str()`, which unpacks; what goes wrong is that the
check quietly stops running while the rule that left it unrun
still says it has not.
4. **The characters mean the same at both ends of the store.**
Two sets that are the same are copied and two that differ are
converted, and both of those put the same characters down; a
binary set at either end does neither. It relabels the bytes,
so a ucs2 document stored into a `BLOB` arrives as bytes that
read back as nothing at all while the mark says a document is
there and the check is left unrun.
`String_copier::conversion_keeps_characters()` sits on the
class whose branch it reads: `well_formed_copy()` takes that
branch by asking `conversion_copies_bytes()`, and the
character question is the same call read the other way round,
so the two cannot drift apart.
It is deliberately not the plain `my_charset_same()` the two
standing grants ask for: those are said once, about values not
yet made and stores not yet done, while this one is said with
the store behind it - and asking it here would take the mark
away from a `utf8mb4` document going into a `latin1` column,
which is the commonest shape there is, and the whole saving
with it.
5. **Nothing has already gone wrong.** A value that did not
arrive says nothing about what is in the column, and what is in
it is whatever the store got as far as putting down before it
gave up - which it can do while answering 0. The question is
put to the session rather than to the store, an error raised
while the value was being worked out being no less an error
than one the store raised itself.
Anything else clears the mark. Not marking costs a reading that
was going to pass anyway; marking wrongly admits a value that is
not a document, so every rule here is written to fail the first
way.
`Field::store_answered_for()` asks the five, and the store site is
written in terms of it rather than in terms of them. The item is
asked before the field type is: every value a table is given
reaches here, and one virtual answering no for everything that is
not a document costs less than the two virtuals and the switch
that work out what this field would have done with it.
Where a value is stored by `save_val()` nothing is said about how
the store went, so nothing can be answered for - and nothing has
to be taken back either, the marks having been cleared where that
row image was begun and this route setting none.
## How long a mark lasts
One row image. The marks are cleared as an image is BEGUN, rather
than once something is finished with them, because a row whose
writing ends before anything reads what was written leaves its
marks standing: an update that finds nothing to change never
reaches the reading, and neither does a view whose own check
refuses the row. Clearing on the way in is reached by every row
that can set a mark. Clearing on the way out is not.
`TABLE::verify_constraints` clears on every way out of itself as
well, which is what closes the two readings of
`INSERT ... ON DUPLICATE KEY UPDATE`: the first is drained before
`restore_record()` puts the old row's bytes back.
Whatever replaces a row image wholesale leaves nothing that was
said about the one before it still true, and the list of places
that do it is open where the list of places that say something is
closed. `restore_record()` is the one macro every such overwrite
goes through, and it is `TABLE::restore_record_image()` behind
that name - the memcpy it always was, followed by the drain -
because naming the sites one at a time is how one of them gets
missed, and a missed one is a check that quietly stops being run.
An update over part of a period is where that shows. The pieces
the period does not cover are written by putting the saved row
back into the buffer the checks read, once per piece, and the
check on the second piece runs over bytes the first piece's
trigger answered for.
A stored generated column is written twice - once from the value
it was given and again from its own expression - so the recompute
clears what the first writing said.
## Which check is the column's own
`json_valid(other_column)` types a column as JSON, and has done
for as long as the type has existed, so the question the type
handler asks is not the question this one needs. What was written
into one column says nothing about another. The check must
therefore be json_valid of the very field it guards, which is
asked once at open and kept in
`Virtual_column_info::json_valid_field_index`. A check wrapped in
anything else, and a check belonging to the table rather than to a
column, are never left unrun.
## Nothing is taken on faith
The places that SET a mark are three, and that is a closed list.
Each of them asks first whether any column of the table carries a
check that could ever read one - the assignment in a trigger among
them, so that `SET NEW.col` does not put a mark down on a table
where nothing can pick it up.
The places that must CLEAR one are every place that writes
`record[0]` without going through those three, and that list is
open: a site added later that forgets produces no wrong answer, no
warning and no failing test, only a constraint that quietly stops
being enforced.
So a debug build runs the check anyway wherever one is about to be
left unrun, and stops the server if it would not have passed. The
reading is taken back off `Json_scans`, being the debug build's
work rather than the server's: `Json_scans_unbilled` holds the
count still across a whole check constraint and sits beside the
helper that takes a single reading off, the one being an
expression whose readings the caller cannot count and the other
one call. Backing any one of the rules above
out turns a value silently getting in into an assertion on the
first row that reaches it.
A debug build asks the gate once more at the field itself, because
the closed list is what that reading rests on: a site added later
that marks a table nobody asked about is a site whose marks nothing
polices, and it would be exactly as quiet.
Nothing reads a mark through the field. The one place that reads
one has the bitmap in front of it and asks it by name, so
`Field::is_valid_json()` would be the shortest question on a class
that answers several better-scoped ones - true of one row image
between a store and the next check, and false everywhere a reader
would think to ask. There is no such accessor, and the comment
where it would sit says why there is nothing there to reach for.
## The function the check is made of
A column's check IS `json_valid()`, so the reading left unrun at
the field boundary is the reading `Item_func_json_valid` makes -
and that one was still being made where the value had already
answered. It now returns true on `args[0]->is_valid_json()`.
The two cannot disagree: a mark says the characters read back as
a document, which is exactly what the walk would go and see.
## Asked only where it can be answered
`Field::set_is_valid_json()` is an out-of-line call, and both
`fill_record()` variants made it for every stored column of every
row whatever the table looked like - a hundred calls a row on a
hundred column table, discarded unless that column carries a
check about itself. Whether any column does is worked out once,
where the checks are, and kept in
`TABLE::has_own_json_valid_check`.
The bitmap is drained through `TABLE::clear_is_valid_json_marks()`,
which asks that question first, exactly as the three store sites
do. Where no column of a table carries such a check no bit can
ever be set, so an unguarded drain walks a bitmap nothing will
ever read - and these drains are not on the tables this is about.
They are on every row written or updated in the instance.
## Measured
Writing a document built out of scalars into a checked column
falls from one reading per row to none, and editing one from two
to one. A value written out as a literal still costs its reading:
nothing has read it, so nothing can answer for it.
## Tests
`func_json_check_skip` says what the check still refuses, and was
recorded before any of this was written. `func_json_check_store`
says the same for the stores that do not put down what they are
given, and for the rows whose writing ends early; its answers were
recorded against a released server. Its section 5 has both
directions of the store that only looks like it kept the value -
a wide document into a binary column and a binary one into a wide
column - beside the conversion that really does write the
characters again and keeps its mark.
Its section 6 is the row split in two by an update over part of
its period, which writes the pieces the period does not cover as
new rows by putting the saved copy of the row back into the buffer
the checks read: a mark standing from the value the update wrote
would otherwise be believed of a value it never saw. A trigger
writing the document column while the piece is being made is what
gives the check something to catch, and the row the update itself
writes stands beside them, refused where it is set to something
that is not a document.
`func_json_check_scan_count` counts the readings, which is the
only place the saving shows, `json_valid()` asked directly among
them: nothing for two values built out of scalars, one reading
each for the same two written out as literals. It has the
compressed columns too - both classes, a short value put down as
it stands and a long one packed - counting a reading per row,
beside the same two values in uncompressed columns of the same two
types counting none, which is what makes those counts a fact about
compressing rather than about the values or the widths. It has
the two trigger shapes as well: a checked column written by a
trigger and by nothing else, which costs no reading at all, and
the same trigger over a table where no column carries a check,
which is where a debug build stops if a mark is set that nothing
can read. It counts the row split in two by an update over part
of its period as well: three readings, one for the update itself
and one for each of the two pieces. A piece written over an
answer left standing from the piece before would count two.
A JSON function has to name a length for its result before it has seen
a value, so it asks for one out of what the arguments say about
themselves. What was asked for did not cover what writing the result
out comes to.
A result that does not fit what was asked for is cut where it stands,
and what is left of a document is not one. Writing an internal
temporary table does not raise `count_cuted_fields`, so the store that
cuts it answers 0 and warns nobody; the width of that column was never
chosen by anyone but the function itself.
## Reading the arguments two at a time and one at a time
`Item_func_json_insert::fix_length_and_dec()`, which works out the
length for `JSON_SET`, `JSON_INSERT` and `JSON_REPLACE`, steps `n_arg`
by two but indexes `args[n_arg/2]` and `args[n_arg/2+1]`, which step by
one. A document and k pairs is 2k+1 arguments and only the first k+1
are ever read. So the document is counted a second time, the value of
the last pair is not counted at all, and an earlier value, once the
loop reaches it, is added in the term meant for a path.
`Item_func_json_array_append::fix_length_and_dec()` alongside it reads
`args[n_arg+1]` and is right. Both were right after MDEV-29264. The
up-merge of MDEV-32454, written on a branch that fix had never reached,
put the older form back in one of the two, which is why the correct
form has been sitting next to the wrong one since.
## Room for what the writing makes of a value
Correcting the indices takes away the slack that counting the document
twice had been providing, so the room the value really needs has to be
asked for instead.
A value that is not already a document is written as a JSON string, and
a character that cannot be written as it stands is spelt out instead.
`json_escape()` writes a backslash, a `u`, and the hex of the UTF-16
form, which `my_uni_utf16()` gives as two bytes for a character of the
first plane and four for any other. So spelling one out costs six, or
ten at the very worst, and never more. Asking for twice the
characters, which is what the two functions that asked for anything
did, does not cover it. A string carries no separators of its own, so
the spacing below has nothing to add to it.
`Item_func_json_quote::fix_length_and_dec()` had already worked this
out and asked for twelve, its comment reading that a character can turn
into `\uXXXX\uXXXX`. The writer never puts a second `\u` in; it writes
one and eight figures after it. Twelve is therefore two more per
character than anything can use, so that asks for ten now as well and
the two agree on one number.
The rule is written once now, in `json_value_reserve()`, and asked by
all three functions that work out a length this way, so that they
cannot drift apart again. A value that is already a document goes in
as it stands, and is written out again with everything around it. One
spelled as text and not a document is written as a JSON string. A
boolean is written as one of the two words `true` and `false`. What is
left is a number, written as itself. One that is not there at all is
still written, as `null`, so nothing is ever shorter than four
characters.
Which of those a value is has to be the writer's question and not one
of its own, so it asks `is_json_type()`, which looks through the
wrapper that aggregating character sets puts round whichever argument
has to move to get into the set the result is spelled in. Asking the
argument's own type handler instead prices a document that reaches the
writing as a document, and goes in as it stands, as though every
character of it were going to be spelt out - ten apiece for an
escaping that never happens:
CREATE TABLE t (v VARCHAR(20) CHARACTER SET latin1,
u VARCHAR(10) CHARACTER SET utf8mb4);
CREATE TABLE w AS SELECT JSON_ARRAY(JSON_QUERY(v, '$'), u) FROM t;
declares `varchar(314)` that way, where the same two widths in one set
declare 152. Nothing comes out wrong by it - this error is always room
asked for and not taken, never a value cut short - but a wide enough
result passes the width at which a column is given a blob to live in,
and there the asking has decided the type. Only the two that BUILD a
document can meet a wrapper at all, those being the two that aggregate
the sets of their arguments; `JSON_ARRAY_APPEND` and the three that
edit take the set of the document they were handed and aggregate
nothing. The question is settled where it is asked rather than at the
callers, so that a function which comes to aggregate later inherits the
right one.
Seven of the SQL functions are served by those three: `JSON_ARRAY` and
`JSON_OBJECT`, `JSON_ARRAY_APPEND` and `JSON_ARRAY_INSERT`, `JSON_SET`,
`JSON_INSERT` and `JSON_REPLACE`.
## Room for the spacing a document is written with
The document a function is handed is written out again, with a space
after every separator that is copied. So what comes back can be longer
than what went in whether something was taken out of it, put into it,
or neither, and the room for the document has to cover the writing and
not only the reading.
`JSON_REMOVE` shows it plainest, having nothing to add: a dense array
of thirty ones, 61 characters held in a `VARCHAR(64)`, comes back as
87, and a path that matches nothing at all comes back as 90 with
nothing taken out. The same spacing arrives when a document is edited
rather than cut down, and again when one is spliced into another: the
same array comes back from `JSON_SET` as 93, and merged with itself as
180.
None of them allowed for it. For `JSON_SET` and its two the accident
above had been covering it, and correcting the indices takes that away
as well, so what was covered by mistake is now asked for on purpose.
`JSON_ARRAY_APPEND` and `JSON_ARRAY_INSERT` never had it, and neither
did a value that is already a document, which is what `JSON_MERGE` and
`JSON_MERGE_PATCH` are counted with. All of them ask for twice the
document now.
Twice is a true bound rather than a guess: a separator has a value on
either side of it and the shortest value is one character, so at worst
every second character gains one. It is what `JSON_LOOSE` asks for,
adding the same spacing, and `JSON_COMPACT`, which only ever takes
spacing away, asks for exactly the argument.
`JSON_QUERY` and `JSON_KEYS` also ask for exactly the argument and are
right to. `JSON_QUERY` hands back a piece of the document as it
stands without respacing it, and every key `JSON_KEYS` lists costs
less to list than it cost to declare.
## What this moves
The length asked for is a ceiling, and eleven recorded lines move, each
of them a declared column width or the length reported in the metadata
of a result.
Six grow, being the ones that were asked for too little: two in
`func_json`, one in `func_json_agg_limits`, two in `func_json_charset`
and one in `type_json`. One of them passes the width at which a result
is given a blob to live in, so what it declares is a blob.
Five shrink by the two characters apiece that nothing can use, and all
five are ceilings `JSON_QUOTE` names: three in `func_json`, one in
`func_json_charset` and one in `type_json`.
No value moves, and no warning or note appears or goes away.
## The test
`func_json_value_reserve` reads the ceiling off the result itself
rather than off a column the result was put in. A result wide enough
to be given a blob to live in is held by the blob whatever was asked
for, so a case shaped that way passes whether the arithmetic is right
or wrong, and the width above which that happens is not a fixed one.
Read straight off the result, the answer is the arithmetic and nothing
else: unfixed, one line of it says the ceiling is 33 and the value that
same statement produced is 47, and another says 64 against 87.
The spacing is asked about the other way round, through a document
packed tight enough that the spacing alone outgrows it, put through
each of the functions that write one out again and read back off the
column it lands in. Unfixed, the array of thirty ones comes back from
`JSON_SET` as 83 characters of a 93 character document, and `JSON_VALID`
says 0 of what is left.
What a character costs to spell out is asked as a width rather than as
a length, there being no charset here that has to spell one out. A
value of 45 characters is the room for 45 of them and the punctuation
around them, which is under the width at which a result is given a blob
at ten apiece and over it at twelve, so the answer is a `VARCHAR` or a
`TEXT` according to which is asked for.
The wrapper is priced by declaring the same two columns twice over,
once within one character set and once across two, for the array and
for the object: the two declare the same width in both worlds and hold
the same answer. Beside them stands a function that EDITS, declaring
the same width in both worlds for the other reason, which is what says
the two above move because of the wrapping and not because two sets are
in play.
`Copy_field` has two `set()` overloads and only one of them records the fields it works between. `set(Field*, Field*, bool)` records both, but returns early for a `MYSQL_TYPE_NULL` destination without recording either; `set(uchar*, Field*)`, which copies a field into a string buffer, has no destination field to record and records neither. Entries of every kind go in the same array, and callers walk that array with a single loop. So anything that asks an entry which fields it works between reads whatever the memory the array was built on happened to hold, and `Copy_field() = default` left `from_field` and `to_field` indeterminate for it to read. Give the class a constructor that starts the two absent, so that an entry that was never told about a field says so. Nothing in the server asks today - the only readers are the copying functions that the field to field overload installs, and those are reached only for the entries that overload filled in - so this changes no behaviour. The unit test builds an array the way the server builds one, on memory known not to be zero, and reads the two members.
Whether a string column is a JSON column is decided by whether it
carries a `JSON_VALID` check constraint. The test did not ask what that
constraint reads, so `CHECK (json_valid(other_column))` typed this
column, and so did `CHECK (json_valid('{}'))`, which is a constant true
and constrains nothing whatsoever. The column the constraint does
protect is left an ordinary column, so the type lands on the other one
of the pair.
Being typed JSON is what puts a column's contents into a document
verbatim instead of quoted and escaped. A column that keeps no promise
about its contents was therefore spliced, and
SELECT JSON_SET('{}', '$.k', a) FROM t1;
returned NULL for text that is not a document where an ordinary column
would have given `{"k": "not json at all"}`.
Ask the question the callers need: does this column carry a check
constraint that says THIS column holds a document. The predicate that
asks it already existed for deciding which checks a written document
lets stand; the two now share one test of the expression, which the
frm reader wraps in the further question of whether the check belongs
to the column rather than to the table.
THIS COLUMN is same table and same position, and not the same `Field`
object, because a table has more than one `Field` for one column. The
row images a trigger reads through `OLD.` and `NEW.` are reached
through a second and a third set of them, made by
`Field::make_new_field()`, which memdups the field and points the copy
at the other record buffer; the copy carries the same
`check_constraint` pointer, to an expression still naming the field it
was fixed against. Asked of the object, such a copy answers that its
check is about some other column and stops being a JSON column, so
CREATE TABLE t (id INT, j JSON);
CREATE TRIGGER tr BEFORE UPDATE ON t FOR EACH ROW
INSERT INTO lg VALUES (JSON_SET('{}', '$.k', OLD.j));
logs `{"k": "{\"a\":1}"}` - the column's contents quoted into the
document - where a released server logs `{"k": {"a": 1}}`. Which
copies exist is not symmetric, so the cases that would break are not
the ones a reader would guess: the `OLD.` buffer is built for any
update or delete trigger and copies every column, while the `NEW.`
buffer is built only where some column of the table cannot be null and
copies only the columns that cannot, and `OLD.` in a delete trigger
reads the table's own fields.
Two integers give nothing away. A column's check can only read columns
of its own table, so a check naming another position is a check about
another column - which is the thing this exists to refuse, and which a
check over a neighbouring column still gets - and one naming the same
position in another table is a copy that was put somewhere else, where
the constraint it brought along says nothing about where it landed.
Rebinding the constraint on the copy would answer it too, and is not
done: that needs a fresh `Virtual_column_info` and a fresh item tree
for every copied field, on a path every temporary table goes through,
to say what two integers already say.
The declared type was decided a second time and separately, in
`Field_longstr::make_send_field()`, which asked the constraint
expression to name a format through a virtual of its own. That walk
understood a conjunction, so `CHECK (length(js) > 0 AND
json_valid(js))` named `format=json` where the question that types the
column answers no - the client being told a column holds documents
that the server quotes into them.
It asks `has_json_valid_constraint()`, which is the predicate
`Field_string::type_handler()` just below already asks, and the virtual
goes with it: the declaration on `Item`, the descent `Item_cond_and`
made through a conjunction's parts, and the answer
`Item_func_json_valid` gave. What a client is told about a column is
what the column is.
Every column that stops being reported as a document column is one the
server already warns about where it is defined, in the warning that
says the check asks more than the call of it, so the protocol says to
the client what the writer of the table was told to their face. That
is a change to how a released server behaves rather than to anything
introduced here, and it is worth saying so plainly: a column with a
check of that shape has been advertised as a document column for
years, and a connector reading the metadata to decide whether to parse
a value will stop parsing values out of it. The other producer of the
format is untouched - a real `JSON` column and every expression result
are named by the type handler and never came this way - which three
recorded rows say: the shape with the call alone keeps the format
beside the shape that loses it, and the file of expression results
does not move at all.
Both questions about a `JSON_VALID` call - the one that compares a
`Field` and the one that compares a name - are that call over a single
column argument, plus a way of naming the column. The shape is
`json_valid_single_field_arg()` and is asked once. Whether the
argument is followed through `Item::real_item()` is a flag it takes
rather than the difference between two copies of a walk: it is
followed where the expression has been fixed, and where it has not
there is nothing yet to follow.
The walk over a conjunction's parts that is left is not the one that
went. It decides which of two warnings a refused constraint is given,
and it stops at a conjunction on purpose: one that descended
everywhere would call a constraint that can pass without the call
holding a constraint that requires it.
Writing `JSON_VALID` in a column's check constraint is how a JSON
column is asked for, so a constraint that mentions it and leaves the
column ordinary is worth saying out loud rather than leaving to be
found when the column reaches a document. Three warnings, at the time
the column is defined, where the expression is not fixed yet and the
column can only be recognised by its name:
4269 the call reads something other than this column
4270 the constraint can pass without the call holding, as under OR
4271 the call is about this column and holds, and something stands
beside it, as under AND
The numbers start above every one the later series have given out. A
number is a promise to whoever reads it - `mysql_errno()` is what a
connector keys off - and numbering here is positional, an error's
number being how many entries stand before it, so an entry appended to
a series whose tail was frozen at release takes a number a later series
has already spent. A second `start-error-number` is how this file
already opts out of position-derived numbering, and it starts these
three at 4269, one above the highest given out anywhere.
Which of the three is said is decided by the same question the typing
asks, so a check that mentions the call and is refused all the same is
always told about. The last of them is the commonest shape of the
three and is the one the promise is kept in: a conjunction cannot pass
unless every part of it does, so `CHECK (json_valid(j) AND id > 0)`
really does require the call, and the column is still ordinary text.
A check constraint the server writes itself, for a `JSON` column or for
the columns of a temporary table, reads the column it belongs to, so
those stay JSON and say nothing. A table level constraint never typed
a column and still does not. The checking itself is unchanged: a
constraint naming another column is evaluated and enforced against that
column exactly as written.
A column that was typed by a check over another column had its contents
spliced into documents; they are quoted now. Where the column held
text that is not a document that turns NULL into the quoted text, and
where it held a well formed document that turns a nested document into
a string. Both follow from the column never having been a JSON column,
and the warnings say which columns those are.
The constraint shapes that are not a bare check of one column are
recorded with what each of them accepts and refuses, the nested one
among them: an `AND` whose parts must all hold, but whose call sits
inside an `OR` one level down, so the constraint can pass without the
call holding and the column stays ordinary text.
`func_json_valid_constraint` reads a JSON column through `OLD.` and
`NEW.` in update and insert triggers, nullable and not-null, with the
delete trigger and the two direct reads beside them to say which of the
seven go through a copy at all. Nothing in the suite had read a JSON
column that way. Every answer recorded there is the one a released
10.11 server gives, taken off that server rather than off this one.
A value read out of a record is read again to find out whether it is a
document, because a record says nothing about how what is in it was
written. For a temporary table the server built for itself, something
does.
Such a table is written, read and thrown away inside one query, and no
statement can name it. Each of its columns has one producer, known
while the column is being made, so what will be written into it can be
settled then and there - once, rather than once per row - and a reader
of it spared finding out again what was already known.
`TABLE::is_valid_json_static_set` is a bitmap pointer, and the pointer
is the whole of the discriminator. `Create_tmp_table::start()` is the
only place that leaves one, so a base table and a table the user asked
for answer no by having nowhere to keep a yes. Nothing has to remember
to say no: `open_table_from_share()` zeroes the `TABLE` it fills in,
and the one place that copies a `TABLE` wholesale says it outright
through `TABLE::set_filled_by_engine()`, the call the two ways in
already use, so what giving them up amounts to is written down once.
One table is built here and filled elsewhere, and it hands the bitmap
back: a select pushed down whole to an engine borrows this machinery
for its columns and then has its rows written by that engine, so no
store of ours ever sees what goes into them. How well the item a
column was made from answers for the values IT makes says nothing about
values it never made. Both ways in give the answers up - the arm of
`select_handler::prepare()` that builds such a table and the arm that
takes one the engine brought with it - so that every way in has done
it and not merely the one anybody walks, and
`select_handler::execute()`, where the engine's rows are taken, asserts
that some way in did.
Unlike `TABLE::is_valid_json_set` alongside it, these bits say nothing
about one row image. They are written while the table is being built
and they hold for every row the column will ever be given, which is why
a reader can still believe one long after the row it was written in is
gone.
Every row but one. A row that came out SQL NULL has no characters
under it to have been answered for, and the bit stands over it like
any other, so `Field::answers_is_valid_json()` asks whether the field
is holding a value before it reads the bit at all. That is where the
question belongs rather than at the store that acts on the answer: an
answer wrong where it is given is wrong for every reader of it, and
the store is one of four. Without it, a NULL arriving at a column
that cannot hold one - which puts the empty string there instead - is
written down as a document, and the check constraint saying it must be
one is left unrun over the value it exists to refuse. A grouping or a
union is enough to build the table the bit is about:
CREATE TABLE t (j LONGTEXT NOT NULL CHECK (JSON_VALID(j)));
INSERT INTO t (j) SELECT JSON_EXTRACT(d, '$.nosuch')
FROM src GROUP BY d;
`func_json_check_store` has both shapes, in the two modes that decide
whether the row is refused or carried past, and beside them a column
that can hold a NULL, which holds the NULL itself and is not asked to
hold a document.
`Field::set_is_valid_json_static()` is told, at each of the two places
a field of such a table is added, the item that will fill it. It asks
for an item that gives back a document every time it is evaluated
rather than the time somebody happened to look, and for a field that
puts down the characters it is handed. It does not ask whether the
character sets keep them, and says why it does not: the branches are
asked what they always answer, which is not what the column's store
will keep of the value, and the set the column will be in is not
settled until the line below the loop. The question belongs at the
store, and a reader of the grant would otherwise have to find its
answer at the far end.
`Field::confirm_is_valid_json_static()` runs after every store into a
marked field, and is where that question is asked. It asks how much
was kept, and it asks `String_copier::conversion_keeps_characters()`
beside it -
the caller handing over the set the value was written in - because the
length alone answers for a store that converts nothing. A store
between a wide set and the binary one keeps every byte and calls them
by the other set's name, so the length comes back unchanged over
characters that are no longer the ones that were written: a `ucs2`
document arrives as bytes beginning `00 7B`, which nothing reads as a
document. The grant cannot ask this and the confirm says so where it
does: the field is made from a type holder whose own set is the
aggregated one and so is always the field's, which makes the question
answer itself there and decide nothing.
What is left of a confirm once the answer itself has survived it - the
spelling and the depth, which are said about the values rather than
about the column - is one body that both confirms end at. Two copies
of a check are two chances for one of them to stop matching what it is
checking, which is the reason the debug reading below it is written
once as well.
Going wrong is not reported here:
`Field_longstr::report_if_important_data()` answers 0 for a truncation
unless `count_cuted_fields` is raised above `CHECK_FIELD_EXPRESSION`,
and writing a temporary table does not raise it. A store that dropped
the tail of a document would return success and warn nobody. A mark
lost that way is lost for good, one field of one row being enough.
`Item_field::is_valid_json()` hands the bit back, and
`append_json_value_from_field()` asks the field where it used to assume
nothing - which is where an aggregate reads back the elements it
gathered.
A column of one of these tables can be written into a column of the
next, and the second is built before either has a row - so what the
first says at that moment is not what it will say once the row comes.
It is more than it will say: the bit is set in one place, reached only
while such a table is being built, and everything that touches it
afterwards only ever clears. What the second column is given is
therefore an upper bound on what the first can still say, and asking
the first again as each row is put across is what turns the bound into
the answer. Both tables are alive at that moment by construction, and
the first has already stored the row being asked about, so nothing
here rests on the order two tables are written in.
A value reaches such a field off an item or off another field, and
those are two different questions - the first is asked of the store,
the second of the source. A field filled from another field is
confirmed in `field_conv()`, after the dispatch it makes and before it
hands back, so a route that reaches it is covered wherever it is
written rather than by being on a list that has to be kept complete.
Two routes reach what it dispatches to without going through it - the
fast copier a grouping key is written by, and the `copy_fields()` loop
- and those two ask for themselves. The copier is asked although no
column carrying an answer is known to arrive there, which fields a
route is handed being a fact about the queries people have written
rather than about the route.
Whether the loop has anything to ask about is settled where its entry
is made rather than once a row. The bit it would read is set only
while a temporary table is being built, which is before any entry here
is made, and afterwards it is only ever cleared; so a yes recorded at
the entry is an upper bound, and the confirm the loop makes reads the
bit again and turns the bound into the answer. An entry that records
neither field answers no by starting so, both kinds of entry sharing
one array. The unit test over that constructor holds all three of its
members to it, and reads the answer as a byte rather than as a value:
a bool holding neither 0 nor 1 has no comparison the language defines,
so asking one whether it equals false over the poison the test is
built on hands the poison straight back and passes.
What the source is asked is what the store is asked - whether it
answers for its values, and whether every character survived - with
the character sets asked about as well, a length in bytes being able
to agree across two of them while every character disagrees. A column
is only ever answered for where they match, so that decides nothing
today; it is there so the byte count does not quietly become the wrong
question at a fill site the grant never looked at.
A grouping that needs no temporary table of its own still has to hold
a value across the group it belongs to, and holds it in a buffer
instead. A HAVING or an ORDER BY reads what it reads out of that
buffer, so what arrives there is a value nobody had answered for.
A buffer is filled beside the item that made the value and holds one
value at a time, so what that item says goes away with the value and
is about it still when it is read back. That is more than a column
can be told. A column is made before any of its rows exist, so only a
producer that answers for every value it will ever make can be
recorded in one; a buffer is filled afterwards and takes the answer
about the value in it, which is why a function answering for what it
has just written is read here and not there.
A buffer that could not find the room for what it was handed keeps
what it held before and hands that back, as it always has. What is in
it then is a value from another row that nothing here asked about, so
the buffer answers for nothing. The room is asked for by the copy
rather than reported by it, which is the same question a store into a
column is asked, and it is arranged for the same way.
Whether a value held for a group is one a splice puts into a document
is settled by the type, and the type is settled where the buffer is
set up. It is worked out there and kept, the row path having
otherwise asked two virtuals per copied field per row about something
that was true before the query began.
What reads a buffer is a reference, and a reference hands back what it
reads from the result side rather than from the value side. For an
item that keeps a result field those are two different runs of bytes -
the value it made, and a copy of it sitting in a record - and only the
first is what anything said about; for a buffer there is one.
Which of the two a reference reads is not a property of what it refers
to. Two references over the same item read differently, so the
question can only be put to the reference, and it is one the reference
already answers for its own bytes. The three answers therefore grow a
result side beside the value side,
virtual bool is_valid_json_result() const;
virtual bool is_nice_json_result() const;
virtual uint last_depth_result() const;
each class saying there whatever its own `str_result()` reads:
`Item_field` out of its result field, `Item_ref` by the same runtime
condition its `val_str()` uses, `Item_direct_ref` off the value side,
`Item_direct_view_ref` by handing on. A reference then asks the side
it is going to read from and carries what it finds, where before it
had to decide from the referent alone whether carrying anything was
safe - and a class-wide answer cannot settle a per-reference question.
The standing answer gets no result-side twin and could not use one.
It is asked from `Field::set_is_valid_json_static()`, which runs after
`create_tmp_field_ex()` has already pointed the result field at the
very column being granted, so a twin would have read the bit it was
about to set.
A UNION writes one column from a producer per branch, and that column is
read as one column - nothing about a row afterwards saying which branch
it came out of. So what can be said about it is what every one of those
producers says, and one that says nothing is the whole column saying
nothing.
They are all in reach in one place: the branches are walked item by item
at the same position while the types are being agreed, and each position
gathers there the one item each branch put at it. That walk already
aggregates in this direction, taking a column to be nullable if any
branch's is; taking it to be answered for if every branch's is is the
same walk read the other way round. Nobody gathered is a no, an answer
about no producers being about nothing rather than a yes about an empty
set.
Asking at that moment is sound in a way asking a column is not. What is
asked of a producer is a property of the kind of item it is rather than
of anything it has done, so it holds before a row exists as much as
after; a branch whose producer is a column answers no, which is what
keeps one such table read into another out of this and on the footing it
already has.
A recursive CTE is where they are NOT all in reach: the column is built
once the parts that do not go round have agreed on a type, the parts
that do go round not being walked at all. There the ones that were
walked are not allowed to answer in their place - and this is the same
place, and the same reason, as the nullability that is forced on there
already.
Rows written one list at a time rather than one branch at a time gather
in the same holders through the same walk, and are read the same way.
A document spliced into another goes in as it stands, so a caller whose
own answer has to come out spaced a particular way cannot take one that
is not without reading it - and the functions that EDIT a document are
all such callers, each of them reading back everything it has written to
settle the spacing. A value that arrived spaced that way already saves
them that reading, and until now nothing in a record could say so.
This is a second thing a column can answer for, and it is not come by
the way the first is. Being a document is promised by the producer
before a row exists, so it can be written down once and believed; how a
value is SPELLED is settled one value at a time, and there is nobody to
ask about a value nobody has made. So it is granted along with the
first answer and spent afterwards: every value that arrives spelled
another way takes it back, for good, and what the answer amounts to when
it is read is that nothing put there so far was. That is what a reader
with a row in front of it is asking, the row having been put there
before the reading.
It is spent at the places the first answer is already asked about, both
of them: the store, which has the item that has just written the value
and can ask it, and the fill from another field, which has only the
field the characters were copied from and asks that. Nothing new was
added to reach them.
A column of a table the user asked for still answers neither, and by the
same construction - the spelling is only ever read through the answer
above, so the one place that leaves a bitmap is the one place either can
be kept.
A value spliced into a document is read to find out how deep it nests,
because how deep it ends up is what decides whether the answer can be
read back at all. A value something else has already answered for is
not read, so that measurement is not made either, and the only thing
standing in for it was how long the value is - nothing nests deeper
than half its length. That is exact for a run of brackets and says
nothing whatever about a long shallow document, which was read as
before however often it had been read already.
`Item::last_depth()` is a third answer beside is_valid and is_nice,
asked under the same rules and about the same value. A function that
walked the value while writing it knows how deep it went and says so;
everything else answers `JSON_DEPTH_UNKNOWN`, which costs a reading and
nothing else. The two answers are bounds on the same thing, so the
smaller is taken and either alone would do - which is what leaves a
value nobody counted treated exactly as it was.
The counting comes from walks that were happening anyway. `json_nice()`
kept a depth to know how far to indent and now hands it back, which
answers for every function that reads its own result. `JSON_EXTRACT`
already tracked the deepest value it wrote. The functions that BUILD a
document collect it from what went in, and the ones that EDIT one take
the deeper of what went in and what the document said of itself.
Merging takes one level more than its arguments: two arrays laid
together stay where they were, but an array merged with anything else
makes that thing a member of it, which is one level in. It folds that
in one argument at a time, beside the two marks that are already taken
that way, because what an argument answers is about the value it has
just handed over: an argument that reads a variable answers out of what
is in the variable now, and a later argument can assign to it.
Where a value's own answer goes is taken by reference rather than by
pointer, there being no caller with nowhere to put one: every function
that composes a document keeps a set of marks, whether or not it ends
up believing what they say, so the absent case ten guards stood for
cannot arise - and a guard that cannot fire is a reader's question
about when it does. What a caller says as it starts is the depth it
is reckoning from and nothing else, the marks taking that in a
constructor of their own, and where a function hands its document back
rather than reading it the depth it worked out goes over with it. The
bound on how deep a value may go is worked out inside the arm that
uses it, past the questions that would make it moot, rather than ahead
of all of them.
And a record can say this too, but not the way it says the other two.
Those are answers; this is a figure, and the figure a column can offer
is the deepest of the rows written into it so far rather than the depth
of the row being read. That is never less than the row needs, which is
all a reader wants of it - nothing acts on the figure except to decide
that a reading can be skipped, so one larger than the truth costs a
reading and one smaller would let a document be composed that nothing
can read back.
It runs the other way from the two beside it, and for the same reason
they run the way they do. They start as a yes and are taken back; this
starts at nothing, that being the deepest of no rows, and is raised by
each value that arrives. A value nobody counted arrives as the largest
figure there is, so it takes the column there and nothing brings it
back - which is a column that has stopped saying, written as arithmetic
rather than as a rule. It is asked through the answer above like the
spelling is, so a column that has given that up has given this up with
it.
An aggregate that gathers its elements through a table therefore reads
them no longer, which is where the sister that writes its pairs
straight out has been all along.
No answer. The readings that are left out could only have found out
what was already known, so no value changes and no diagnostic appears
or goes away. Three files say so for the column: one puts a document
through the shapes it can travel in and shows the answers standing,
another counts the readings through `Json_scans`, and the third stands
where the server gives a table it built to an engine to fill. That is
three gatherings pushed down whole - one sent straight out, one held in
a temporary table on the way, and two ordered so the rows are read back
out of the table they were held in - each with the `EXPLAIN` that says
the engine took it, since a case the engine declined would run down the
ordinary road and pass without going near what it is there for. An
engine writes
`record[0]` through `ha_write_tmp_row()` with no `Field::store()`
anywhere on the road, so nothing there could confirm a promise or take
one back; they are disclaimed at the handoff rather than at the fill,
a table built out of this one reading them while it is being built.
Nineteen of the forty counts fall: a document through a derived
table, through a GROUP BY, gathered by an aggregate, gathered by one
whose arguments were buffered first, the two that read one such table
into another - a chain of them, and the table an aggregate gathers in,
which is built out of a column rather than out of the function that
filled that column - four held for a group in a buffer: read back from
a HAVING and from an ORDER BY, filled from a column rather than from an
item, and filled by a producer that answers for the value it has just
made, which is the one that reaches nothing read at all - two written by
more than one producer at once, a UNION whose branches all answer and a
column written one list of values at a time - and three where the
spelling is what is answered for rather than the reading: a document
edited out of such a column, the same out of the second of two such
tables, and the same where the row crossed by the copier that fills a
grouping.
Four of the nineteen are the depth rather than the reading, and they
are the only long documents in the file: every other count in it is on
a value short enough that half its length was already inside the limit,
so nothing there was ever read to find out how deep it went. A long
document out of such a column goes in unread, and so do one carried
into a second such table, one an aggregate gathers, and one out of a
column two counting producers wrote.
The other twenty-one stand, among them every shape that must not be
believed: a UNION one of whose branches answers per evaluation and one
whose branch is a column, the same for a list of values, a recursive
CTE, a table the user asked for, and a producer that answers per
evaluation into a column, which is the same producer the buffer above
takes and the difference between them is the point. Three of them are
that same editing over a column filled by a producer that hands its
document back spelled some other way, at each of the three places the
spelling can be given up.
Three more are the depth not being answered for: the same long
document out of a base column, out of a column whose producer copied
what it hands back rather than writing it, and out of one where a
single producer among several counted nothing - the figure being the
deepest of all the rows, one unknown row is an unknown column.
Five of those twenty-one are a place that gave its answer up, and
between them they name every place one can be given up: at the store,
at the store into the first table of a chain, at the row that crosses
between two tables, at the store into a buffer an aggregate gathers
from, and at a buffer that could not take what it was handed. Each
runs the query of the case that believes, so its count is what that
case would have come to with nothing said, and each spells the same
value.
A store that comes up short can no longer be reached through any of
these functions, each of them asking for the room it goes on to use, so
they are arranged for with debug hooks.
The counts for the depth move where the length was answering and now
is not: a long document put inside another was read at every step out
and is read once, so two nested arrays over two rows go from 6 to 2.
The exactness the count has to keep is pinned beside the length's own -
a chain padded until half its length is seven times the limit, where
nothing but a count can tell one that is admitted from one that is
refused, and the same carried through a document that was edited. One
more moves there once a column can answer as well: an aggregate that
gathers a long document through a table, which is the shape that stood
because the table was where the counting stopped.
A debug build reads every marked value back and asserts that it reads
as a document, that it is spelled the way it was said to be, and that
it is no deeper than was claimed, those readings being the debug
build's work and going uncounted. Those detectors are compiled
wherever the assertions that read them are, which is not the same set
of builds as the debug one: an assertion asked to print rather than to
stop survives into a build with `DBUG_OFF` set, and the expression it
is given is compiled there. It is the depth that needed it: two
claims were short when first written, one for a value that goes in
quoted where the wrapping had already spent a level, and one for
merging an array with an object. Six of those readings are taken at
the READ rather than at any of the writes, and they are the only ones
that do not care how the value arrived: a fill site nobody taught to
ask its source has to announce itself somewhere, and the first read of
the column is the last place it still can. Six rather than three
because the two fields an item has are asked separately, each
assertion naming the field its own answer came from; three bodies
serve them, one per question, since the three are asked one at a time
and answering the spelling means writing the whole value out again -
which the two cheap ones would then be paying for as well. Nothing else
reads any of these marks - they appear in no result and change no
output, so a wrong one would go unnoticed until something acted on it,
a long way from the store or the walk that made the promise.
A JSON function keeps its document for the whole of its work. It walks
the document to find the place it was asked about, then works out the
rest of its arguments, and only then reads the pieces of the document
either side of that place. Working out an argument runs whatever the
caller wrote.
What the caller wrote can assign to the very variable the document came
out of. The variables of a package body are in reach of every routine
in that package, so a function called to work out one argument can
store a longer value into the variable holding another, and storing it
gives that variable a new buffer and lets go of the old one. The old
one is what is being read from, so the rest of the document comes out
of freed heap.
Nothing says so. The functions that splice a document back together
re-parse what they built, and bytes off freed heap usually fail that
re-parse, so the answer is `NULL` and a `Broken JSON string` warning -
which is what a caller sees for an ordinary malformed document. The
functions that only read have no such re-parse and answer from
whatever the buffer holds by then.
## Where it comes from
`Item_sp_variable::val_str()` hands back a pointer INTO the variable
rather than bytes of the caller's own:
if (res != &str_value)
str_value.set(res->ptr(), res->length(), res->charset());
That is deliberate, and the comment above it says why: a function that
builds into the buffer it is offered - `CONCAT` is the example given -
must not get to build into the variable. What it leaves the caller
holding is a view, good only for as long as nothing writes where it
points. A caller that means to keep one across working something else
out cannot be given one.
A user variable has never been read this way. `user_var_entry::
val_str()` copies, which is why the same statement written with one of
those holds together, and it is the reason a copy here is no new kind
of cost.
## What is done about it
`Item_sp_variable` answers `val_json()` with a copy in the caller's own
buffer. That is one place rather than one per function: it holds for
every reader that asks for a document, whenever that reader was
written. A reader has to ask, though. The copy is made in the answer
to a question, so anything that reads the variable some other way is
handed the view as before, and a reader that keeps what it read has to
be the one to put the question. The document argument is worked out
before the others, so the value the variable held then is the one the
answer is owed about - which is what a copy gives.
One place, provided everything standing between the function and the
variable asks the same way. A reference did not: `Item_ref::val_str()`
reads its referent through `str_result()` and `Item_direct_ref::
val_str()` through `val_str()`, but neither said anything about
`val_json()`, so a request for a document fell back to
`Item::val_json()`, which is `val_str()` - and the variable handed out
the view again with a wrapper over it. So the two spellings get
result-side twins beside the marks that already had them,
virtual String *val_json_result(String *str);
virtual String *val_json_at_once_result(String *str);
each class transcribing whichever side its own `val_str()` reads:
`Item_field` and `Item_func_set_user_var` out of their `str_result()`,
`Item_direct_view_ref` by handing on, `Item_ref` by the same runtime
condition it uses for the marks. `Item_direct_view_ref` keeps the row
check its other val_XXX have, so a row an outer join filled with NULLs
is answered without the referent being worked out at all, and
`Item_ref_null_helper` notes the NULL down for its owner as its
`val_str()` does. Of the five shapes that put a reference in front of
a variable, only `ORDER BY` over a select-list alias reaches this -
each was run on its own rather than taken as equivalent to another.
A call with nothing left to work out after reading the document is
finished with it before anything else can run, and has no use for a
copy. Those read it with `val_json_at_once()`, which hands back the
view `val_str()` would. Asked of the argument count rather than
written into each function, so that one which gains an optional
argument later goes back to being read the careful way without anyone
having to remember to say so.
One is left out of it. `JSON_UNQUOTE` hands its argument straight on
when there is nothing to unquote, so what it would hand its own caller
is the view.
`JSON_COMPACT`, `JSON_LOOSE` and `JSON_DETAILED` hand their argument's
value straight on as well, where one of them is read as a document
rather than as a string and there is nothing to rewrite - so a caller
that is finished with what they hand back is finished with what the
argument holds. They pass the promise along rather than stopping it,
asking the argument the way they were asked themselves; the body that
is common to both ways of asking is written once.
`JSON_NORMALIZE` writes its answer out afresh and hands back nothing
of what it was given, so it takes its argument the way every other
such function does - through the one place that decides which way to
ask, rather than by reading the value for itself.
Nothing else changes. A document that arrives some other way is
untouched, and so is `val_str()`.
## A walk that outlives the row
A table built out of a document keeps it for longer than any function
does. `ha_json_table::rnd_init()` is told where the document stands
and the walk reads on from there each time a row is asked for, so that
note outlives everything the query does between one row and the next.
Working out a row can write to whatever the document was read from - a
package body's variable is the plain case - and a value stored over it
that wants a bigger buffer lets the old one go while the walk is still
pointing into it. So the document is asked for the way that hands back
bytes the reader can keep, into a buffer the handler owns for as long
as the walk lasts.
A released server answers
SELECT jt.v FROM
JSON_TABLE(d, '$.arr[*]' COLUMNS (v INT PATH '$')) AS jt
WHERE grow() = 7;
with `4038 Syntax error in JSON text ... at position 10` in place of
its rows. Under a sanitizer the same read is a use-after-free in
`get_first_nonspace()`, reached from
`Json_table_nested_path::scan_next()`, on the buffer that
`Field_blob::store()` let go while the condition was being worked out.
Both go, and the rows come back.
## What it reaches
Twenty-one functions over seventeen implementations, each one shown
under AddressSanitizer to read freed memory: `JSON_INSERT`, `JSON_SET`,
`JSON_REPLACE`, `JSON_ARRAY_APPEND`, `JSON_ARRAY_INSERT`,
`JSON_CONTAINS`, `JSON_EXISTS`, `JSON_VALUE`, `JSON_QUERY`,
`JSON_EXTRACT`, `JSON_CONTAINS_PATH`, `JSON_LENGTH`, `JSON_REMOVE`,
`JSON_KEYS`, `JSON_EQUALS`, `JSON_MERGE`, `JSON_MERGE_PRESERVE`,
`JSON_MERGE_PATCH`, `JSON_OVERLAPS`, `JSON_SEARCH` and `JSON_DETAILED`.
It is not about value arguments, and not about the functions that edit
a document. Every argument is worked out just as late as every other,
so a path reaches it - `JSON_REMOVE` has no value argument at all - a
string to search for reaches it, and so does the number of spaces
`JSON_DETAILED` is asked to indent by.
Functions taking a document and nothing else have nothing to work out
afterwards and are not affected.
## The test
`func_json_aliasing` gained five sections. Its sections 4 to 6 already
hold a user variable, a routine's variable and a column, but nothing a
routine calls can reach the routine's own variables, so none of them
could write where the document stands. A package body's variables are
shared by everything in the package, which is what the new ones use.
Two of the four are about the road above. A reference exists only for
a select-list alias and a select-list entry is always printed, so no
shape both routes a variable through a reference and prints what the
function answered; the document is therefore grown and put back, and
the row shows something short. A buffer is let go only where the next
value will not fit, so each case needs a variable of its own written
away exactly once, or the first grow makes room for every later one
and nothing is freed at all. The fourth of those cases is quiet and
earns its place by a liveness probe rather than by failing: a call
finished with its document may hold a view, which is the whole of what
`val_json_at_once()` is for. The last section reads a depth after the
arguments have run, over a document written shallower than the one
measured, which is what makes a wrong depth rather than a freed read
the thing that shows. It covers the merging and the patching beside
the five that ask their document early, and those two are reached
differently: the five take a value or a path that can be made to write
the variable on its way past, while the merging and the patching take
documents, so the writer sits inside one of them - here inside a
`JSON_OBJECT()` standing as the second argument. Merging makes the
answer a level deeper than the deepest of its arguments and patching
leaves it where it was, and both take that figure from what the
arguments say rather than from a reading, which is what puts them
there.
The walk of a table gets a section of its own, where the writing is
done once by the condition deciding whether to keep a row and once by
an entry printed for it, those being different points in working a row
out. The array has more than one element so that there is a row left
to read after the write.
Twenty cases, grouped by which argument does the writing, over a
document with a nested object, an array and a string so that every one
of them asserts an answer rather than `NULL` - `NULL` being what a
freed read tends to produce as well. The width `JSON_DETAILED` indents
by is read off the answer, so the case shows that the argument which
frees the document really was worked out.
The second section is the same package read the cheap way: once for
the answer each gives, and once more with that answer made the
document of something that keeps it across working out an argument
which writes to the variable. That second group is where a function
wrongly let out of the copy would show up.
It has two `JSON_UNQUOTE` cases, and the character set on the variable
is the whole of the difference between them. The answer is declared
utf8mb4 whatever went in, so an argument in any other set is converted
into a buffer belonging to the caller and no view of the variable goes
anywhere; a variable declared utf8mb4 is handed straight on instead and
the answer is a view of it. Take the careful read away and only the
second stops the build, on `watch.unchanged(js)`.
`func_json_handover_oom` fails the copy itself. The copy is a write
and can run short of room like any other, and no ordinary question
reaches the arm that copes with that; what the arm has to do is refuse,
rather than fall back on the view this commit exists to stop handing
out. Reaching it means making the copy fail, and the only thing that
can make it fail is an injection point a release build does not have,
set through a variable a release build does not know - so it is a file
of its own, sourcing `have_debug.inc`, rather than a section of
`func_json_aliasing`, which asks nothing about how the server was built
and runs wherever it was.
`func_json_result_side` says which side of a select list entry a
reference hands over. A name repeated in `HAVING` or in `ORDER BY` is
a reference, and once a temporary table has been built for the
grouping or the sorting it reads a column of that table rather than
the thing that made the value, so a store that shortened the value or
wrote it in another character set is what every question about the
value has to be put to. No query shape prints what a reference handed
over, so the outcomes are told apart by which side of a comparison the
rows come back on: a value answered for joins the array it is put in,
and one that is not is written out as a string and quoted. A column
comes back answered for and a view column hands the question on to the
column underneath it, while a user variable assigned in the same
select list does not - reading it back would perform the assignment
again, and nothing keeps marks for one in any event.
CONCAT works out its first argument into the buffer it will answer
with, and every argument after that into a buffer of its own, appending
each to the first as it goes. An argument that answers in the buffer
it was handed rather than in one of its own leaves CONCAT holding that
very buffer, and CONCAT keeps it as its accumulator without copying
anything - which is the reason for handing the buffer down at all.
Growing that accumulator is where the value goes. `realloc_result()`
asks the buffer how much room it has, and reads none-at-all as
nothing-in-it: it calls `alloc()`, which empties the string it
allocates for. That reading holds for the buffer CONCAT starts with,
which really is empty, and stops holding once an argument has handed
it back pointing at bytes it does not own - a static string, say,
whose room to grow is none and whose length is six.
So `CONCAT(JSON_TYPE('{"a":1}'), 'X')` answers `X`. Nothing is
raised; the answer is a string of the right type and the wrong value.
Appending an empty string is enough to lose it, and only the first
argument can be lost, every later one being appended to an accumulator
that owns its bytes by then.
## What is done about it
The shortcut is removed. What is left asks for twice the room or the
room needed, whichever is more, and `realloc()` carries the bytes over
into the new buffer. With no room at all those come to the same
thing, so the buffer is no larger than the shortcut made it; what is
already in it survives the move. A buffer that really is empty is not
copied out of - `realloc_raw()` skips that - so nothing is paid for by
the case that was right all along.
`Item_func_concat_operator_oracle` is the `||` operator of
sql_mode=ORACLE, and appends through the same two functions, so the
one change answers for it too.
## What reaches it
`JSON_TYPE` and `GET_FORMAT` are the two functions in the tree that
finish this way. `ENCRYPT` finishes the same way and then copies the
bytes into a buffer of its own, which is what keeps it out of this.
## The test
`func_concat` gained a section over both functions: first argument
with one, two and no further arguments after it; a later argument,
which was always right; over a `JSON` column, so the case is not one
of literals; and through the Oracle `||` operator. `CONCAT_WS`, which
accumulates differently and was never affected, is pinned beside them.
A value read out of a stored program's variable is read again to find out whether it is a document, because nothing about a variable says what was put in it. Something does: every assignment to one goes through a single funnel, and it arrives there with the expression that made the value still in hand. So what is in a variable can be said at the moment it is put there, once per assignment rather than once per reading, and a function later handed that variable as the document to work on can be spared finding out again what was already known. A stored program keeps its variables in the fields of a table it builds for itself. No statement can name such a table, and nothing writes one but the funnel, so `TABLE::json_held_marks` is left there and nowhere else - the array being there at all is the whole of the discriminator, as the bitmap beside it is for a temporary table the server built for a query. A base table and a table the user asked for answer no by having nowhere to keep a yes. The room is given where a stored program's variables are built, rather than in the base that builds every table of the kind. Two of the three callers of that base build something else - the table `SUM(DISTINCT)` counts in, and the one row-based replication converts through - and neither holds anything anybody assigns to, so neither has anything to say about a value. Both ways a stored program gets a table come through the one caller, the row a cursor is read into among them, so nothing of the kind is left to be given room afterwards. Unlike the marks a column of a query's temporary table carries, these say nothing about rows. There is one value, the item that made it is right there, and it is replaced whole by the next assignment - so the spelling and the depth are taken exactly rather than as a bound over everything the field will ever hold. The value is read back out of the field only where something reads it. Nothing but the debug check ever looks at it - the marks appear in no result - so a released server does not write it out to hand it to a check that is not there. The rule is the one every store site applies: the item said it was a document, the store kept every character of it, and nothing went wrong while it happened. It is the column's rule, so the assignment asks `Field::store_answered_for()` rather than writing the same questions out beside it. What stays here is the NULL: a variable holding one is holding nothing that reads as a document, whatever the item it was assigned says about itself. That is asked of the destination as the answer is put down, and it is asked again of whatever field the answer is later read off, because a field with nothing in it answers for nothing whichever of the two channels is carrying the answer. Nothing here needs the second asking - an answer put down one value at a time is cleared for a NULL where it is set, so it never stands over a row with nothing in it - and it is asked of both anyway, so that a reader does not have to know which channel it is reading. Keeping the characters is two questions and not one. Whether the type puts down what it is given is one of them, and whether the two character sets put the same characters down is the other. A store into or out of the binary set answers the first yes and the second no - the bytes go across untouched and are then read as whatever the other set makes of them - so a document written in a wide set arrives in a BLOB as bytes that read as no document at all. Both are asked here, in the words the column stores beside this one ask them in. The store is asked what it returned and not merely whether anything went wrong, because on this path the two come apart. Assigning more than fits raises an error under the strict mode and the value never lands; under the loose one the front of it is put down and neither an error nor a warning is raised, leaving a variable holding something that is not a document and nobody saying so. Asking the store what it kept is what refuses that. Nothing is cleared on the way in. Every way out of the funnel ends either at the answer or at the clear beside it, so a value that never arrived leaves nothing standing over the bytes of the one before it. Not clearing first is also what lets a variable assigned to itself keep its answer. A row is written a member at a time whichever form the assignment takes, so each member is answered for by whatever made the value that landed in it. Where a row assignment stops part of the way through, the members it never reached give their answers up: what they hold is what the assignment before this one left. The ones it wrote keep theirs, having gone through the funnel like any other. Assigning one variable to another arrives at the store as a read of the field behind the source, dereferenced on the way, so the item the funnel is handed is the same kind of item a reader of that variable reads through. Answering there answers both, and a value moved from variable to variable carries what was said about it without anything having to move it. The standing question is left unanswered on purpose. A variable holds what the last assignment put in it and the next one will put something else there, so nothing about the value in it now holds for every evaluation - which is what stops a column built out of a variable being granted an answer about values it has not been given yet. A variable is not typed as a document however it was declared, so a value out of one is quoted into a document rather than spliced into it, exactly as before. Which of the two happens is decided by the type; what is answered for here is the other thing, and a value can be answered for and still be quoted. The assertion that said otherwise was written when the two always agreed and is now gone: nothing in that arm reads the answer, and quoting a document writes the same bytes as quoting anything else. `func_json_sp_trust` records the answers, which do not move. Among them is the declared type that changes nothing on the way in: a conversion written out in the assignment, and a variable already holding the wide spelling, each landing in a BLOB, with a reading of the wide variable itself beside them so that what those two show is about the store and not about the set. Its last section writes a row variable from a list of values rather than from a row: `SELECT INTO` and `FETCH INTO` write the members one at a time and stop at the first that fails, leaving a mixture of what this assignment wrote, a member holding nothing, and what the assignment before left. An answer standing for either of the last two would be an answer from an assignment that did not finish. `func_json_sp_scan_count` records the readings, which do: one edit over a variable falls from two to one, five edits from ten to five, and a value moved between variables is not read at all. Both a value the store cut short and an assignment that never landed are in there, being the two ways an answer has to be refused.
The keys of an object are handed back as a JSON array, and the
documentation says as much, but the function was declared
`Item_str_func` rather than `Item_json_func`. So `is_json_type()`
answered false for it, and every function given it in value position
escaped the array into a string:
JSON_ARRAY(JSON_KEYS('{"a":1,"b":2}')) -> ["[\"a\", \"b\"]"]
where the members of the same family that ARE typed put the same shape
of value in as a document:
JSON_ARRAY(JSON_EXTRACT('{"a":[1,2]}','$.a')) -> [[1, 2]]
It is the only function in this file documented to return a document
and left out of `Item_json_func`. Deriving from it is the whole of
the change; the value it writes is untouched.
The array itself was never in doubt. Read as a document it always was
one - `JSON_VALID` 1, `JSON_LENGTH` 2, `JSON_TYPE` ARRAY - and the
workaround for the mismatch said so: `JSON_EXTRACT(JSON_KEYS(...),
'$')` parsed the array back out of itself before it could be put in.
Document position is unchanged and so is that workaround, which now
reaches the same document by the longer route.
## The answer is answered for, and by the writing not by measuring
Typing it is only half of what a splice needs. A caller splicing a
document reads it again unless whatever wrote it says what it is, so
the type alone would have left every splice of an array of key names
parsing it back out of itself - which is the reading the type exists
to make unnecessary. All three things there are to say about the
answer are settled by how it was written:
A name goes in as it was spelled in the document it came from, escapes
and all - a key written `a\"b` there is written `a\"b` here - out of an
object that was read to its end, so whatever needed escaping in it is
escaped already and a name that would not go into an array would not
have come out of an object. Read to its end is the whole of what is
needed, and it is not the same as the argument having parsed: this
reads one object and stops where that object closes, so text standing
beyond it is never met, and
JSON_KEYS('{"a":1} rubbish') ["a"]
JSON_KEYS('{"a":1}{"b":2}') ["a"]
JSON_KEYS('{"a":1}}') ["a"]
JSON_KEYS('{"a":{"z":1} "b":2}', '$.a') ["z"]
all answer, while a break inside the object refuses the answer whole
and says where it was:
JSON_KEYS('{"a":1 "b":2}') NULL, position 8
JSON_KEYS('{"a":{"z":1 "y":2}}', '$.a') NULL, position 13
Between two names goes the pair of characters the loose spelling puts
there, written from the constant that spelling is defined by rather
than from a copy of it, and nothing goes after the opening bracket or
before the closing one, so the answer is spelled the way reading it
back would spell it. Nothing is wrong until the two disagree, so it is
measured that way: with the constant temporarily made comma-tab,
`JSON_KEYS('{"a":1,"b":2}')` answers its two keys separated by a comma
and a tab, where a hand-written separator stops a debug build on the
niceness half of the self-check. And an array of strings is one level
deep, a key name being a string whatever it holds.
The character set is the argument's, `fix_length_and_dec()` taking the
collation from it and the copy being same-set, so the one thing left
to ask is whether a document can be spelled in that set at all. That
guard is written as `JSON_SEARCH`'s is, and it is a live one. A set
with no bracket to write has no braces either and never parses a
document to take keys off, but that is not the question the guard asks:
`json_charset_can_spell()` asks whether the set is ASCII-compatible,
and a set can decode every ASCII byte to itself and still be flagged as
not. `sjis` is one. It parses a document, the answer is a real array,
and the mark is withheld from it:
SELECT JSON_KEYS(j) FROM ts; -- sjis
["a", "b"] -- JSON_VALID says 1
SELECT JSON_ARRAY(JSON_KEYS(j)) FROM ts; Json_scans 2
SELECT JSON_ARRAY(JSON_KEYS(j)) FROM tu; Json_scans 1
so a splice there reads again exactly the array the mark exists to stop
it reading. Nothing computed is wrong - an underclaimed mark costs a
reading and never an answer.
Those two figures are `func_json_keys_typed_scans`, which sources
`have_debug.inc`: `Json_scans` is a debug build's own bookkeeping and
is registered nowhere else, so a file that reads it runs wherever it is
built and answers only where the counter is there to answer.
`is_valid_json_static()` is granted for the same reason. Every row is
an array of names or it is NULL, neither of which has to be read, and
that is what a temporary table asks as it is built - which is before a
row of it exists.
## What does not move
`MAX()` over it still quotes, being typed by nothing it is taken over.
`MAX(JSON_EXTRACT(...))` does the same and always has; it is not this.
`JSON_QUOTE` is left as it is. Its documentation calls the result a
string, so implementation and documentation already agree, and typing
it would make the quoting it exists to do a no-op in the one position
it was meant for.
New test `func_json_keys_typed`, which records value position in the
constructors and the mutators, document position beside it, the keys
that need an escape, the two-argument form, an aggregate and a
temporary table, and the result metadata that decides all of it. Its
section 12 puts both character sets side by side, the one that spells a
document and the one that parses a document and is flagged all the
same; its section 13 records the boundary of the reading from both
sides, the four shapes that answer and the two that refuse.
What is answered for shows in no result at all, only in readings not
made, so it is counted rather than printed. `func_json_scan_count`
gains four rows whose key names are long enough that the length says
more levels than a document is allowed, leaving the depth the only
thing keeping the answer from being read again; two of the four are
the answers that are not arrays of names, an object with no keys and a
document that is not an object. Both forms of the call go from seven
readings to four, the four being the function's own.
`func_json_tmp_scan_count` puts the same names through a temporary
table, where the standing promise is what answers, and four readings
become two - beside a control running the same query with the store
into that column arranged to come up short, which reads four whether
the grant is there or not, so the difference between the two is the
column and nothing else.
Neither JSON aggregate cut its group to a document. They failed in
opposite directions and one fix answers both: a group that has to be
cut is cut at a boundary between elements, and what is left still
reads as JSON.
## JSON_ARRAYAGG cut at a byte offset
It derives from `Item_func_group_concat` and so inherited the check
`dump_leaf_key()` makes once per row, which cuts at the byte the limit
falls on because for GROUP_CONCAT the result has no structure to
respect. The override it carried repaired one landing position out of
several - a cut immediately after a quote, backed off one byte with a
quote written back - and left every other one as GROUP_CONCAT made it:
cap 7 ["aaaa""] not JSON
cap 8 ["aaaa","] not JSON
cap 9 ["aaaa",""] JSON, and the second element is invented
The last of those is the serious one. It is valid, so nothing
downstream rejects it, and it holds an empty string that no row of the
group ever had: the cut landed just past a separator and an opening
quote, and backing off one byte to write a quote closed a string with
nothing in it. Warning 1260 says a row was cut, not that one was
manufactured.
`old_length`, which `dump_leaf_key()` hands the override already, is
the length the result had before the separator and this element were
written - the end of the last whole element, and the one place a cut
leaves something the brackets can go round. So the override is now
that cut and nothing else.
## JSON_OBJECTAGG did not cut at all
It derives from `Item_sum`, keeps its own `String`, and its `add()`
had no length test on any path, so the object grew with the data and
stopped only where the allocator refused it. `group_concat_max_len`
appeared nowhere in the file. A group of 2000 rows of 200 bytes came
to 422893 bytes against a cap of 1024, and lowering the cap to 4
changed nothing.
`LENGTH()` sends one integer to the client, so the whole buffer is
built server side and `max_allowed_packet` is never consulted - as it
is not for an assignment to a routine variable, an INSERT ... SELECT
or a materialised derived table. `max_session_mem_used` does not
apply either: the buffer is not `MY_THREAD_SPECIFIC`, so it is not
counted against the session.
`add()` now remembers where each pair began and takes the pair back
off whole where it overran, closing the group there and saying so with
the same warning 1260. Nothing goes in after that: a later pair small
enough to fit in what is left would go in behind the pair that did
not, putting the object out of the order the group was read in.
The session the cut is measured against is taken once, where everything
else the group starts with is set, rather than looked up again for
every row of it: a group runs on one connection, so the lookup answers
the same thing every time it is made. What is read THROUGH it is read
per row still - the limit is the session's setting as it stands while
the group is being built, which is the whole reason it is asked for
there rather than taken at the width the item was fixed for.
Nothing goes in, but the rows still go by. An argument is an
expression, and the object works one out for every row of the group
whether it has room for the answer or not; a group that stopped
reading would stop whatever the expression does, which is the
caller's to decide and not this function's to drop. So a row past
the cut is read exactly as a row before it is - the key, and the
value where the key makes a pair - and only the writing stops.
`Item_func_group_concat::add()` divides the work the same way, its
own cut reaching no further than the append `dump_leaf_key()` makes.
`report_cut_value_error()` stops being static so the object aggregate
can raise the warning the same way, having no GROUP_CONCAT machinery
under it to raise it.
## The brace is one of the bytes handed back
Both aggregates are documented the same way - *"the maximum returned
length in bytes is determined by the `group_concat_max_len` server
system variable"* - and `GROUP_CONCAT`, which shares that sentence,
ends on it exactly. The brace that closes the object is written when
the group is asked for, which is after every pair has been through the
length test in `add()`, so a group whose pairs ended exactly on the
limit would come back at the limit plus one:
cap 14 {"aaaa":"aaaa"} len 15
The room for it is kept back from the limit, and the pair that would
need that room does not go in. The brace is not one byte wide in a set
that writes no character in one byte, so the width is asked for rather
than assumed - at two bytes to the character the same group would come
back two over. Where the room cannot be spared the answer is `{}`,
which is what a cap too small for a whole pair means here anyway, and
the warning names the row that would not fit rather than the one after
it. A pair that fits with its brace fits as it always did.
## What the cut costs and what it counts
A cap that does not reach the end of the first element leaves an empty
container rather than half of one - `[]` and `{}` - which is what a
whole-element cut means when no whole element fits.
The row the warning names is counted per group for the object, over
the rows that have a key to make a pair of. The array's counter is
GROUP_CONCAT's, which runs on across the groups of a statement; that
is left as it is.
The mark block over `Item_func_json_arrayagg::val_str()` says what the
cut now leaves, and so does the test beside it: a group cut to fit the
length limit is cut back to the last whole element.
New test `func_json_agg_cut`, which walks the cut across every offset
within an element for both aggregates, over strings, numbers and
documents, and covers the empty container, DISTINCT and ORDER BY,
characters wider than a byte, and a NULL key.
It counts what an argument does over the same rows under a cap that
cuts and under one that does not, on either side of the pair and with
the array beside it, and the counts match.
`func_json_agg_limits` records the old shapes; its queries are
unchanged and its result is re-recorded, with the prose that described
the old cut corrected.
A pair that could not be written at all is a different thing from one
that would not fit, and `func_json_agg_pair_oom` is where the difference
is recorded. A pair taken off for the limit leaves an object the brace
can still go round; a pair the buffer would not take is written in part,
and what is left is an object no brace can close, so the group is
refused whole. Nothing reaches that second road by asking an ordinary
question - the room a pair takes is asked for before it is written - so
each of the three writes a pair is made of names a debug keyword and is
failed on its own, with a single row beside them where there is no
separator to write at all.
`Item_func_group_concat::val_str()` walks the tree only while
`result_finalized` is false, and `dump_leaf_key()` raises that flag for
the first row it writes. A row that falls inside the offset is skipped
by an earlier return, which decrements the offset counter and leaves the
flag alone, so a walk in which every row was skipped writes nothing and
raises nothing.
The next caller therefore walks again, and by then the counter has been
spent, so the rows skipped the first time are written - into a result
buffer that belongs to the item and has already been handed out once.
Nothing says how often a statement asks for the result of a group.
`HAVING` on the alias is the shortest one that asks twice, and the
answer must not depend on how often it was asked:
SELECT GROUP_CONCAT(a ORDER BY a LIMIT 2 OFFSET 4) FROM t1;
''
SELECT GROUP_CONCAT(a ORDER BY a LIMIT 2 OFFSET 4) AS v FROM t1
HAVING v LIKE '%';
'a,b'
The second is wrong on its own terms as well: the group has four rows
and the offset is four, so there is nothing left for the limit to take.
`DISTINCT` reaches the same walk by the other route and does the same.
The walk is what produces the result, so the walk is what finishes it,
whether or not it had anything to write.
`JSON_ARRAYAGG` is built on this class and puts its brackets round what
the parent hands back, so the second walk appends the group's elements
after the closing bracket the first one put on and the value stops
being a document:
SELECT JSON_ARRAYAGG(a ORDER BY a LIMIT 2 OFFSET 4) AS v FROM t1
HAVING v LIKE '%';
[[]"a","b"]
`func_gconcat` gains the case: asked once, twice and three times; the
`DISTINCT` route; a group per row of output, where one group spends its
offset and the other does not; offsets that stop inside the group and
no limit at all, none of which move; and the same expression through
the aggregate built on this one, with `JSON_VALID` over it.
`JSON_MERGE` and `JSON_MERGE_PATCH` walk an object argument's keys with
while (json_scan_next(je) == 0 && je->state != JST_OBJ_END)
which ends on a refusal exactly as it ends on the object's end, the
condition asking only whether the scanner moved. Each function has two
such loops, one per argument, `copy_value_patch()` has a fifth, and not
one of them looks at `je->s.error` after the loop. So a document that
broke before its first value was complete is merged from as far as it
got:
SELECT JSON_MERGE('{}', '{"a":1 "b":2}');
{"a": 1}
`{"a":1 "b":2}` has no comma between its members and is not a document
at all. What comes back is an object no argument held - composed, well
formed, and silently missing whatever stood past the break. Nothing
downstream has any reason to reject it, which is what makes it the
quiet kind of wrong: `JSON_VALID` answers 1 for it and `JSON_LENGTH`
answers 1.
The documentation says this does not happen. Of `JSON_MERGE`:
An error occurs if any of the arguments are not valid JSON
documents.
and the function already does exactly that whenever the same breakage
is met anywhere else - one loop further in, or in an argument that is
not an object. It is the same characters every other JSON function
refuses, `JSON_EXTRACT` naming position 8 for the document above.
Only the ends of these five loops were letting it through, and which
side of the inconsistency an argument falls on is decided by something
that has nothing to do with it - whether the *other* argument has any
keys, because that is what decides whether an inner loop with a check
in it ever runs:
SELECT JSON_MERGE('{}', '{"a":1 "b":2}'); -- {"a": 1}
SELECT JSON_MERGE('{"b":9}', '{"a":1 "b":2}'); -- NULL, warning
Both, in one statement, over one column:
SELECT id, JSON_MERGE(a, b) FROM t ORDER BY id;
1 {"a": 1}
2 NULL
The fifth loop is the same inconsistency one step further out.
`JSON_MERGE_PATCH` walks two objects together only when both arguments
are objects; where argument 1 is not one there is nothing to merge
with, so argument 2 is copied whole by `copy_value_patch()`, which
walks an object's keys the same way and whose caller propagates
whatever it returns. So the outcome turned on whether argument 1 was
an object at all:
id v -- argument 2 is '{"a":1 "b":2}' throughout
1 {"a": 1} -- argument 1 is []
3 NULL, warning -- argument 1 is {}
That loop is reached for every object nested inside the value as well
as for the value itself, so a break below the top goes through it too:
`JSON_MERGE_PATCH('[]', '{"a":{"p":1 "q":2}}')` composes
`{"a": {"p": 1}}`.
Each loop end now refuses what its scanner refused. Every case above
becomes NULL with the warning the scanner raised, at the position it
raised it - position 13 for the nested one - which is what the other
side of the inconsistency has always answered. The whole documents
beside them do not move.
A value read to its end keeps everything it had. Text standing after a
complete value is not this and does not move: it is still answered, and
still noted rather than warned about, for the reasons given over
`report_json_trailing_note()`. `func_json_merge_trailing` said every
value in it kept what it always had, which four of its cases no longer
do, and it now says which cases the claim is about.
`func_json_merge_broken` carries the shapes: both functions, both
argument positions, a break inside, a document that stops after a
comma, one that stops at the opening brace, a nested break, three
arguments, a merge onto SQL NULL, out of a table so one row does not
settle another, and `JSON_VALID`/`JSON_TYPE`/`JSON_LENGTH` over the
result. Arrays, scalars, whole documents and trailing text are the
controls and none of them move.
The defect is older than this branch: a released 10.11 answers
`{"a": 1}` here too. Both servers compose the same answer, so no
comparison between them could have shown it; it comes of reading the
functions against their documented contracts instead.
`JSON_ARRAYAGG` is documented in the same words as `GROUP_CONCAT`, which
it derives from: *"the maximum returned length in bytes is determined by
the `group_concat_max_len` server system variable."* `GROUP_CONCAT`
ends on that limit exactly. This ended two bytes past it.
SET SESSION group_concat_max_len = 25;
SELECT LENGTH(JSON_ARRAYAGG(s)) FROM t1;
27
`dump_leaf_key()` tests what it has accumulated once per row. For
`GROUP_CONCAT` that is the whole answer, but this function's brackets
are written later, in `val_str()`, when the group is asked for. So the
row test was made against a body that was not yet the answer, and the
answer came back longer than the body it was tested by.
Where the elements ended exactly on the limit there was nothing to cut
and no warning was raised, so the answer went over in silence - which is
the shape above, `["aaaaaaaaaa","bbbbbbbbbb"]` at a cap of 25.
`Item_func_group_concat` now says how much of the limit is spoken for by
something not in the buffer yet. It answers nothing, so `GROUP_CONCAT`
is not moved by any of this; the array aggregate answers with its two
brackets, which go in through `String::append()` and are as wide as the
narrowest character of the result's set. `JSON_OBJECTAGG` builds its
opening brace into the buffer as the group begins and already kept back
the room for the closing one, so nothing about the answer it hands back
moves either.
The cost is that the last element goes rather than the last two bytes.
A whole-element cut has nothing smaller to give back, so `["ab","cd"]`
at a cap of 9 becomes `["ab"]`, and a cap that no longer reaches the end
of the first element leaves `[]` - which is what a cap too small for any
element already meant here. Nothing stops reading as JSON, and the
warning now names the row that would not fit rather than the one after
it.
## One limit that cannot be kept
`group_concat_max_len` may be set as low as 4, and an empty array is as
short as this function goes. In a set of four bytes to the character
that is eight bytes, so a cap of 4 is answered by `[]` and the answer is
over the cap with nothing left to cut.
That is why `Item_func_json_arrayagg::fix_fields()` carries room for the
brackets on top of the cap in the width it declares. It was written for
the overshoot this commit removes, and it is still needed for this: the
most this function returns is the cap or the two brackets, whichever is
longer. That width asks for the room the brackets take rather than
working it out a second time: it is the same figure the limit keeps
back, and a figure written down twice is one that can be changed in one
place and not the other - which is how the two came to disagree here in
the first place.
## The same floor, and the same width, for the object
`JSON_OBJECTAGG` goes no shorter than `{}` and writes its braces the
same way, and declared a width worked out from the cap and nothing else.
A cap too small for them left a column narrower than the shortest answer
the function gives, and what did not fit was cut into it as the column
was filled. At a cap of 4 in utf32 the column is one character wide,
holds `{`, and no longer reads as JSON:
obj_h obj_ok
0000007B 0 -- the answer was {}, of which this is half
Nothing said so. The note raised alongside is about a row of the group
being cut, which is a different thing and true anyway. It carries the
same allowance now, and how wide a brace is is said in one place, which
the row test below reads as well rather than spelling out its own.
The width is worked out in a type wide enough to hold it. `max_length`
is 32 bits and the cap alone can fill it: a gigabyte, the most
`max_allowed_packet` allows, in a set of four bytes to the character.
An addition made in that width carries round to nothing, and what comes
out is a column too narrow to hold any answer at all - which the array
aggregate's allowance, added in that width, did.
The width the client is told about moves with it, in `type_json` and in
two tests here that record the two aggregates one after the other. Each
had the object short of the array by the room for the punctuation; the
two now agree, which they should, having the same floor and writing the
same two characters round the group.
New test `func_json_agg_cap`, which walks the cap across a whole element
and past the end of the group, over one-byte and two-byte sets, asserts
every answer against the cap it was given, pins the empty-array floor,
and stands `GROUP_CONCAT` beside the two aggregates under one cap. Two
further sections are about the width rather than the answer: a column
made out of both aggregates under a cap too small for their punctuation,
read back as bytes and asked whether it is still JSON, and one made
under a cap that fills the width on its own.
arcivanov
force-pushed
the
json_phoenix
branch
from
August 12, 2026 07:09
1dc58b9 to
6d3b3aa
Compare
The runner can put every statement through a prepared statement, a cursor or a wrapping view, and three of the things these tests do come out differently when it does. 1. **A prepared SELECT is run a second time**, so that the two answers can be checked against each other. Anything counted rather than read counts twice over - the scan counter, a stored function's call count - and a statement that writes to the variable it is reading is asked the second time about a document it has already replaced. `--disable_ps2_protocol` over those sections. 2. **A check constraint is looked at when the statement is prepared**, so the warnings one draws are prepare warnings, which the runner does not show unless it is asked for them. `--enable_prepare_warnings` over the tables that draw one. 3. **A wrapping view is built on a connection of its own**, so it carries neither the session's `sql_mode` nor its connection collation; it keeps its body as printed text, so a boolean literal comes back as the number it equals; and its columns, like a cursor's temporary table, are as wide and as nullable as the expression says rather than as the answer turned out. `--disable_view_protocol` and `--disable_cursor_protocol` over the sections that read result metadata, that set a character set or an `sql_mode`, and over the few answers a declaration does not fit. No recorded result moves: every directive here is inert in the protocol the results were recorded in.
gkodinov
added
the
External Contribution
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It's a working draft PRed to run through the entire BuildBot, including the
ASAN/UBSAN and other sanitizer builders. Not ready for review - please
do not spend time on it yet.
The branch stops the JSON functions from re-parsing and re-validating values
the server itself has just produced: an item attests to the validity, the
niceness and the depth of the value it hands over, and downstream functions
consult those marks instead of scanning the bytes again. A number of genuine
bugs uncovered along the way are fixed in their own commits, each with its
own MDEV and its own tests.
Locally this is green on
main,json,federated(1367 tests) in a Debug build,with the
-DDBUG_OFFandDBUG_ASSERT_AS_PRINTFshapes building clean. Whatthis PR is for is the coverage a local run cannot give: the sanitizer
builders, the other platforms and compilers, and the rest of the suite.
Branched off
1dab253482d; not yet rebased onto the current 10.11 tip.