Skip to content

fix: match Lua 5.1 number rendering and string.format %s/%c/%q semantics - #557

Open
madhavschahar wants to merge 1 commit into
yuin:masterfrom
madhavschahar:fix-lua51-string-format-semantics
Open

madhavschahar wants to merge 1 commit into
yuin:masterfrom
madhavschahar:fix-lua51-string-format-semantics

Conversation

@madhavschahar

Copy link
Copy Markdown

GopherLua renders values to text through Go's formatting machinery (fmt.Sprint in LNumber.String, fmt.Sprintf in string.format), which makes a number of results differ from the reference Lua 5.1 interpreter. This PR makes the rendering core match the reference. Every "reference 5.1" output below was checked against a lua-5.1.5 binary built from the lua.org sources (linux/amd64, glibc).

1. Number → string rendering (tostring, concatenation, table.concat, %s)

Reference 5.1 converts numbers with LUAI_NUMFMT = "%.14g" (luaconf.h), and spells infinities and NaNs the way C printf does.

Lua reference 5.1 gopher-lua before gopher-lua after
tostring(1e15) 1e+15 1000000000000000 1e+15
tostring(2^53) 9.007199254741e+15 9007199254740992 9.007199254741e+15
tostring(0.1 + 0.2) 0.3 0.30000000000000004 0.3
tostring(1/0) / tostring(-1/0) inf / -inf +Inf / -Inf inf / -inf
tostring(0/0) -nan (glibc/amd64) NaN -nan
local a = 0.0; tostring(-a) -0 0 -0

The same rendering is used by concatenation, table.concat, io.write, print and string.format("%s", n), so all of those change together (1e15 .. "" is now "1e+15", etc.).

The -0 row needed a second fix besides the renderer: allocator.LNumber2I maps any value that compares equal to a small non-negative integer onto a preloaded constant, so -0 (which compares equal to 0) was replaced by the shared +0 and silently lost its sign bit before it could be rendered.

2. math.huge

Reference 5.1 defines math.huge as HUGE_VAL, i.e. positive infinity; gopher-lua used math.MaxFloat64.

print(math.huge == 1/0, math.huge > 1e308, tostring(math.huge))
output
reference 5.1 true true inf
before false false 1.7976931348623157e+308
after true true inf

3. string.format: %s, %c and %q

Reference 5.1 implements these conversions through C sprintf and strlen (str_format/addquoted in lstrlib.c), which gives them byte-oriented semantics that fmt.Sprintf does not reproduce:

Lua reference 5.1 before after
string.format("%s", nil) error: string expected, got nil "nil" error
string.format("%s", {}) error: string expected, got table "table: 0x..." error
#string.format("%s", "a\0b") 1 (stops at NUL) 3 1
#string.format("%s", s) where #s == 121 incl. a NUL, no precision 121 (kept whole: a string of 100+ bytes with no precision bypasses the C buffer) 121 121
#string.format("%.1s", "é") (two-byte UTF-8) 1 (precision counts bytes) 2 (counted runes) 1
string.format("%c", 72) H H H
#string.format("%c", 0) 0 (strlen sees nothing) 1 0
#string.format("%c", 256) 0 ((char)256 is byte 0) 2 (emitted U+0100 as UTF-8) 0
string.format("%q", "a\nb") "a\ + real newline + b" "a\nb" (literal \n) "a\ + real newline + b"
string.format("%q", "x\0y") "x\000y" "x\x00y" "x\000y"
string.format("%q", "a\rb") "a\rb" (escaped) raw CR kept "a\rb"

%q output round-trips through loadstring for every byte value 0–255 (raw CR/LF inside a quoted literal would otherwise break the lexer); a test asserts this.

strFormat now scans the format string itself and implements only %s, %c and %q; every other verb (%d, %x, %f, %g, %e, %o, ...) is still delegated to fmt.Sprintf per-directive exactly as before (LNumber/LString implement fmt.Formatter), so their behaviour is unchanged.

Tests

  • Assertions covering each behaviour above were added to _glua-tests/base.lua, _glua-tests/math.lua and _glua-tests/strings.lua. They fail on master and pass with this change.
  • The previously commented-out string.format assertions in _lua5.1-tests/strings.lua (lines 102–105 of the official 5.1 suite, which diverged before) are re-enabled and now pass.
  • go test ./... passes; gofmt clean; go vet output unchanged from master.

Compatibility notes

  • string.format("%s", v) with a boolean, table or nil now raises bad argument as the reference does; callers relying on the old lenient behaviour need tostring(v) (which still works for every type).
  • Numbers with 15+ significant digits now render in exponent form (1e+15 instead of 1000000000000000), matching %.14g.
  • The nan/-nan spelling follows the sign bit, which matches C printf on glibc; the new tests accept either spelling and only assert that negation flips it.

Related: #468 (LNumber.String() going through fmt.Sprint).

🤖 Generated with Claude Code

GopherLua renders numbers with Go's default formatting and implements
string.format by delegating to fmt.Sprintf, so several results differ
from the reference Lua 5.1 interpreter:

- tostring/concat/%s rendered numbers the Go way (1000000000000000,
  0.30000000000000004, +Inf, NaN) instead of the reference's
  lua_number2str "%.14g" (1e+15, 0.3, inf, nan), and negative zero lost
  its sign because the allocator mapped -0 onto the preloaded +0.
- math.huge was math.MaxFloat64 instead of HUGE_VAL (positive
  infinity), so math.huge == 1/0 was false.
- string.format accepted any value for %s/%q, measured %s width and
  precision in runes, kept NUL bytes, and %c emitted a UTF-8 rune. The
  reference formats through C sprintf/strlen, so it accepts only
  strings and numbers, counts bytes, stops %s at the first NUL byte
  (unless the string is 100+ bytes long with no precision, in which
  case it is kept whole), writes %c as a single byte where a zero byte
  adds nothing, and escapes %q output as backslash+newline, \r and
  \000.

Replace LNumber.String with the reference "%.14g" rendering (plus the
inf/-inf/nan/-nan spellings), keep -0 out of the non-negative preload
table, set math.huge to +Inf, and give strFormat a small scanner that
implements %s, %c and %q with Lua 5.1 semantics while still delegating
every other verb to fmt (LNumber and LString implement fmt.Formatter,
so %d, %x, %f, ... behave as before).

Assertions covering each behaviour are added to
_glua-tests/{base,math,strings}.lua, and the previously diverging
string.format assertions in _lua5.1-tests/strings.lua (lines 102-105 of
the official suite) are re-enabled and now pass.

Related: yuin#468

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@kimjooyoon kimjooyoon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked head 6d5c36cc856e0825fcfde8fd00e1332d90696846 and base 75f497656b1c6864139dd2a7d88cf96d09550814 against unmodified C Lua 5.1.5 on macOS arm64. The unpadded NUL %c, wrapped-byte %c, embedded-NUL %s and %q round-trip cases in this sample now match the reference. Two remaining format-parity gaps are detailed inline; both were already different on the base as well.

Reproduction: identical 22-case program, captured byte outputs and pinned sources. No newly introduced reference mismatch was observed in this small sample; it is not a full number-rendering or Lua-conformance review.

Comment thread stringlib.go
Comment on lines +264 to +268
item := ""
if b := byte(n); b != 0 {
item = string([]byte{b})
}
strFormatPad(&out, item, width, leftAlign)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply %c width before truncating its NUL byte

With zero and a field width, removing the byte before padding changes the result: on this head string.format("L%3c|R", 0) returns "L |R", while Lua 5.1.5 returns "L |R". For "L%-3c|R", the reference returns "L|R", but this branch again inserts three spaces. The same applies to 256 after the byte conversion. Lua pads the one-byte character with sprintf first, then takes strlen of that formatted item. Please pad the one-byte item before appending its prefix up to the first NUL, and cover both alignments. The base also differed from Lua here; this is a remaining gap in the NUL handling rather than a new regression.

Comment thread stringlib.go
Comment on lines +199 to +205
width := -1
for i < len(str) && str[i] >= '0' && str[i] <= '9' {
if width < 0 {
width = 0
}
width = width*10 + int(str[i]-'0')
i++

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve Lua 5.1's two-digit format limits

The scanner accepts a third width digit here and a third precision digit in the loop below. Consequently pcall(string.format, "%100s", "xy") and pcall(string.format, "%.100s", "xy") both succeed on the head; C Lua 5.1.5 rejects them with invalid format (width or precision too long). The %99s and %.99s controls succeed on both. Please mirror scanformat's two-digit limits for width and precision and add these boundary cases. This is also an existing compatibility gap on the base; the new scanner has not closed it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants