Tcl Source Code

View Ticket
Login
Ticket UUID: 7bad415e3ec8baf40d087374000fbcbab511689c
Title: Free-form clock scan reads "+02" as two minutes in 8.6 (backport of [51aa53616067cb63] without semantic predicates); the 9.x fix misreads a zone followed by a number
Type: Bug Created on: 2026-09-03 19:55:01
Submitter: kenneth.kienle Assigned to: nobody
Subsystem: 06. Time Measurement Severity: Important
Priority: 5 Medium Last modified: 2026-09-10 22:28:45
Status: Open Closed by: nobody
Resolution: None Closed on:
Version: 8.6.18, 9.0.4
Description:

8.6: the bug of [51aa53616067cb63] is still there. generic/tclGetDate.y's rule zone: sign tUNUMBER computes -$1*($2 % 100 + ($2 / 100) * 60), so every signed number is hhmm and +02 is two minutes. The tz project's abbreviations for zones without a letter name are +02, -03, +0530 (hh or hhmm), so any zoneinfo-derived %Z of that form round-trips wrongly through the free-form scanner, 7,080 s off for +02:

% clock scan "2023-11-15 00:13:20 +02" -gmt 1
1700007080          ;# expected 1700000000
% clock scan "2023-11-15 00:13:20 +0200" -gmt 1
1700000000

Tcl's own suite cannot see it because tools/tclZIC.tcl pads every %z abbreviation to %+03d%02d; a build reading system zoneinfo fails clock-59.2 on exactly this, which is how [51aa53616067cb63] was found. That ticket was fixed for 9.x in [a37536d4d7] and [7018ffb43e] and closed with: "I am unsure I shall backport it to 8.6, but if yes theoretically it may be possible only partial, because I don't know earlier bison/yacc versions support semantic predicates." 8.6.18 (2026-05-11) shipped without it. The attached backport uses no semantic predicate, only a plain action, so it compiles with whatever bison built the shipped tclDate.c.

9.x: the fix takes the wrong digit count when a number follows the zone. yyDigitCount is the lexer's count for the last number it read. The parser cannot reduce sign tUNUMBER without a lookahead (the same prefix opens sign tUNUMBER tDAY and the relative sign tUNUMBER unit), and when the lookahead is itself a number the count already belongs to it. So nmzone reads +02 2023-11-15 by 2023's four digits, and reads +0200 15 Nov 2023 by 15's two, which 8.6.18 gets right:

% clock scan "+0200 15 Nov 2023 00:13:20" -gmt 1
1699935200          ;# 9.0.4; 8.6.18 returns 1700000000

Fix (attached clock-freescan-hh-zone-lookahead-safe.patch, 30 lines against core-8-6-branch, tclGetDate.y and the shipped tclDate.c alike): one or two digits are hours, as 9.x rules; and the lexer keeps the previous count (datePrevDigitCount, saved before each overwrite) so the action can choose by the lookahead's token type — yychar, bison's documented action variable, against tUNUMBER and tISOBASE, the two tokens the number path produces:

int digits = (yychar == tUNUMBER || yychar == tISOBASE)
        ? yyPrevDigitCount : yyDigitCount;
yyTimezone = -$1*(digits <= 2 ? $2 * 60 : $2 % 100 + ($2 / 100) * 60);

Both counts are initialized with the rest of DateInfo. The same idea transfers to 9.x's nmzone.

Verification: the attached tcl-tz-repro.tcl (also on [832d098fdfec8f004cf1c8e8836f7c335da12d54]) has seven clock scan checks: stock 8.6.18 fails four (+02, -02, +3, and +02 2023-…), stock 9.0.4 fails two (+02 2023-…, +0200 15 Nov …), patched 8.6.18 none; +2 days and +14 days relatives unchanged; clock-59.2's six pairs round-trip against system zoneinfo; tests/clock.test shows no new failure. Proposed: the seven lines as a clock-59.x block beside clock-59.2.

User Comments:
oehhar added on 2026-09-04 12:20:23:

Kenneth, thanks for the proposals, look great.

Sergey the clock wizard will hopefully comment.

Thanks for all, Harald


sebres added on 2026-09-04 12:25:27:

Although, IMHO, it is not standard order for a timestamp (TZ before time), I am unsure free-scan really shall scan that, but I'd take a look at this later.


kenneth.kienle added on 2026-09-08 05:59:54:

Sergey,

You are right about the ordering.

The zone-first cases are now in their own test, clock-59.5. If you decide free-form scan should not accept a zone before the time, delete that one test and nothing else changes. That test is the only coverage of the tISOBASE half of the lookahead check. I measured this - remove the tISOBASE clause and clock-59.5 is the only test that fails. So if the zone-first order goes, that clause can go with it.

How this came up: I build and test Tcl 8.6 against the system's tzdata rather than the bundled copy. Running the 8.6 suite in that configuration is what surfaced it.

The main bug is the trailing form:

clock scan {2023-11-15 00:13:20 +02} -gmt 1

Stock 8.6.18 returns 1700007080. The correct value is 1700000000. A signed zone is read as hhmm whatever its length, so "+02" becomes two minutes. That is how the system zoneinfo writes the %Z abbreviation, so clock format output does not survive a round trip through clock scan (Tcl's bundled tzdata writes "+0200" for %Z, which is why clock-59.2 passes today. %z is always four digits and is never affected).

The clearest demonstration is to build 8.6 without bundled tzdata and run clock.test. Stock fails clock-59.2, and with this branch it passes. On a system whose zoneinfo also lacks CET, US/Eastern and US/East-Indiana you will see clock-30.33, 30.34, 39.1 and 44.1 fail as well, in stock and in the branch alike. Those are missing-zone errors from the tzdata install rather than anything to do with this bug, and I should not have listed them alongside 59.2.

The branch is bug-7bad415e3e-8.6. It uses no semantic predicate, because your comment on [51aa53616067cb63] raised the older-bison question. The existing "sign tUNUMBER" action decides hours or hhmm itself. When the parser holds a numeric lookahead, it reads the digit count recorded before that lookahead. tclDate.c was regenerated with bison 3.8.2 by gendate. 14 shift/reduce conflicts, the same as stock.

The branch also initialises the two digit-count fields in TclClockOldscanObjCmd. That function sets every other yy* field before yyparse but not yyDigitCount, and "number: tUNUMBER" reads it at the "yyDigitCount <= 2" test, which is a path "last" reaches because LookupWord returns it as tUNUMBER without passing the digit scanner. So stock reads an uninitialised stack value there.

The read is real and I can move it, but only from C. Seeding yyDigitCount with 3 or more before the parse changes

clock scan {last} -base 1700000000 -gmt 1

from 1699916400 to 1699919940. Seeding yyPrevDigitCount alone changes nothing: the keyword path assigns that field and never assigns yyDigitCount, which is the one "number: tUNUMBER" reads. What I could not do is reach it from a script - stock as shipped gives 1699916400, and so does stock after a preceding clock scan of {1234}, {20231115 001320}, {12} or {1130}, because the stack slot does not carry over. Because of that, I have written no test. Without editing the C there is no build on which one would discriminate, so it would pass either way and be a false control.

One behaviour change from the lookahead work itself:

clock scan {+last} -base 1700000000 -gmt 1
    stock 1700000060    this branch 1700003600    9.0.5 1700003600

That one is not the initialisation, removing those two lines leaves it at

  1. The guard is yyDigitCount <= 2, so every negative value behaves exactly like 0, and I could not produce a build where the removal alone was visible, including one compiled with -ftrivial-auto-var-init=pattern, which changes nothing because the fill is negative as an int (0xFEFEFEFE under gcc, 0xAAAAAAAA under clang). Seeding the field with 3 or more does give stock's
  2. The read is real, and it is undefined behaviour either way. The removal is quiet because of what the caller's stack and the compiler happen to leave, not because the lines do nothing. With the lines removed the slot arrives as 0 on plain -O2 and as 0xFEFEFEFE under the pattern flag, both on the <= 2 side. The branch as submitted initialises the field and returns 1700003600 for that reason rather than by luck, which is the argument for keeping those two lines.

Two measurements on 9.0.5, in case they are useful to you:

clock scan {2023-11-15 00:13:20 +02} -gmt 1        -> 1700000000  correct
clock scan {2023-11-15 00:13:20 +02 2024} -gmt 1   -> 1731629480  7080 s off

The second should differ from the first only in the year. This branch gives 1731622400, which is what glibc gives for the same instant written as a date it accepts: date -u -d '2024-11-15 00:13:20 +02' +%s. So 8.6 with this change and 9.0.5 disagree on a zone followed by a number. I think 9.x is the one to correct, I just didn't want the difference to be a surprise.

My first version of this patch had the same kind of bug I am describing in 9.x: "last" is returned as tUNUMBER by LookupWord without passing the digit scanner, so "clock scan {+0200 last month}" regressed against stock. That is fixed and clock-59.6 pins it. A later check found that my ISO-base test no longer covered the clause it was written for, because I had changed it to a four-digit zone. That is fixed too. Both errors were mine.

There are three related pre-existing issues I did not touch, I'm happy to file them separately if that is useful. I checked each against 9.0.5.

  • "number: tUNUMBER" reads yyDigitCount with the same lookahead problem. clock scan {1130 15 Nov 2023} -gmt 1 gives 1704074400 in stock 8.6, where GNU date gives 1700047800. 9.0.5 raises an error on that input rather than answering it.
  • the non-J "n" day-of-year form is one day out. DeterminePosixDSTTime puts the 0-based n into the 1-based dayOfYear field. This one is in 9.0.5: with EST5EDT,59,300 in 2032, glibc starts DST on 29 February and 9.0.5 on the 28th, and n=58 and n=60 are a day early there as well.
  • <zone> tomorrow parses through a different rule and is an issue on stock. 9.0.5 is correct, {+0200 tomorrow} and {tomorrow +0200} both give 1699999200 there, where stock 8.6 gives 1717200000 and 1699999200.

Kenneth


kenneth.kienle added on 2026-09-08 06:09:53:
Two numbers dropped out of my previous comment. The renderer treated a line
beginning "1700003600." and one beginning "1700000060." as ordered-list
markers and consumed them. The sentences should read:

    That one is not the initialisation, removing those two lines leaves it
    at 1700003600.

    Seeding the field with 3 or more does give stock's 1700000060.

Both values are in the table further up the comment, so nothing is
ambiguous, but the sentences read oddly without them. Nothing else changed.

sebres added on 2026-09-10 22:28:45:

Well, I'm still unsure how to handle this...

I'd like to avoid change of 8.6 to cover this obscure case (ordering etc), due to compat reasons et al (don't underestimate the inventiveness of users in usage of free-scan formats).

For 9.0 I'd probably partially cherry-pick this (at least the tests), but need more time to review. For example, I don't understand why the test successfully covers free-scan conversion of {2023-11-15 00:13:20 +02 2024}... IIRC, free-scan shall fail here with "more than one date in string" error (due to 2 years in input string). Another thing is that the lookahead supposedly increases yyDigitCount, what makes no sense to me (or rather it shall not at all).


Attachments: