Tcl Source Code

First, have a look at Tcl 8.7:

$ tclsh8.7
% clock scan "2012-06-30 23:59:59" -gmt 1
1341100799
% clock scan "2012-06-30 23:59:60" -gmt 1
1341100800

Now Tcl 9.0:

$ tclsh9.0
% clock scan "2012-06-30 23:59:59" -gmt 1
1341100799
% clock scan "2012-06-30 23:59:60" -gmt 1
unable to convert input string: invalid time

Finally, Tcl 8.6:

$ tclsh8.6
% clock scan "2012-06-30 23:59:59" -gmt 1
1341100799
% clock scan "2012-06-30 23:59:60" -gmt 1
1341014399
% clock format 1341014399 -gmt 1
Fri Jun 29 23:59:59 GMT 2012

> However I'm still unsure whether the leap second need to be considered as valid (and my branch need to be merged). But I'm definitely against your solution, that you merged to 8.7/9.0 despite all objections from my side.

I was hoping to be able to come to an agreement before Tcl 9.0.1 (which will be there soon). It's not going to happen.

Given your doubts and my objections to your overly-complex solution, I'm not going through with it. Eventually there should be a --enable-leap option, which follows POSIX 'enhanced' timestamps (which follow leap-seconds and leap-minutes), but such a standard doesn't exist yet. I will revert the changes.

Still, we should document that leap-seconds are not supported. TIP #690 didn't mention that in its "compatibility" section. So, it should be documented now. Only "24:00" should be kept: It's clearly documented by the ISO 8601 2022 amendment as belonging to the next day. We should document that as well.

% clock scan "2012-06-30 23:59:60 UTC"
1341100799
% clock scan "2012-06-30 23:59:60 UTC" -valid 1
1341100799

% clock scan "2012-06-29 23:59:60 UTC"
1341014400
% clock scan "2012-06-29 23:59:60 UTC" -valid 1
unable to convert input string: invalid time

> And I don't stop to ask you, why you are in mode to continue to prefer a false result to no result (or even to right result, like in my branch)?

There is no 'right' answer here. Returning 1341100799 is as wrong as 1341100800. What UNIX does is smearing out, so we could map 23:59:59 to 1341100799, 23:59:60 to 1341100799.5 and 00:00:00 (next day) to 1341100800. But we currently don't support fractions, so we have to round down (your propsal) or up (my proposal, and current Tcl 8.x behavior). Returning a different timestamp for -valid 1 and -valid 0 is not logical: -valid should only influence the validation, nothing else.

The challenge is providing a consistent behavior. I'm doing that by assuming a second value of "60" to be valid unless the time string contains something which disallows this. Such as a date which is not jan 1, june 30, july 1 or dec 31. Or when the (minute % 15) != 14. Or (recently) when the year is < 1972 or > 2017.

% clock scan "23:59:59"
1732316399
% clock scan "23:59:60"
1732316400
% clock scan "00:00:00 tomorrow"
1732316400

This makes 100% sense to me, even though I know today we don't have a leap-second.

In 2035, the current leap-second system will be abandoned, maybe replace by leap minutes. Until then no positive leap-seconds are expected, only a negative leap-second. That means the current list of 27 entries is final: there won't be more of them any more. Do we really need to generate this list with Tcl-code, and then import it? Wouldn't a simple static array in C suffice?

It also means that the risk Nathan is describing is non-existent: Leap seconds only existed between 1972 and 2016. For current timestamps they are illegal already. What's the risk? Tcl 8.6 already accepted leap-seconds (even though the POSIX timestamp was wrong, which is now corrected).

I now added a year-check to the code in core-8-branch and trunk. I really think it doesn't need to be much more complicated than that.

Nice patch. However, this might still be introducing an extra complexity for no good reason. If a script is using clock scan with -validate in order to validate a time representation, and after validation stores the original representation into a persistent database, all other consumers of the data must then be prepared to deal with leap second representations. Sqlite, for example, is not:

sqlite> select date('2012-06-30 23:59:00');
2012-06-30
sqlite> select date('2012-06-30 23:59:60');

sqlite>

So in this case another consumer must be prepared for a time representation that was already validated by Tcl to evaluate to an empty string in an SQL script. This seems like a footgun. Sebres, despite the time that you invested in preparing this patch, is it really a good idea? If the answer is yes, and this is going to go into Tcl, then given that most applications don't need and don't want to deal with leap seconds, should an option like -leapseconds be added to activate this behaviour?

Is the new option -strict-leap-sec really necessary? Given that parsing leap second representations is somewhat niche, maybe anyone that wants that functionality should just accept the cost of calculating strict leap seconds.

I still think that leap seconds are something for a system that calculates clock time to be aware of, but not to represent.

This was my first assumption too, but it was based on:

• the complexity (or rather the relation between the complexity and needs of such scan)
• the effort to implement it properly, as it were, to comply the ISO 8601 and OS behaviour (freeze the second) at the same time;
• the common sense on top (leap seconds is not really representable in result of scan, at least as long as it remains unix-time) and the fact that it has been implemented quasi nowhere.

However, I think I managed it in-between, here is the branch that shall satisfy all of them: bug-f2b5f89c0d-alt-leapsec.

The pros:

• leap second detected more precise, because estimated from UTC tokens or/and complying real leap-second table of iana;
• leap second remains in the same calendar day as previous second;
• almost no performance loss (excepting the single load of leap-sec table once per process); not affected at all for all times excepting Thh:mm:60; and it uses very optimal algorithm (interpolation binary search, IBS), which checks whether the second is leap or not, in optimal case with single, mostly with 2, maximally with 3 iterations in the table, so it is almost to neglect;
• besides the scan result of leap second covers ISO 8601 standard more sane and it is also more similar to OS behaviour (long, frozen second Thh:mm:59), it also matches better to the input string (e. g. all tokens excepting the second remain the same - calendar year, month, week of year, day, weekday, hour and minute are the same as specified in input or are in accordance with the timestamp of leap second);
• with the table it'd be able to capture every leap second (after update with tclZIC from iana database), even if it will not follow the current rules, e. g. belong to different dates than "31 Dec" or "30 Jun";
• apart from the table, it can also estimate the leap second ("31 Dec" or "30 Jun", "T23:59:60 UTC"), what will detect possible future leap second without an update (of "tzdata/.leapsec");
• on demand, it is possible to disable estimation of leap second, so it'd only inspect the leap table.

The cons:

• unknown, excepting the deviation to 8.6 (but before the fix [595fad24d70e1693] it never worked properly, so it does no matter, in my opinion);
• a bit grown code base.

I believe it is safest to not to provide for the representation of leap seconds.

At https://www.cl.cam.ac.uk/~mgk25/posix-clocks.html, Markus Kuhn states, "Later I came to the conclusion that attempting to present inserted leap seconds as an out-of-scale timestamp of the form 23:59:60 to applications at an operating-system-API or network-protocol level is not very useful for most applications and that a standardized smoothed form of UTC, such as my UTC-SLS proposal (2006), is far more practical in almost all applications, except for those concerned with precisely tracking the motion of physical masses, where TAI is a more useful timebase."

This, along with the position of PostGreSQL that sebres pointed out, is further evidence that the best policy is not to try to accept and handle representations of leap seconds. I.e. Tcl should behave as follows:

% clock scan "2012-06-30 23:59:60" -gmt 1
unable to convert input string: invalid time

No matter formatted or not, your variant produces a result which going to the next day, month or even year, what is definitely wrong. According to ISO 8601 the leap second shall remain in the same calendar day! Your leap seconds moves to next calendar day, month or even year. This is neither expected result, nor correlates with the standard.

All the time I told you about the problematic or its representation, and you ignored it. Now you just tell - don't bring format into play, to justify your mistake. The problem is not the format, but the fact that the unix-time you generated is in the next calendar day. That's it.

How one can ignore format (or whatever action with resulting unix-time), if it is a target of conversion? Your approach may work only if someone will check whether the given time is not rejected, but would not work if the result of scan used somewhere in further calculations.

Imagine: someone needs to obtain day of week, week of year, day, month or year from the result of scan operation - everything this will be incorrect for this input. And, again, not conform to the standard too and to the handling of the OSs.

Either this must be fixed (remains the same second than "23:59:59 UTC", how also the OSs will do by such seconds), or it must be reverted and leap second is then basically invalid as before by scan operations.

And you continue to prefer a false result to no result.

> No, you did not

Yes, I did. Please don't bring "clock format" into play, because it doesn't produce ISO-8601 dates but POSIX dates, starting with POSIX timestamps. POSIX doesn't know anything about leap-seconds, so this is expected.

POSIX is a subset of ISO-8601. Since we are on UNIX, and POSIX timestamps may be used when interfacing with the OS, we cannot change the behavior of "clock format". We cannot re-define POSIX timestamps such that days are not a multiple of SECONDS_PER_DAY any more.

% clock format [clock scan "Jun 30 23:59:60 GMT 2012"] -gmt 1
Sun Jul 01 00:00:00 GMT 2012
% clock format [clock scan "Jul 01 01:59:60 CEST 2012"] -gmt 1
Sun Jul 01 00:00:00 GMT 2012

% clock format [clock scan "Jan 01 00:59:60 CET 2017" -gmt 1] -gmt 1
Sun Jan 01 00:00:00 GMT 2017
% clock format [clock scan "Dec 31 23:59:60 GMT 2016" -gmt 1] -gmt 1
Sun Jan 01 00:00:00 GMT 2017

% clock scan "Jan 01 23:59:60 GMT 2013"
1357084800

> There are many corner cases where it'd not work, e. g. try "Fri Jun 29 23:59:60 GMT 2012" (shall fail, not a leap second), or where the relative time scanned together with leap second.

<pre>
    $ tclsh9.0
    % clock scan "Fri Jun 29 23:59:60 GMT 2012"
    unable to convert input string: invalid time
    % 
</pre>

As expected

> And your implementation, as already said, switches to the next calendar day (and next calendar month or even still worse to next calendar year)

No, it doesn't do that any more. I found a solution for that. ;-) https://core.tcl-lang.org/tcl/file?name=generic/tclClock.c&ci=d3744ffffc620ecc&ln=3736-3737

In case of a leap-second or "24:00", yySecondOfDay can become equal to SECONDS_PER_DAY, without jumping to the next day any more. That already solves all corner-cases related to jumping to the next calendar day/month/year.

> Let's try to fool it:
> % clock scan "Sun Jun 30 23:59:60 GMT 2012"
> unable to convert input string: invalid day of week

There are many corner cases where it'd not work, e. g. try "Fri Jun 29 23:59:60 GMT 2012" (shall fail, not a leap second), or where the relative time scanned together with leap second.

> The "ACST" timezone is - apparently - unkown to the parser

Yes, by TZs scanned from input string it is still a bit inconsistent (let alone they are hardcoded). At some time I prepared a solution for that, removing special coded handling for TZ and considering tzdata instead, just it resulting to many further issues, one of them is https://github.com/sebres/tclclockmod/issues/22#issuecomment-1183083552 I'll try to reopen it later.

> Thanks for all your input!

You are welcome... Unfortunately, it seemed like you read only half of it. Just kidding...
I'll provide my fix soon, that closes many corner cases, and important one - result of scan remain in same day by leap second (how all OSs do it with the unix-time, therefore this will reflect it more consistently).

> I don't think the ISO 8601 standard changes that often. Now we are conformant to the 2022 version.

The emphasis was not on ISO-change, but on its hardly possible representation in unix-time. But if we are talking about standards, here is a quote from it:

And your implementation, as already said, switches to the next calendar day (and next calendar month or even still worse to next calendar year).

Closing, since valid leap-second timestamps are now handled as being valid.

> I'm trying to implement this properly (see my new branch bug-f2b5f89c0d-alt-leapsec) by consulting the leap seconds table with converted UTC time

Sure, some non-valid timestamps are accepted as valid now. But - still - the current approach is useful even for an additional table-based check: If yyMinute % 15 != 14 (for example) or the date is not January 1st, June 30th, July 1st or December 31th, there is no need even to consult the table. That's already handling 99.93% of all cases!

> ISO can change something every year, and one can and probably must reflect it on the scanning of the input side, but not if the result of conversion is impossible in principle.

I don't think the ISO 8601 standard changes that often. Now we are conformant to the 2022 version. If - in the future - "leap minutes" are introduced, we will need to change the implementation again. For now, the 2022 version is the best we can do.

Thanks!

Thanks for all your examples! I know it's complicated, but it appears to be working now (I always like a challenge ...):

    $ tclsh9.0
    % clock scan "Sat Jun 30 23:59:60 GMT 2012"
    1341100800
    % clock scan "Sun Jul 01 01:59:60 CEST 2012"
    1341100800
    % clock scan "Sat Jun 30 19:59:60 EDT 2012"
    1341100800
    % clock scan "Sun Jul 01 09:29:60 ACST 2012"
    unable to convert date-time string "Sun Jul 01 09:29:60 ACST 2012": syntax error (characters 20-23)
    % clock scan "Sun Jul 01 08:44:60 +0845 2012"
    1341100800

Let's try to fool it:

    % clock scan "Sun Jun 30 23:59:60 GMT 2012"
    unable to convert input string: invalid day of week

The "ACST" timezone is - apparently - unkown to the parser, but all of them give the right answer now.

Thanks for all your input!

Maybe it is not clear, so for the record, just to emphasize the complexity of the problem - the time "23:59:60", as well as the dates "31 Jan" and "30 Jun" are only timestamps in UTC, in other time zones they are completely different, for instance the leap second "Sat Jun 30 23:59:60 GMT 2012" corresponds "Sun Jul 01 01:59:60 CEST 2012" and "Sat Jun 30 19:59:60 EDT 2012", but also "Sun Jul 01 09:29:60 ACST 2012" and especially nice "Sun Jul 01 08:44:60 +0845 2012" (:Australia/Eucla). All this timestamps represents the same unix time and the same leap second.

This also explain why the usage of "24:00:00" or "23:60:00" is not practicable (and I never saw that before), because it makes sense only for +0000/GMT/UTC time zone, for any other zone (depending on its offset) it may be completely "valid" time, that doesn't reflect the leap.

After all the approach to validate it in the way like in branch bug-f2b5f89c0d is hardly possible, because all the scan tokens are always local time, and the check would be (pseudo) correct only for GMT/UTC and wrong for any other TZs.

I'm trying to implement this properly (see my new branch bug-f2b5f89c0d-alt-leapsec) by consulting the leap seconds table with converted UTC time.

Again, this ISO amendment only affects the input side, the output of clock scan is and still remains the unix time, what simply doesn't know leap second, and so unable to represent it correctly at all.
ISO can change something every year, and one can and probably must reflect it on the scanning of the input side, but not if the result of conversion is impossible in principle.

I don't understand why you prefer a wrong result to the impossible result in case of -validate 1.

Validity is basically a new mechanism to control whether the input is correct, but also and probably more important that the result is correct and the input was converted properly. And initially I implemented it in the way that the result was converted back and compared with input tokens and validation failed if the tokens deviated (what would happen in this particular case)... Later I reimplemented it more optimal by direct comparison, but it shall not change the principle - if back conversion generates different tokens than input, the simplest math doesn't work and the validation must fail. In my opinion the leap second is not such important thing, for which we shall make exceptions in this simple mechanic.

I guess, there are also another corner cases what your branch doesn't consider at the moment and they could fail, for instance the check of the scanned weekday would probably fail (because of the day change, e. g. Sat to Sun):

% clock scan "Sat Jun 30 23:59:60 2012" -gmt 1 -valid 1
unable to convert input string: invalid day of week
% clock scan "Sat Jun 30 23:59:60 2012" -format "%a %b %d %T %Y" -gmt 1 -valid 1
unable to convert input string: invalid day of week

In my opinion, a note in the documentation like this would be fully enough:
Since leap second is not representable in the result of scan (unix time), scan of time like 23:59:60 is generally impossible with -valid 1 (default in 9.0).
That's it. Let alone that all programming languages doing it also.

However if the decision will be nevertheless "let it be", I would recommend another variant to fix it - freeze the second in the same date: e. g. if the scanned second is 60 and date matches leap conditions, decrement time, so we'd remain in the same date and every validity rules, covering date tokens would be correct.

I found a Preview of the latest amendment (2022) to the ISO-8601 specification. It states:

NOTE 6 The last instant of the day exists within the last second of the day, which can be one of the 
following:
      with no leap second 23:59:59
      with a negative leap second 23:59:58
      with a positive leap second 23:59:60

So, seconds can be "60", but only in case of a positive leap second. Negative leap seconds didn't ever occur in history.

EXAMPLE 2 The following are valid reduced precision expressions of the ending of the day:
—   ‘T24’ (time of day expression with omission of minutes and seconds);
—   ‘24’ (time of day expression with omission of minutes, seconds and time designator);
—   ‘T24:00’ (time of day expression with omission of seconds);
—   ‘24:00’ (time of day expression with omission of seconds and time designator)

This means that a hour value of "24" is possible, as indication of the end of the day, but only when the minutes, seconds and milliseconds are all zero.

Since free-scan of 8.6 was indeed wrong (regardless the conclusion of this issue with leap second), because of day decrement caused by -1 from ToSecond(), I fixed it now by backporting the fix from tclclockmod (8.7) in [595fad24d70e1693], additionally it covers now formatted scan too. So now it works with the same result as formatted scan (also the adjustment of test clock-46.6 was necessary, so it is also cherry-picked from tclclockmod and it is equal now to 8.7+).
And [92493234f7da30f0] illustrates the coverage change for 8.7/9.0 - the same tests as 8.6 with a constraint valid_off and additional tests (leap in wrong days) with constraint !valid_off, inclusive formatted scan for all tests too.

However (just for the record) all the changes are not affected by the decision about this (and are pure fixes and regression tests), I'm still against considering the leap second as valid input.

Noting that you are constantly refering to UNIX or POSIX time, where leap seconds don't exist. I'm refering to ISO 8601 time, where "60" are valid seconds and "24:00" is a valid time (refering to 00:00 the next day). Both are well-defined in ISO 8601. How can we resolve that? We can discuss for hours whether 140 km/h is a valid speed on highways. For you (Germany) it is, for me (Netherlands) it isn't ;-)

So, should the "clock" command implement/accept POSIX time or ISO 8601 time? That's the fundamental question

% clock format 1711846799 -format "%Y-%m-%d %H:%M:%S" -timezone :Europe/Berlin
2024-03-31 01:59:59
% clock format 1711846800 -format "%Y-%m-%d %H:%M:%S" -timezone :Europe/Berlin
2024-03-31 03:00:00

% clock scan "2024-03-31 02:00:00" -format "%Y-%m-%d %H:%M:%S" -timezone :Europe/Berlin
unable to convert input string: invalid time (does not exist in this time-zone)
% clock scan "2024-03-31 02:59:59" -format "%Y-%m-%d %H:%M:%S" -timezone :Europe/Berlin
unable to convert input string: invalid time (does not exist in this time-zone)

> And if someone really needs to parse the textual form "23:56:60" (e. g. in the way you trying to do that), it'd be possible either using own pre-processor or just with a `clock scan "23:56:60" -valid 0`. That's it.

Yes, that's the only thing I want. You are making it much too big an issue. Do you really want to extend the migration notes with this information? Using "-valid 0" would allow 23:99:99 too, it's not a solution I'm recommending.

$ TZ=GMT date -d @1483228800 '+%Y-%m-%d %H:%M:%S'
2017-01-01 00:00:00
$ tclsh <<<'puts [clock format 1483228800 -f {%Y-%m-%d %H:%M:%S} -gmt 1]'
2017-01-01 00:00:00
$ python -c 'from datetime import datetime; print(datetime.utcfromtimestamp(1483228800).strftime("%Y-%m-%d %H:%M:%S"))'
2017-01-01 00:00:00

$ TZ=GMT date -d '2017-01-01 00:00:00' '+%s'
1483228800
$ TZ=GMT date -d '2016-12-31 23:59:60' '+%s'
date: invalid date ‘2016-12-31 23:59:60’

$ python -c 'from datetime import datetime; print(datetime.fromisoformat("2017-01-01 00:00:00+00:00").timestamp())'
1483228800.0
$ python -c 'from datetime import datetime; print(datetime.fromisoformat("2016-12-31 23:59:60+00:00").timestamp())'
ValueError: second must be in 0..59

Let's correct the distinction between "valid" and "invalid" first. I'm OK (for now) with the behavior of 8.7, I'm aware of the imperfections/limitations of that.

My goal is to let valid string representation go through with the conversions, and detect as much invalid situations as possible. The actual conversion output like 8.7 is reasonable, and can be handled separate from this ticket.

Possible fix [4d72246575|here]. Since "leap minutes" could be introduced in the future (even leap hours, but that looks far away), we could prepare for that already.

The ISO 8601 standard allows leap seconds and leap hours, but no leap minutes.