Tcl Source Code

View Ticket
Login
Ticket UUID: 3544685
Title: Threaded build failures on OpenBSD-current
Type: Bug Created on: 2012-07-16 17:50:18
Submitter: stwo Assigned to: stwo
Subsystem: 49. Threading Severity: Minor
Priority: 8 Last modified: 2026-10-22 17:19:27
Status: Closed Closed by: serhiy.storchaka
Resolution: Fixed Closed on: 2026-10-19 11:58:16
Version: 9.0.5, 9.1.0
Description:
Trunk and 8.5.12rc0.

async-4.3 fails on i386 and amd64 (sporadic) "Test file error: child killed: SIGABRT"
interp-36.7 fails on i386 and amd64
event-14.1 fails on i386 (sporadic), difficult to reproduce, try repeatedly running the event tests with minimal delay between runs. I am currently able to reproduce this one with trunk but not 8.5.12rc0.
User Comments:
dgp added on 2012-07-18 00:20:42:
What does the interp-36.7 failure look like?

stwo added on 2012-07-19 11:20:20:
This is with 8.5.12rc1.
I can't get info for async-4.3 anymore since it just SIGABRTs, I guess, but iirc the reason for failure was similar.

==== interp-36.7 SlaveBgerror sets error handler of slave [1999035] FAILED
==== Contents of test case:

    slave eval {
        variable done {}
        after 0 error foo
        after 10 [list ::set [namespace which -variable done] {}]
        vwait [namespace which -variable done]
    }
    set result

---- Result was:
untouched
---- Result should have been (exact matching):
foo
==== interp-36.7 FAILED

dgp added on 2012-07-19 20:57:36:
so, timing sensitivty could be an issue.  Does it help to
change the "after 10 ..." to "after 200 ..." ?

stwo added on 2012-07-19 22:00:06:
interp-36.7 passes if "after 10 ..." is changed to "after 200 ..."

stwo added on 2012-07-19 23:03:24:
Re async-4.3: There may be a specific problem with async-4.3 but for now the SIGABRT comes from other code in async.test which can be boiled down to:
$ make runtest
% testasync delete [testasync create q]
Abort trap (core dumped)

stwo added on 2012-07-19 23:48:02:
The async test problems appear to be caused by an imperfect backport resulting in a mismatched mutex lock/unlock pair (lock was missing). Bug 2981154, checkin 15a55ecb19.

nijtmans added on 2012-07-20 00:34:33:
Looks like you'r right. Fix committed to branch bug-3544685. Please test.

ferrieux added on 2012-07-20 00:39:14:
Sorry, committed to trunk in the meantime. The chat is a nice place to synchronize ;)

nijtmans added on 2012-07-20 00:43:36:
No problem. I first assigned this issue to me, because it was still assigned to Joe.

Now assigned to you, for handling it further.

ferrieux added on 2012-07-20 00:44:01:
Of course, by "trunk" I mean "tip", of core-8-5-branch.

nijtmans added on 2012-07-20 00:47:24:
> Of course, by "trunk" I mean "tip", of core-8-5-branch.

Sure

ferrieux added on 2012-07-20 00:48:30:
Hum, this only fixes the elephant in the front chair. What's the status of the other failures reported here ?

nijtmans added on 2012-07-20 00:54:04:
Stuart is the best person answering that first. That's why I put it in a branch ;-)

ferrieux added on 2012-07-20 01:08:31:
The fix was required, regardless of other lurking nasties. That's why I put it in the main 8.5 line.
Anyway Stu, please update us now.

stwo added on 2012-07-20 01:57:10:
Async tests all pass now; the mutex lock was the thing needed, thanks.
With the latest 8.5 trunk I just got an event-12.4 failure which looks like the other event failure and the interp one as well. Is it just a matter of the tests or is there a problem? I've never seen these errors in a non-threaded build.
The event-12.4 failure looks like this:
==== event-12.4 Tcl_UpdateCmd procedure FAILED
==== Contents of test case:

    foreach i [after info] {
        after cancel $i
    }
    after 10; update; # On Mac make sure update won't take long
    after 200 {set x x-done}
    after 600 {set y y-done}
    after idle {set z z-done}
    set x before
    set y before
    set z before
    after 300
    update
    list $x $y $z

---- Result was:
x-done y-done z-done
---- Result should have been (exact matching):
x-done before z-done
==== event-12.4 FAILED

ferrieux added on 2012-07-20 04:48:11:
Thanks to a ktrace output of a similar failure for event-11.4, here's a detailed analysis:

event-11.4 is:

    after 10; update
    after 100 {set x x-done}
    after 200 {set y y-done}
    after 300 {set z z-done}
    after idle {set q q-done}
    => vwait on y and expect z to be not done yet

the test fails because the pthread_condwait from 100 to 200 actually sees 500ms elapsed.
As a consequence, the Tcl core immediately fires all due timers, including z.

So my take is that on this system, mysterious chunks of 400ms get stolen by the system clock, or the PCLOCK_REALTIME mode of pthread_condwait is awfully buggy.

ferrieux added on 2012-07-20 05:01:24:
Hum, scary: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2012-02/msg00486.html

ferrieux added on 2012-07-20 05:03:23:
Even closer to our case: http://old.nabble.com/Possible-pthread_cond_timedwait()-regression-on-kfreebsd-i386-td33950701.html

dkf added on 2012-07-20 18:00:05:
It's when you see suggested patches like http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2012-02/msg00538.html that you start to wonder...

ferrieux added on 2012-07-20 18:54:23:
I'm scared.
Stu, what about switching over to the bright side now ?

stwo added on 2012-07-20 20:35:44:
OpenBSD != FreeBSD != NetBSD.
On OpenBSD, pthread_cond_timedwait() ends up at sys__thrsleep which is in kern/kern_synch.c.
This little program appears to demonstrate that pthread_cond_timedwait() is ok.
#define NS to use nanosleep instead of pthread_cond_timedwait().

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#include <errno.h>
#include <time.h>

extern int errno;

pthread_mutex_t m;
pthread_cond_t c;

/*#define NS 1*/
void go (void) {
struct timespec ts1, ts2, tts;
int r;
pthread_mutex_init(&m, NULL);
pthread_cond_init(&c, NULL);
pthread_mutex_lock(&m);
clock_gettime(CLOCK_REALTIME, &ts1);
#ifdef NS
tts.tv_sec = 2;
tts.tv_nsec = 200000000;
r = nanosleep(&tts,NULL);
#else
tts.tv_sec  = ts1.tv_sec  + 2;
tts.tv_nsec = ts1.tv_nsec + 200000000;
r = pthread_cond_timedwait(&c, &m, &tts);
#endif
clock_gettime(CLOCK_REALTIME, &ts2);
pthread_mutex_unlock(&m);
#ifdef NS
if (r != 0) { perror(NULL); return; }
#else
if (r != 0 && r != ETIMEDOUT) { errno = r; perror(NULL); return; }
#endif
printf("%d %ld\n", ts1.tv_sec, ts1.tv_nsec);
printf("%d %ld\n", ts2.tv_sec, ts2.tv_nsec);
}

int main (int argc, char *argv[]) {
go();
return 0;
}

ferrieux added on 2012-07-21 18:04:50:
Uh, sorry, but it's not testing exactly the same thing ;)
In the ktrace you sent me, we see that Tcl is basically doing:

   gettimeofday(T1)
   clock_gettime(REALTIME,NOW)
   pthread_cond_timedwait(NOW+100ms)
   gettimeofday(T2)

   /* Oh ! T2-T1 is much bigger than 100ms !!! */

While it seems that your program uses clock_gettime(REALTIME) instead of gettimeofday (=> own dogfood)

And with NS, it is testing yet another thing (discrepancies between the realtime clock and relative timers). If the bug is in the time comparison inside pthread_cond_timedwait, I don't expect such a discrepancy to exist; additionally, such an obvious shift between the two clocks would have been noticed elsewhere, not confined to pthread as the bug seems to be.

Can you please retry a minimal test along the above lines ? Also, using the same order of magnitude (100ms) and many runs in a row could help reproduce the issue.

ferrieux added on 2012-07-27 05:44:54:
Update: updated snippet and new ktrace by Stu; still no repro, but show something interesting by contrast: 
In fact, the Tcl sequence of syscalls does *not* include clock_gettime ! It is only based on gettimeofday.

   1342727996.861547 CALL  gettimeofday(0xcfbe1788,0)
   1342727996.861558 STRU  struct timeval { "Jul 19 15:59:56 2012".861555 }
   1342727996.861565 RET   gettimeofday 0
   1342727996.861574 CALL  __thrsleep(0x2e62abe0,CLOCK_REALTIME,0xcfbe17e4,0x83596ea0,0x2e62ac78)
   1342727996.861584 STRU  struct timespec { "Jul 19 15:59:56 2012".951731000 }
*** 500ms delay, 400ms late !!! ***
   1342727997.361283 RET   __thrsleep 35/0x23
   1342727997.361400 CALL  gettimeofday(0xcfbe17d8,0)
   1342727997.361430 STRU  struct timeval { "Jul 19 15:59:57 2012".361426 }
   1342727997.361441 RET   gettimeofday 0

So for some reason Tcl trusts gettimeofday and pthread_cond_timedwait' s absolute target to be congruent. Maybe that plays a role in the bug. The log above shows another strange phenomenon (as if Time itself had slipped), but the analysis is muddied by the fact that we don't know which of the two clocks feeds the ktrace timestamps...

ferrieux added on 2012-07-27 06:08:10:
Okay, after a bit of diving, it seems that clock_gettime(CLOCK_REALTIME) is the same as gettimeofday(), so maybe this difference is a red herring after all. Suggestions:

(1) rewrite the snippet to use gettimeofday exclusively, incl. to compute the target of the pcond. Ktrace it to verify we get the same sequence as below

(2) if the gaps are still not reproduced, it's tough since the original trace shows absolutely zero activity on other threads (of the tcltest process tree) during the gap... maybe other processes ?

An interesting experiment would be to sync the system's clock (to the precision of an eye-and-finger, should be enough) to an outside wall clock; then wait for Tcl to show the gap ; then check the clocks. If the system clock has drifted by a visible amount (400ms in the previous tests), we have our analysis !

ferrieux added on 2012-08-02 01:14:41:
Stu has just made the wall-clock experiment. Executive summary: no system clock drift.

Details:
 (1) Get timestamp on a wall-clock minute boundary
 (2) make several failing tests, under ktrace, verifying the above gap in pthread_condwait's actual delay
 (3) Get 2nd minute-aligned timestamp

Result: delta is 119.88, which to the precision of eye and finger is 2mn. (and any drift would have been in the other direction).

Conclusion: pthread_condwait alone (not the system clock) has a problem.

I suggest to try the patch mentioned earlier (the one Donal found, about second/nanosecond arithmetic)

jan.nijtmans added on 2015-10-07 14:07:37:
Status? I think this one is fixed now (as "Resolution" suggests), but why is it still Open?

stwo added on 2015-10-07 16:30:23:
The async behaviour in a couple of the Tcl tests in a threaded tclsh is not 100% predictable.
So there isn't really a bug in Tcl but since the symptoms still persist I left the ticket open as a reminder.

serhiy.storchaka added on 2026-10-17 06:24:49:

async-4.3 has not been run outside MSVC builds since [c0c61fa7cf] (2021): the knownMsvcBug constraint became true on MSVC instead of false. It passes on OpenBSD 7.9 (main, 10 of 10 runs) and on Linux (20 of 20 runs) with the constraint fixed in branch async-test-msvc-constraint.