|
2026-10-21
| ||
| 10:03 | • Closed ticket [550646ffff]: exec (fork) from huge process fails plus 4 other... artifact: 5ba9364c75 user: serhiy.storchaka | |
|
2024-08-18
| ||
| 01:26 | Merge [9643b6d086fbfd19], " Fix [c54e4a1aeb] (again): High Tcl latencies with fork() in larger syste... check-in: 2f276901cb user: pooryorick tags: unchained, INCOMPATIBLE_LICENSE | |
| 01:22 | Merge [160f2d03a7b57141], " Fix [c54e4a1aeb]: High Tcl latencies with fork() in larger systems" and ... check-in: 70d8eda742 user: pooryorick tags: unchained, INCOMPATIBLE_LICENSE | |
|
2023-07-26
| ||
| 07:16 | • Closed ticket [c54e4a1aeb]: High Tcl latencies with fork() in larger systems... artifact: 21c2cf0ba2 user: jan.nijtmans | |
| 07:15 | Fix [c54e4a1aeb] (again): High Tcl latencies with fork() in larger systems. vfork() is now prefered... check-in: 42d3f0cdb9 user: jan.nijtmans tags: core-8-6-branch | |
|
2023-07-24
| ||
| 15:58 | • Open ticket [c54e4a1aeb]: High Tcl latencies with fork() in larger systems p... artifact: 5030582bba user: jan.nijtmans | |
| 10:37 | • Pending ticket [c54e4a1aeb]. artifact: cab9421825 user: jan.nijtmans | |
| 10:34 | Keeping open [c54e4a1aeb]: Can the exec-17.1 test-failure on MaxOS be fixed? check-in: 98f43375cf user: jan.nijtmans tags: bug-c54e4a1aeb | |
| 10:19 | Fix [c54e4a1aeb]: High Tcl latencies with fork() in larger systems check-in: b5b3c28401 user: jan.nijtmans tags: core-8-6-branch | |
|
2023-07-07
| ||
| 14:34 | • Ticket [c54e4a1aeb] High Tcl latencies with fork() in larger systems status ... artifact: 77e3100753 user: jan.nijtmans | |
| 13:52 | • Ticket [c54e4a1aeb]: 3 changes artifact: 7c6693b077 user: gustafn3 | |
|
2023-07-04
| ||
| 12:12 | • Ticket [c54e4a1aeb]: 3 changes artifact: de7610a5e1 user: gustafn3 | |
|
2023-07-03
| ||
| 20:11 | • Ticket [c54e4a1aeb]: 3 changes artifact: 17dc41212b user: jan.nijtmans | |
| 15:09 | • Ticket [c54e4a1aeb]: 4 changes artifact: 36a6bfe065 user: jan.nijtmans | |
| 12:21 | • Ticket [c54e4a1aeb]: 3 changes artifact: 9635cc6416 user: dkf | |
|
2023-06-30
| ||
| 10:27 | • Ticket [c54e4a1aeb]: 4 changes artifact: 85b6f3d65c user: oehhar | |
| 10:03 | • Ticket [c54e4a1aeb]: 3 changes artifact: f0633caf65 user: gustafn3 | |
|
2023-06-23
| ||
| 19:19 | • Ticket [c54e4a1aeb]: 3 changes artifact: e2ca3da696 user: gustafn3 | |
|
2023-06-19
| ||
| 20:42 | • Ticket [c54e4a1aeb]: 3 changes artifact: 48229dcbc3 user: pointsman | |
|
2023-06-15
| ||
| 17:10 | • Ticket [c54e4a1aeb]: 3 changes artifact: c28631fdd1 user: gustafn3 | |
|
2023-06-13
| ||
| 00:41 | • Ticket [c54e4a1aeb]: 3 changes artifact: 6fd50a7aa9 user: pointsman | |
|
2023-06-08
| ||
| 12:04 | • Ticket [c54e4a1aeb]: 4 changes artifact: f0ce11e111 user: oehhar | |
| 12:01 | ticket [c54e4a1aeb] patch: Use vfork instead fork. check-in: 98e93f779e user: oehhar tags: ticket-c54e4a1aeb-use_vfork | |
|
2023-06-06
| ||
| 13:09 | • New ticket [c54e4a1aeb] High Tcl latencies with fork() in larger systems. artifact: 82d6b8be2f user: anonymous | |
| Ticket UUID: | c54e4a1aeb999fe8d878eae790243739ef05517a | ||
| Title: | High Tcl latencies with fork() in larger systems | ||
| Type: | Bug | Created on: | 2023-06-06 13:09:42 |
| Submitter: | anonymous | Assigned to: | jan.nijtmans |
| Subsystem: | 49. Threading | Severity: | Important |
| Priority: | 5 Medium | Last modified: | 2023-07-26 07:16:52 |
| Status: | Closed | Closed by: | jan.nijtmans |
| Resolution: | Fixed | Closed on: | 2023-07-26 07:16:52 |
| Version: | 8.6 | ||
| Description: | ||||
Management summary: The usage of fork() within Tcl becomes a problem,
when the memory footprint of the process becomes large and virtual machines
are used. The usage of more modern system calls solves this (patch and
empirical data included)
When Tcl forks() on a Unix system, it has the consequence that the full
process is stopped for the duration of the fork() system call. Under
normal conditions, this is not a big issue, but it is very bad in
multithreaded applications. The duration of the fork() system call
depends on the memory footprint of the process. Even worse, when running
in a virtual machine, the duration of the fork() call grows
significantly compared to a bare metal. It was observed that a fork()
takes up to e.g. 6 seconds [1]. This is especially bad when running a
multithreaded web server (with, e.g., 60 connection threads), since all
threads are stopped during fork(). When the server handles e.g. 1000
pages/second, a substantial backlog will be the consequence,
video-streams will hang shortly, some mutex locks might take very long
(without visible evidence when looking at the source code), etc. The
service quality degrades.
So, the moral of [1] is to try to avoid such cases. In general, fork()
is an issue for "exec" and pipe-open, for NaviServer/OpenACS, this is
just an issue for starting a process in a pipe, since it has its own
exec implementation (working with a pool of separate worker processes
connected via a queue). However, when e.g. tcllib is used, calls of
pipe-open can't be avoided.
Here are some figures (script is below) from Linux-based servers
(server1, server2, test-server) running in a virtual machine, using
pipe-open via fork(). The specified sizes are in GB. The physical memory
sizes of all servers is >=128 GB.
server1:
pipe open took 302ms (rss 20.50 vsize 22.95)
server2:
pipe open took 819ms (rss 17.10 vsize 19.62)
pipe open took 1176ms (rss 21.07 vsize 21.81)
On a test machine, one can see nicely the degradation of the
performance, when the memory footprint grows:
test-server:
pipe open took 174ms (rss 3.48 vsize 4.19)
pipe open took 215ms (rss 4.41 vsize 5.12)
pipe open took 254ms (rss 5.34 vsize 6.06)
pipe open took 309ms (rss 6.27 vsize 6.99)
pipe open took 331ms (rss 7.20 vsize 7.92)
pipe open took 401ms (rss 8.13 vsize 8.85)
pipe open took 421ms (rss 9.05 vsize 9.78)
pipe open took 472ms (rss 9.98 vsize 10.72)
pipe open took 546ms (rss 10.91 vsize 11.65)
pipe open took 639ms (rss 11.84 vsize 12.58)
pipe open took 607ms (rss 12.77 vsize 13.51)
pipe open took 721ms (rss 13.69 vsize 14.45)
pipe open took 733ms (rss 14.61 vsize 15.38)
pipe open took 743ms (rss 15.54 vsize 16.31)
pipe open took 781ms (rss 16.47 vsize 17.24)
pipe open took 809ms (rss 17.39 vsize 18.17)
pipe open took 848ms (rss 18.32 vsize 19.10)
pipe open took 878ms (rss 19.25 vsize 20.03)
pipe open took 957ms (rss 20.17 vsize 20.97)
When replacing the fork() system call with posix_spawn(), the
performance can be substantially improved in every case. Furthermore,
with posix_spawn(), the duration is independent of the memory footprint
of the process.
test-server:
pipe open took 5ms (rss 2.87 vsize 3.57)
pipe open took 6ms (rss 3.68 vsize 4.50)
pipe open took 5ms (rss 4.61 vsize 5.43)
pipe open took 5ms (rss 5.54 vsize 6.36)
pipe open took 4ms (rss 6.47 vsize 7.30)
pipe open took 5ms (rss 7.40 vsize 8.23)
pipe open took 5ms (rss 8.33 vsize 9.16)
pipe open took 5ms (rss 9.26 vsize 10.09)
pipe open took 5ms (rss 10.20 vsize 11.02)
pipe open took 4ms (rss 11.13 vsize 11.95)
pipe open took 6ms (rss 12.06 vsize 12.88)
pipe open took 5ms (rss 12.99 vsize 13.82)
pipe open took 10ms (rss 13.92 vsize 14.75)
pipe open took 6ms (rss 14.85 vsize 15.69)
pipe open took 5ms (rss 15.79 vsize 16.62)
pipe open took 5ms (rss 16.72 vsize 17.55)
pipe open took 5ms (rss 17.65 vsize 18.48)
pipe open took 5ms (rss 18.58 vsize 19.41)
pipe open took 4ms (rss 19.51 vsize 20.34)
One can see, that for e.g. a footprint of 20 GB, the new version is more
than 100 times faster... but more important, the long lock latencies of
about 1 second are gone.
The patch below is against tcl8.6.13. It is intended to be least
invasive and has been running for a while on e.g. the server
openacs.org. I have tested it with Ubuntu 20.4, FC 37, CentOS 7 and
macOS 13.4 (Ventura).
Actually, a similar benefit could be achieved via vfork(), which is
intended to be used under macOS (check: "Darwin" configure.in sets
USE_VFORK, but magically this is unset by the include files included
from tclInt.h in tclUnixPipe.c). So setting USE_VFORK for recent macOS in
configure.in is useless. When compiling tclUnixPipe.c with vfork(),
recent versions of macOS (starting with macOS Monterey) complain that
vfork() is deprecated, ... and the compiler suggests using either
posix_spawn() or fork().
So, posix_spawn() is the better choice, since it also works for other
Unixes. When this is to be used, one should check via configure.in
for posix_spawn() in libc and clean up tclUnixPipe.c a little more.
All the best
-g
[1] https://openacs.org/forums/message-view?message_id=5539060
============================================================================test-pipe-open.tcl
proc memsizes {pid} {
#
# Return a dict of memory sizes of pid in GB
#
lassign {0 0} rss vsize
if {[file readable /proc/$pid/statm]} {
#
# result in pages, typically 4K
#
set F [open /proc/$pid/statm]; set c [read $F]; close $F
lassign $c vsize rss shared
set rss [format %.2f [expr {$rss * 4.0/(1024*1024)}]]
set vsize [format %.2f [expr {$vsize * 4.0/(1024*1024)}]]
}
return [list rss $rss vsize $vsize]
}
for {set i 0} {$i < 19} {incr i} {
set t [clock milliseconds]
set f [open "|cat" w]; puts $f "hi"; close $f
puts stderr "pipe open took [expr {[clock milliseconds] - $t}]ms ([memsizes [pid]])"
append _$i [string repeat 01234567890123456789012345678901234567890123456789 20000000] ;# 1GB
}
============================================================================
============================================================================tclUnixPipe.patch
--- tclUnixPipe.c-orig 2023-06-05 18:00:14
+++ tclUnixPipe.c 2023-06-06 12:04:25
@@ -13,6 +13,12 @@
#include "tclInt.h"
+#define USE_POSIX_SPAWN 1
+
+#ifdef USE_POSIX_SPAWN
+# include <spawn.h>
+#endif
+
#ifdef USE_VFORK
#define fork vfork
#endif
@@ -439,7 +445,7 @@
newArgv[i] = Tcl_UtfToExternalDString(NULL, argv[i], -1, &dsArray[i]);
}
-#ifdef USE_VFORK
+#if defined(USE_VFORK) || defined(USE_POSIX_SPAWN)
/*
* After vfork(), do not call code in the child that changes global state,
* because it is using the parent's memory space at that point and writes
@@ -458,8 +464,45 @@
Tcl_GetStdChannel(TCL_STDERR);
}
#endif
+#ifdef USE_POSIX_SPAWN
+ {
+ posix_spawn_file_actions_t actions;
+ posix_spawnattr_t attr;
+ posix_spawn_file_actions_init(&actions);
+ posix_spawnattr_init(&attr);
+
+ posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF|
+# ifdef POSIX_SPAWN_USEVFORK
+ POSIX_SPAWN_USEVFORK
+# else
+ 0
+# endif
+ );
+ posix_spawn_file_actions_adddup2(&actions, GetFd(inputFile), 0);
+ posix_spawn_file_actions_adddup2(&actions, GetFd(outputFile), 1);
+ posix_spawn_file_actions_adddup2(&actions, GetFd(errorFile), 2);
+
+ status = posix_spawnp(&pid, newArgv[0], &actions, &attr, newArgv, environ);
+
+ posix_spawn_file_actions_destroy(&actions);
+ posix_spawnattr_destroy(&attr);
+
+ /*
+ * Fork semantics:
+ * - pid == 0: child process
+ * - pid == -1: error
+ * - pid > 0: parent process
+ *
+ * Mimic fork semantics to minimize changes below
+ */
+ if (status != 0) {
+ pid = -1;
+ }
+ }
+#else
pid = fork();
+#endif
if (pid == 0) {
size_t len;
int joinThisError = errorFile && (errorFile == outputFile);
============================================================================
| ||||
| User Comments: | ||||
oehhar added on 2023-06-08 12:04:18:
Thank you for the patch. Patch entered into branch [ticket-c54e4a1aeb-use_vfork] starting with commit [98e93f779e]. I am not on Linux and the Linux guys are quiet. I suppose, the line #define USE_POSIX_SPAWN 1 should be set by some external mechanism like a configuration option. Take care, Harald pointsman added on 2023-06-13 00:41:44:
The example script provided by Gustaf demonstrates for me even on bare metal (not a vm) the described behaviour. On a laptop with 12 GByte physical RAM (I'm on hollidays), debian 11.7 and the loop reduced to 9 I get which core-8-6-branch: pipe open took 2ms (rss 0.00 vsize 0.01) pipe open took 3ms (rss 0.94 vsize 0.94) pipe open took 4ms (rss 1.87 vsize 1.87) pipe open took 4ms (rss 2.80 vsize 2.80) pipe open took 5ms (rss 3.73 vsize 3.73) pipe open took 46ms (rss 4.66 vsize 4.67) pipe open took 80ms (rss 5.59 vsize 5.60) pipe open took 114ms (rss 6.52 vsize 6.53) pipe open took 142ms (rss 7.46 vsize 7.46) And with ticket-c54e4a1aeb-use_vfork branch: pipe open took 2ms (rss 0.00 vsize 0.01) pipe open took 2ms (rss 0.94 vsize 0.94) pipe open took 2ms (rss 1.87 vsize 1.87) pipe open took 1ms (rss 2.80 vsize 2.80) pipe open took 2ms (rss 3.73 vsize 3.73) pipe open took 1ms (rss 4.66 vsize 4.67) pipe open took 2ms (rss 5.59 vsize 5.60) pipe open took 1ms (rss 6.52 vsize 6.53) pipe open took 2ms (rss 7.46 vsize 7.46) The test suite with branch ticket-c54e4a1aeb-use_vfork runs with 6 bogus failures - the expected result was "couldn't fork child process: no such file or directory" and the result was "couldn't execute "<somefoo123>": no such file or directory", perhaps because the error message of the underlying library call is returned and that library call changed. From what is presented this is an improvement on linux. I'm not able to say, what drawbacks that may include. If I take right what Gustaf writes there is a similiar problem with exec and NaviServer goes to some length to avoid the impact. So from NaviServer viewpoint this is the least invasive because they already have their nsproxy module to manage the exec case. For the core the patch should perhaps enhanced to exec? gustafn3 added on 2023-06-15 17:10:40:
I have modified the branch "ticket-c54e4a1aeb-use_vfork" to pass the regression test under Linux. @Rolf: the same code is used for "exec" and "pipe open", so one gets the same speedup for "exec" also. I have committed the change to the branch. Please double-check, if the commit went well. If everything looks good, I will address the "configure" magic to get rid of the hard-coded USE_POSIX_SPAWN. pointsman added on 2023-06-19 20:42:33:
The current state of the branch ([52e6e2e9117d83fe]) builds without error for me on 64-bit debian 11.7 and runs the test suite now without error. gustafn3 added on 2023-06-23 19:19:32:
Here are some more performance figures using pipe-open and exec on a Rocky 9.2 installation running in a virtual machine. The virtual machine has 29 GB of memory. The test runs 30 iterations (29 GB are added) and compares Tcl 8.6.13 with the version with posix-spawn. The test show exactly the same benefit on exec and the pipe-open. The times using posix_spawn are essentially constant on the memory footprint, whereas with Tcl 8.6.13, the performance degrades when more memory is used. Actually, the branch name "ticket-c54e4a1aeb-use_vfork" is misleading, since the code uses posix_spawn() and not vfork(2). In modern libc versions (since 2.24), posix-spawn is calling clone(2). The used flag "POSIX_SPAWN_USEVFORK" has in newer versions no effect, but might help for older Versions of the OS. There seems no easy way to rename branches - so I left the branch name untouched. One can see from the data, that on this machine, the performance degradation is not so bad as on other OS/machine combinations. Still, posix_spanwn is much better. On the next iteration, I'll update configure. Tcl 8.6.13 exec took 2 ms (rss 0.00 vsize 0.21) exec took 3 ms (rss 0.94 vsize 1.15) exec took 4 ms (rss 1.87 vsize 2.08) exec took 5 ms (rss 2.80 vsize 3.01) exec took 6 ms (rss 3.73 vsize 3.94) exec took 9 ms (rss 4.66 vsize 4.87) exec took 6 ms (rss 5.59 vsize 5.80) exec took 7 ms (rss 6.52 vsize 6.73) exec took 8 ms (rss 7.46 vsize 7.66) exec took 10 ms (rss 8.39 vsize 8.60) exec took 10 ms (rss 9.32 vsize 9.53) exec took 11 ms (rss 10.25 vsize 10.46) exec took 11 ms (rss 11.18 vsize 11.39) exec took 12 ms (rss 12.11 vsize 12.32) exec took 13 ms (rss 13.04 vsize 13.25) exec took 15 ms (rss 13.97 vsize 14.18) exec took 15 ms (rss 14.91 vsize 15.12) exec took 16 ms (rss 15.84 vsize 16.05) exec took 16 ms (rss 16.77 vsize 16.98) exec took 16 ms (rss 17.70 vsize 17.91) exec took 16 ms (rss 18.63 vsize 18.84) exec took 17 ms (rss 19.56 vsize 19.77) exec took 18 ms (rss 20.49 vsize 20.70) exec took 18 ms (rss 21.43 vsize 21.63) exec took 19 ms (rss 22.36 vsize 22.57) exec took 19 ms (rss 23.29 vsize 23.50) exec took 22 ms (rss 24.22 vsize 24.43) exec took 22 ms (rss 25.15 vsize 25.36) exec took 26 ms (rss 26.08 vsize 26.29) exec took 27 ms (rss 27.01 vsize 27.22) pipe-open took 1 ms (rss 0.00 vsize 0.21) pipe-open took 3 ms (rss 0.94 vsize 1.15) pipe-open took 4 ms (rss 1.87 vsize 2.08) pipe-open took 4 ms (rss 2.80 vsize 3.01) pipe-open took 4 ms (rss 3.73 vsize 3.94) pipe-open took 5 ms (rss 4.66 vsize 4.87) pipe-open took 6 ms (rss 5.59 vsize 5.80) pipe-open took 7 ms (rss 6.52 vsize 6.73) pipe-open took 8 ms (rss 7.46 vsize 7.66) pipe-open took 9 ms (rss 8.39 vsize 8.60) pipe-open took 13 ms (rss 9.32 vsize 9.53) pipe-open took 10 ms (rss 10.25 vsize 10.46) pipe-open took 11 ms (rss 11.18 vsize 11.39) pipe-open took 11 ms (rss 12.11 vsize 12.32) pipe-open took 12 ms (rss 13.04 vsize 13.25) pipe-open took 13 ms (rss 13.97 vsize 14.18) pipe-open took 14 ms (rss 14.91 vsize 15.12) pipe-open took 14 ms (rss 15.84 vsize 16.05) pipe-open took 15 ms (rss 16.77 vsize 16.98) pipe-open took 15 ms (rss 17.70 vsize 17.91) pipe-open took 16 ms (rss 18.63 vsize 18.84) pipe-open took 17 ms (rss 19.56 vsize 19.77) pipe-open took 18 ms (rss 20.49 vsize 20.70) pipe-open took 19 ms (rss 21.43 vsize 21.63) pipe-open took 19 ms (rss 22.36 vsize 22.57) pipe-open took 20 ms (rss 23.29 vsize 23.50) pipe-open took 26 ms (rss 24.22 vsize 24.43) pipe-open took 22 ms (rss 25.15 vsize 25.36) pipe-open took 23 ms (rss 26.08 vsize 26.29) pipe-open took 24 ms (rss 27.01 vsize 27.22) posix-spawn (branch ticket-c54e4a1aeb-use_vfork) exec took 1 ms (rss 0.00 vsize 0.21) exec took 1 ms (rss 0.94 vsize 1.15) exec took 1 ms (rss 1.87 vsize 2.08) exec took 1 ms (rss 2.80 vsize 3.01) exec took 1 ms (rss 3.73 vsize 3.94) exec took 1 ms (rss 4.66 vsize 4.87) exec took 1 ms (rss 5.59 vsize 5.80) exec took 1 ms (rss 6.52 vsize 6.73) exec took 1 ms (rss 7.46 vsize 7.66) exec took 2 ms (rss 8.39 vsize 8.60) exec took 1 ms (rss 9.32 vsize 9.53) exec took 1 ms (rss 10.25 vsize 10.46) exec took 1 ms (rss 11.18 vsize 11.39) exec took 1 ms (rss 12.11 vsize 12.32) exec took 1 ms (rss 13.04 vsize 13.25) exec took 2 ms (rss 13.97 vsize 14.18) exec took 1 ms (rss 14.91 vsize 15.12) exec took 1 ms (rss 15.84 vsize 16.05) exec took 2 ms (rss 16.77 vsize 16.98) exec took 1 ms (rss 17.70 vsize 17.91) exec took 1 ms (rss 18.63 vsize 18.84) exec took 1 ms (rss 19.56 vsize 19.77) exec took 1 ms (rss 20.49 vsize 20.70) exec took 1 ms (rss 21.43 vsize 21.63) exec took 1 ms (rss 22.36 vsize 22.57) exec took 2 ms (rss 23.29 vsize 23.50) exec took 1 ms (rss 24.22 vsize 24.43) exec took 1 ms (rss 25.15 vsize 25.36) exec took 2 ms (rss 26.08 vsize 26.29) exec took 2 ms (rss 27.01 vsize 27.22) pipe-open took 1 ms (rss 0.00 vsize 0.21) pipe-open took 1 ms (rss 0.94 vsize 1.15) pipe-open took 1 ms (rss 1.87 vsize 2.08) pipe-open took 1 ms (rss 2.80 vsize 3.01) pipe-open took 1 ms (rss 3.73 vsize 3.94) pipe-open took 1 ms (rss 4.66 vsize 4.87) pipe-open took 1 ms (rss 5.59 vsize 5.80) pipe-open took 1 ms (rss 6.52 vsize 6.73) pipe-open took 1 ms (rss 7.46 vsize 7.66) pipe-open took 1 ms (rss 8.39 vsize 8.60) pipe-open took 1 ms (rss 9.32 vsize 9.53) pipe-open took 1 ms (rss 10.25 vsize 10.46) pipe-open took 1 ms (rss 11.18 vsize 11.39) pipe-open took 1 ms (rss 12.11 vsize 12.32) pipe-open took 1 ms (rss 13.04 vsize 13.25) pipe-open took 1 ms (rss 13.97 vsize 14.18) pipe-open took 1 ms (rss 14.91 vsize 15.12) pipe-open took 1 ms (rss 15.84 vsize 16.05) pipe-open took 1 ms (rss 16.77 vsize 16.98) pipe-open took 2 ms (rss 17.70 vsize 17.91) pipe-open took 1 ms (rss 18.63 vsize 18.84) pipe-open took 1 ms (rss 19.56 vsize 19.77) pipe-open took 2 ms (rss 20.49 vsize 20.70) pipe-open took 1 ms (rss 21.43 vsize 21.63) pipe-open took 1 ms (rss 22.36 vsize 22.57) pipe-open took 1 ms (rss 23.29 vsize 23.50) pipe-open took 1 ms (rss 24.22 vsize 24.43) pipe-open took 1 ms (rss 25.15 vsize 25.36) pipe-open took 2 ms (rss 26.08 vsize 26.29) pipe-open took 1 ms (rss 27.01 vsize 27.22) gustafn3 added on 2023-06-30 10:03:52:
The updated branch contains now:
a) Added auto-configuration of posix_spawnp()
b) Removed the automatic setting of USE_VFORK for macOS
c) an updated configure script to ease testing of this branch
(separate commit, to further ease cherry-picking, not sure,
what the tcl-core policy for configure scripts is)
Point (b) is probably the most complex one, and not directly related
with the ticket.
The automatic setting of USE_VFORK in unix/configure.in is useless,
confusing, and deprecated.
Background: configure.in had a section, where it sets
automatically USE_VFORK for macOS.
if test "`uname -s`" = "Darwin" ; then
...
AC_DEFINE(USE_VFORK, 1, [Should we use vfork() instead of fork()?])
...
However, Daniel Steffen has added the following section 17 years ago
which effectively deactivated USE_VFORK for the x86_64 platform, when
it is set, due to a bug report in RDAR (Apple's bug tracking
system). 16 years ago, a section was added to deactivate USE_VFORK for
all platforms with llvm-gcc-4.2 or newer.
/*
*---------------------------------------------------------------------------
* At present, using vfork() instead of fork() causes execve() to fail
* intermittently on Darwin x86_64. rdar://4685553
*---------------------------------------------------------------------------
*/
# if defined(__x86_64__) && !defined(FIXED_RDAR_4685553)
# undef USE_VFORK
# endif /* __x86_64__ */
/* Workaround problems with vfork() when building with llvm-gcc-4.2 */
# if defined (__llvm__) && \
(__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > 2 || \
(__GNUC_MINOR__ == 2 && __GNUC_PATCHLEVEL__ > 0))))
# undef USE_VFORK
# endif /* __llvm__ */
#endif /* __APPLE__ */
This means, that the last 17 years, USE_VFORK was not doing anything
on macOS. Furthermore, macOS 12 (Monterey, released 2021) deprecated
vfork() officially.
Therefore, it is safe to remove the setting of USE_VFORK from the
default configuration for macOS.
It is still possible to set USE_VFORK manually for certain platforms if
needed.
oehhar added on 2023-06-30 10:27:21:
Great to read the name of Daniel Steffen. We have enjoied him in person on ETCL conferences some decades ago. I hope, this branch goes to continuous integration. So, we will get test results until tomorrow. I hope, some Linux Wizards will comment. I understood, it is deactivated on Mac-OS due to depreciation. Otherwise, it may be merged next week to the 3 branches. Thanks again, Gustaf, for the great contribution, Harald dkf added on 2023-07-03 12:21:22:
jan.nijtmans added on 2023-07-03 15:09:54:
I'm currently having a look ..... jan.nijtmans added on 2023-07-03 20:11:07:
Test-result on MacOSX: ==== exec-9.1 commands returning errors FAILED ==== Contents of test case:set x [catch {exec gorp456} msg] list $x [string tolower $msg] [string tolower $errorCode] This looks solvable, but it means the branch is not ready to be merged yet gustafn3 added on 2023-07-04 12:12:03:
I have fixed locally the errors due to different error messages under macOS, except exec-17.1, which is different. The problem with exec-9.1 etc. was that the C code relies on the fact that the global variable "errno" is kept until the error message, which is not the case under macOS.
First, I have to understand, what exec-17.1 tests in detail ("inheriting standard I/O") and why this is a problem. The test chains two exec commands
catch {exec $exe $echo foobar &}
exec $exe $sleep 2
and seems to assume that the output from the first "exec" is preserved by the second "exec".
gustafn3 added on 2023-07-07 13:52:41:
Now, the regression test passes at least under macOS and Linux. Are other unix-like platforms included in the automated tests? In other words, do i have to care about tests under OpenBSD/FreeBSD by myself? The only reason, posix_spanwp() is not activated under macOS right now is test exec-17.1, where stdout is closed and replaced by a different stream. jan.nijtmans added on 2023-07-07 14:34:54:
> Are other unix-like platforms included in the automated tests? In other words, do i have to care about tests under OpenBSD/FreeBSD by myself? I would be nice if someone would test this on OpenBSD/FreeBSD. Anyone willing to do that? Stuart Cassoff, maybe? jan.nijtmans added on 2023-07-24 10:37:22:
Committed to 8.6 now (will be forward-merged). Someone still wants to test this on OpenBSD/FreeBSD? Could there be a way to make this work as expected on MacOS? Therefore, I leave this ticket "Pending" for now. jan.nijtmans added on 2023-07-26 07:16:52:
[Fixed|42d3f0cdb9b6bb69] again, now by prefering vfork() if available. | ||||
