Tcl Source Code

I suspect this is unavoidable behavior (on Windows where stdin is the console). The issue, as I remember it, is that the Windows console API does not support non-blocking or async i/o so it is simulated with blocking i/o on another thread which in turn requires a read of at least one byte. Further detail would require me to source dive to recall all the issues.

In the general case, I'm not sure what the "official" stance, if any, is on mixing i/o between Tcl channels and C RTL or system calls. Even aside from Windows consoles, I think the channel subsystem will also do read ahead in certain cases like looking for the CRLF->LF translation or completion of a UTF-8 multibyte sequence. That is conjecture on my part and someone who fully understands the channel subsystem in Tcl (probably only one or two such in the universe :-)) can answer definitively.

Use of *WaitFor* functions is something of a red herring. It is not a question of detecting availability of data, but rather not blocking on the ReadConsole either because the wait fired because of a non-character event (e.g. mouse) in which case ReadConsole would still block, or because some thread read the data in the meanwhile.

Your comment implies you are using 8.6. I'm afraid any changes to the console code there are very unlikely to happen for a number of reasons.

Are you willing to move to 9.0 (which has a complete console rewrite albeit with the same read ahead)? If so, I can attempt a fix though I am still skeptical of intermixing Tcl channels with raw read/fread calls.

Also, thanks for your sample program. Can you also link me to the original code from which the sample is derived (assuming it is open source)?

The branch [apn-winconsole-noreadahead] contains a straw implementation that works with the sample code you posted. Note it is based off the 9.1 code base (though it should also be portable to 9.0). There is no chance of an 8.6 version (from me, anyways).

I'll wait to hear back whether it is useful for your purposes before proceeding further.

Thank you for your patch.

First I tested if the program reads character-by-character input correctly by adding

HANDLE hStdin = GetStdHandle(STD_INPUT_HANDLE);
DWORD mode;
GetConsoleMode(hStdin, &mode);
SetConsoleMode(hStdin, mode & ~ENABLE_LINE_INPUT);

to the test program. It works correctly, generating one callback for every character separately. However, pressing e.g. the ALT key also generates a callback, and then the read function will block.

If I don't change the mode and keep ENABLE_LINE_INPUT to read line-by-line, then the callback function is already called after typing the first character, and the read function will block.

For the purpose of tkinter, it is actually sufficient if we can read character-by-character correctly. It would be great if reading character-by-character can be fixed in Tcl 9. Python currently still uses Tcl 8.6, but the Python developers also seem eager to move to Tcl 9, so I assume we don't need to worry about Tcl 8.6.

Yes, tkinter has been around for a long time, but the event loop interface between Python and Tcl/Tk was always clunky.

In older versions of Python, tkinter would use Tcl_CreateFileHandler on Unix/Mac to monitor stdin while calling Tcl_DoOneEvent(0).

On Windows, Tcl_CreateFileHandler would not work for stdin, so instead tkinter uses a busy-wait loop. Recent versions of Python use a busy-wait loop on all platforms, which is inefficient and wasteful.

If we could use Tcl_CreateChannelHandler (which is available on Windows), then we could call Tcl_DoOneEvent(0) on all platforms, and get rid of the busy-wait loop.

The problem is though that after exiting the Tcl_DoOneEvent loop, Python tries to read the next line from stdin, which doesn't work with Tcl_CreateChannelHandler on Windows if the channel reads data from stdin in advance.

It may be that this problem is just not solvable in this way because of the inherent limitations on Windows. Perhaps a better solution is for Python to implement its own event loop, and use it as an external event loop in tkinter. Then Python retains control over the event loop and monitoring stdin.

I have fixed the apn-winconsole-noreadahead branch to filter out the Alt keypresses. That was just a plain oversight.

It also turns out that to enable per-character read() calls, in addition to SetConsoleMode you also need to call _setmode() on stdin with value _O_BINARY. Otherwise, carraige-return is not handled properly. This is independent of Tcl channels.

The program below demonstrates the working (pass line or char to enable the two modes). There are still two issues:

• While other control characters are handled correctly, for some reason Ctrl-J and Ctrl-Z are not, possibly special handling in the CRT? The callback is still invoked but the read() fails.

• Unicode characters are not handled by the read() call. Non iso8859-1 (on my system) read ? as they are not encodable.

Both above may possibly be fixed by using ReadFileW or ReadConsoleW, not sure, but don't think it has anything to do with Tcl.

Confusing things will probably happen if any Tcl code reads from stdin in addition to tkinter. I doubt that can be avoided.

I suspect you are right that it might be better for Python to implement its own event loop but I also suspect it would be considerably more complex.

As a completely unrelated aside, it would be better if any code that works with Tcl 9 calls TclZipfs_Apphook as the very first call to initialize Tcl.

Anyways, below is the modified version of your sample code.

// cl -Zi -I../../generic -I. ../x.c tcl91.lib
// gcc -o x -I/usr/include/tcl8.6 x.c -ltcl8.6
#include <tcl.h>
#include <stdio.h>
#ifdef _WIN32
#include <fcntl.h>
#include <windows.h>
#else
#include <unistd.h>
#endif

void printHex(const char *label, const char *data, int length) {
    for (int i = 0; i < length; i++) {
	printf("%s%02x", label, (unsigned char)data[i]);
	label = ",";
    }
    printf("\n");
    fflush(stdout);
}

void StdinCallback(ClientData clientData, int mask) {
    char buffer[16];

    // Read a line from stdin
    int n;
    n = read(0, buffer, sizeof(buffer) - 1);
    if (n > 0) {
        buffer[n] = '\0';
	if (buffer[0] == 'q')
	    Tcl_Exit(0);
	printHex("Got input: ", buffer, n);
    }
}

int main(int argc, char *argv[]) {
    int charMode = 0; // default to line mode
    if (argc > 1) {
	if (!strcmp(argv[1], "char")) {
	    charMode = _O_BINARY;
	} else if (strcmp(argv[1], "line")) {
	    fprintf(stderr, "Usage: %s [char|line]\n", argv[0]);
	    exit(1);
	}
    }
    Tcl_Interp *interp = Tcl_CreateInterp();

    if (Tcl_Init(interp) != TCL_OK) {
        fprintf(stderr, "Tcl_Init failed: %s\n", Tcl_GetStringResult(interp));
        return 1;
    }

    Tcl_Channel in = Tcl_GetStdChannel(TCL_STDIN);
    if (!in) {
        fprintf(stderr, "No stdin channel available\n");
        return 1;
    }

    if (charMode) {
	HANDLE hStdin = GetStdHandle(STD_INPUT_HANDLE);
	DWORD mode;
	GetConsoleMode(hStdin, &mode);
	SetConsoleMode(hStdin,
		       mode & ~(ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT));
	_setmode(_fileno(stdin), _O_BINARY);
    }

    Tcl_CreateChannelHandler(in, TCL_READABLE, StdinCallback, NULL);

    printf("Type something and press Enter:\n");

    // Run the event loop
    while (1) {
        Tcl_DoOneEvent(0);
    }

    return 0;
}

is the proposal by Ashok an improvement and should this be integrated, even if it is not perfect?

Outside of a small change to filter out Alt keys etc. to prevent unnecessary wakeups, I would say no unless there is a tangible use case where it is a complete solution (it does not seem to be so for this ticket). The Windows console reader code is quite "delicate" and the changes made for this ticket are more experimental than production ready until a thorough review.

There should also be some kind of "official" stance on whether Tcl supports intermixing of Tcl channel calls with direct C I/O API's through OS handles/fds and under what conditions.

In line mode, I noticed that the StdinCallback callback gets called immediately when I press the first key. Then the program sits in read() until enter is pressed.

This indeed would be an issue that I'm not sure can be fixed. At the channel driver level, there is no knowledge of characters vs lines, encoding etc. or whether the application is doing character-by-character reads or gets. That's all within the core channel infrastructure so the driver generates notifications whenever there is data available, irrespective of whether it is a complete line or not.

I guess the way forward then is to close this ticket, and for Python to rethink its API for interfacing with Tcl.

Probably. If that does not work out, we can explore further on this path.