| Ticket UUID: | 39fed4dae51e4fd7f1abf6baf3ba094a072b8beb | ||
| Title: | Tcl_PkgRequireEx, Tcl_InitStubs broken in 8.6.9 | ||
| Type: | Bug | Created on: | 2019-02-27 18:19:46 |
| Submitter: | edhume3 | Assigned to: | dgp |
| Subsystem: | 39. Package Manager | Severity: | Important |
| Priority: | 5 Medium | Last modified: | 2019-03-08 14:46:47 |
| Status: | Closed | Closed by: | dgp |
| Resolution: | Fixed | Closed on: | 2019-03-08 14:46:47 |
| Version: | 8.6.9 | ||
| Description: | ||||
Apps that have copied previous versions of tkMain.c have checked for an exact Tcl match with code like this early in the startup:
if (Tcl_InitStubs(interp, TCL_VERSION, 1) == NULL) {
abort();
}
In Tcl 8.6.9 the request for an exact match (3rd argument =1) fails when it should not.
The problem is caused by changes in Tcl 8.6.9 Tcl_PkgRequireEx() in generic/TclPkg.c. The logic is using a volatile result obtained with Tcl_GetStringResult(), so that the return value of Tcl_PkgRequireEx() is not a valid pointer. I verified the issue by adding printf statements in the flawed code and saw that the value returned by Tcl_PkgRequireProc() was overwritten instead of being the return value of Tcl_PkgRequireEx(). The problem can be seen by inspection if the inspector realizes that the return value of Tcl_GetStringResult() is volatile and needs to be copied. A demo fix that is not threadsafe, would be to declare a static buffer and use it to cache the return value like this:
const char *
Tcl_PkgRequireEx(
Tcl_Interp *interp, /* Interpreter in which package is now
* available. */
const char *name, /* Name of desired package. */
const char *version, /* Version string for desired version; NULL
* means use the latest version available. */
int exact, /* Non-zero means that only the particular
* version given is acceptable. Zero means use
* the latest compatible version. */
void *clientDataPtr) /* Used to return the client data for this
* package. If it is NULL then the client data
* is not returned. This is unchanged if this
* call fails for any reason. */
{
Tcl_Obj *ov;
static char result[50]; /* demo fix */
...
/*
* Translate between old and new API, and defer to the new function.
*/
if (version == NULL) {
if (Tcl_PkgRequireProc(interp, name, 0, NULL, clientDataPtr) == TCL_OK) {
#if 0
result = Tcl_GetStringResult(interp);
#else
/* not threadsafe - demo fix not a real fix */
strncpy(result, Tcl_GetStringResult(interp), sizeof(result)-1);
result[sizeof(result)-1] = 0;
#endif
Tcl_ResetResult(interp);
}
} else {
if (exact && TCL_OK
!= CheckVersionAndConvert(interp, version, NULL, NULL)) {
return NULL;
}
ov = Tcl_NewStringObj(version, -1);
if (exact) {
Tcl_AppendStringsToObj(ov, "-", version, NULL);
}
Tcl_IncrRefCount(ov);
if (Tcl_PkgRequireProc(interp, name, 1, &ov, clientDataPtr) == TCL_OK) {
#if 0
result = Tcl_GetStringResult(interp);
#else
/* not threadsafe - demo fix not a real fix */
strncpy(result, Tcl_GetStringResult(interp), sizeof(result)-1);
result[sizeof(result)-1] = 0;
#endif
//printf("result=%s\n", result); "8.6.9"
Tcl_ResetResult(interp);
}
TclDecrRefCount(ov);
}
//printf("result=%s\n", result); ""
return result;
}
The problem likely surfaces for most package require API calls that seek an exact match.
| ||||
| User Comments: | ||||
jan.nijtmans added on 2019-02-27 22:14:42:
So, this commit is making the described change: [5a70d5bef163545a]. Therefore, pouryorick, please have a look! dgp added on 2019-03-02 11:46:13:
In Tcl's "apartment model" of thread usage, a (Tcl_Interp *) is only permitted to be passed to routines of the Tcl library from the same thread where a call to Tcl_CreateInterp() return that (Tcl_Interp *) value in the first place. Honoring that restriction, the thread safety concerns expressed here should not be relevant, unless I'm misunderstanding something. Is that the real root of trouble here? dgp added on 2019-03-02 11:55:22:
Indeed I was missing something. Although the ticket makes the complaint in the language of threading and volatility, the brokenness is simpler than that. As modified, the routine is returning a pointer to a value that's already been invalidated. Just these two lines are wrong-headed. https://core.tcl-lang.org/tcl/artifact/4c83c1db6c1901ec?ln=303-304 My next Q is why the rewrite here thinks a Tcl_ResetResult() is necessary or desirable? dgp added on 2019-03-02 12:18:06:
Tcl provides a number of routines that return to the caller a pointer into internal structs managed by Tcl itself. This is a fragile, but efficient arrangement that can work so long as the caller is aware of the limited period of reliability of the returned pointer, and makes copies of anything needed beyond that timeframe. Before this rewrite, the pointer returned pointed to a pkgPtr->version field of the "Package" struct for the provided package. This pointer could be expected to be reliable until the next [package forget] removes that Package from our database of package information known to the interp. After the rewrite, the pointer points into the interp's string result, which is only reliable until the next Tcl_ResetResult() call on that interp -- a much shorter time frame. That's a disturbance in the public interface we should not have accepted. Even worse the rewrite makes the Tcl_ResetResult() calls itself, invalidating the pointer before it is even returned. Removing the Tcl_ResetResult() calls is not an effective solution. Test load-10.1 among others fails. sebres added on 2019-03-06 11:25:51:
Nice solution in bug-39fed4dae5, Don... Apropos test-cases, I'd like to have proper coverage ("dltest") for non-posix platforms too, the question is - could I move folder "unix/dltest" and if yes, then to "generic"? dgp added on 2019-03-06 15:29:31:
That sounds like a very good idea. pooryorick added on 2019-03-06 15:49:41:
My submission for a fix is [87689a5b66bc2669]. It uses a aku added on 2019-03-06 16:40:28:
Of the two proposed solutions, so far, I like Don's more. While Nathan's solution is smaller in terms of actually changed code it also feels much more hackish to me, twiddling the internals of a string-type Tcl_Obj like it does. Don's change looks to be more principled. jan.nijtmans added on 2019-03-07 09:49:19:
Well, Don's solution has the problem that the ->version stored in the package list might be shared in multiple threads: If multiple threads muggle with the internal rep of a Tcl_Obj, that could result in strange results. That's why package versions were stored as a string, not as Tcl_Obj. Nathan's solution was - indeed - hacky. My proposal is: Follow Nathan's solution, but add proper protections and a proper internal Tcl_Obj type. Additional advantage: The string representation of the Tcl_Obj only has to be allocated when needed. Committed now for evaluation. sebres added on 2019-03-07 10:51:10:
> If multiple threads muggle with the internal rep of a Tcl_Obj... Where it can be shared across threads, Jan?
Package structure is hashed per interpreter at the moment (iPtr->packageTable, and pkgPtr is supplied via clientDataPtr inside NRE-calbacks). Therefore I'd prefer Don's solution. jan.nijtmans added on 2019-03-07 11:06:41:
Well, looking further at it, I think Sebres is correct: There is a global shared list of loaded packages in tclLoad.c, but that's not used in tclPkg.c. So - indeed - sharing is no problem. If sharing is no problem, the Don's solution is indeed better. No hard feelings whatever way ;-) dgp added on 2019-03-07 13:47:56:
Yes, the "Pkg" vs "Package" confusion has caught me before too. Anyone have a test yet? dgp added on 2019-03-08 14:46:47:
Test and fix committed for 8.6.10 release. Thanks for the good report! | ||||
