| Ticket UUID: | 624919 | ||
| Title: | Tcl_AppendToObj docs confusing | ||
| Type: | Bug | Created on: | 2002-10-17 22:07:20 |
| Submitter: | nobody | Assigned to: | dgp |
| Subsystem: | 10. Objects | Severity: | |
| Priority: | 4 | Last modified: | 2009-07-29 17:00:42 |
| Status: | Open | Closed by: | dkf |
| Resolution: | Remind | Closed on: | 2002-10-22 12:21:21 |
| Version: | obsolete: 8.4.0 | ||
| Description: | ||||
TCL 8.4.0 Windows XP info exists array(name) fails when name is long. Example (show both working and failing functions) % set studyuid $a(0020 000d) 1.2.840.113619.2.43.16112.2141964.87.55.870876696.1 % set seriesuid a(0020 000e) 1.2.840.113619.2.43.16112.2141964.41.48.870879458.1 5 % set set study($studyuid) $seriesuid 1.2.840.113619.2.43.16112.2141964.41.48.870879458.1 5 % puts $study($studyuid) 1.2.840.113619.2.43.16112.2141964.41.48.870879458.1 5 % set study(test) 2 2 % puts [info exists study(test)] 1 % set b test test % puts [info exists study($b)] 1 % puts [info exists study($studyuid)] 0 This should be 1 | ||||
| User Comments: | ||||
dgp added on 2002-10-18 10:55:23:
Logged In: YES user_id=80530 Can anyone else make sense of this? Can the submitter try again? A cut and paste of an actual interactive session, or a demo script would be an improvement. dkf added on 2002-10-18 16:42:29:
Logged In: YES user_id=79902 Beats me what's going on, though those long strings remind me of LDAP, so if there's a problem with an extension mutating objects when it shouldn't, that could be what's going on. In any case, it works for me going on the basis of what I can type in. nobody added on 2002-10-21 22:49:03:
Logged In: NO
Yeah, worked for me on what I could type in, too, but the bug
persists. I did some more digging. Here's the scenario.
The offending array arguments have embedded nulls at the
end. The objects are created in C:
dicomDumpObject( ...tcl args ...) {
char buffer[8192];
Tcl_Obj *theSize, *element, *theTagd, *theTagl,
*elementsize, *theList, *theString;
sprintf(buffer,"");
element = Tcl_NewStringObj(buffer, strlen(buffer));
theString = Tcl_NewStringObj("ELEMENTLIST", -1);
theList = Tcl_NewListObj(1, &theString);
stringLength = elementItem->element.length;
strncpy((char *)scratch, elementItem->element.d.string,
stringLength);
scratch[stringLength] = '\0';
Tcl_AppendToObj(element,(char *)scratch,stringLength);
Tcl_ListObjAppendElement(NULL,theList,element);
Tcl_SetObjResult(interp,theList);
return(TCL_OK);
}
There are other items in the list, such that the entire list is
putarray format, so to read the objects into TCL, I use:
array set a [dicomDumpObject $o]
This populates the array correctly, but the subscripts that
contain appended nulls fail when using [info exists a($v)].
Yes, the string array names are ASN values, probably much
like LDAP.
This procedure uses TCL from end to end, after the values
are copied in using Tcl_AppendToObj, which according to the
man page, handles almost anything.
-Mahlon
dgp added on 2002-10-21 23:08:17:
Logged In: YES user_id=80530 I think we have to assume this is a bug in your program, unless you provide a complete bit of code that supplies legal arguments to Tcl_AppendToObj(), but then produces results that are contrary to the documentation. Your followup is an improvement over the original report, but still does not provide enough information for anyone else to reproduce your problem. (What is "elementItem" ? What values do *scratch and stringLength have when passed into Tcl_AppendToObj(). etc...) nobody added on 2002-10-21 23:55:12:
Logged In: NO
Fair enough. Here's a sample that fails.
C Source code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#ifdef _MSC_VER
#include <io.h>
#include <fcntl.h>
#include <winsock.h>
#endif
#include "tcl.h"
int test_tcl(ClientData clientData, Tcl_Interp *interp, int argc,
char **argv) {
int i;
char buffer[512];
Tcl_Obj *element, *theList, *theString;
theString = Tcl_NewStringObj("ELEMENTLIST", -
1);
theList = Tcl_NewListObj(1, &theString);
sprintf(buffer,"");
element = Tcl_NewStringObj(buffer, strlen(buffer));
strcpy(buffer,"21 test");
i = strlen(buffer);
buffer[i] = (char) NULL;
i++;
Tcl_AppendToObj(element,buffer,i);
Tcl_ListObjAppendElement(NULL,theList,element);
Tcl_SetObjResult(interp,theList);
return(TCL_OK);
}
#ifdef WIN32
__declspec(dllexport)
#endif
int Testtcl_Init(Tcl_Interp *interp)
{
Tcl_CreateCommand
(interp, "testobject", test_tcl, (ClientData) NULL,
(Tcl_CmdDeleteProc *)NULL);
return(TCL_OK);
}
Compile the source into a shared library (I've done this on
both PC and SGI, and both fail).
Then start tclsh and execute this script:
% load testtcl.dll
% array set a [testobject]
% parray a
a(ELEMENTLIST) = 21 test
% set b $a(ELEMENTLIST)
21 test
% string length $b
8
% set r($b) 2
2
% puts $r($b)
2
% info exists r($b)
0
-Mahlon
hobbs added on 2002-10-22 00:14:06:
Logged In: YES user_id=72656 Woah, bogosity filter hitting hard: strcpy(buffer,"21 test"); i = strlen(buffer); buffer[i] = (char) NULL; i++; Tcl_AppendToObj(element,buffer,i); What's with i++ here? That's telling AppendToObj to take more bytes than are valid out of buffer ... mahlonstacy added on 2002-10-22 00:25:04:
Logged In: YES user_id=595029 No, I don't think so. strlen(buffer) is 7; i = 7 range of buffer[] is 0 - 6 buffer[7] = NULL sets the 8th char to NULL i++ increments the length to 8; in this example, buffer[7] was already null because we used strcpy. But in my working program, the values are not null terminated, they are described by length. Using the construct above just guarantees a null at the end of the value. Also, buffer is declared as an array... there's no overrun. hobbs added on 2002-10-22 00:30:45:
Logged In: YES user_id=72656 That's not the correct thing to do. Tcl_Obj's are supposed to be utf-8 correct as strings, with the minor exception that NULLs are represented as two bytes (\xC0\x80 IIRC) to allow them to be passed around safely. The violation of this *may* cause problems, which was the red flag that waved at me. mahlonstacy added on 2002-10-22 00:34:47:
Logged In: YES user_id=595029 Makes sense. But shouldn't either a) Tcl_AppendToObject catch and fix an embedded NULL, and/or b) whatever the subscript, if you can print the value of an object, shouldn't [info exists] on that same object always be true? hobbs added on 2002-10-22 00:40:19:
Logged In: YES user_id=72656 Do you really intend to have the NULL there, or do you just want to ensure that it's null terminated? If the latter, don't do anything extra - Tcl handles that. If the former, then you should either be using the Tcl_ByteArrayObj stuff, or you should use Tcl_ExternalToUtf and friends. That said: a) No, there are lots of other APIs to handle that, as noted above. b) This may indicate exactly where the problem is. While you can print it just fine (and it may be holding the NULL in there, you just can't see it), the info exists may not include the null when it passes the value through a strlen or such (that's why NULL get's special encoding), which is the source of the problem you are seeing. mahlonstacy added on 2002-10-22 00:48:46:
Logged In: YES user_id=595029 OK, thanks Jeff. This complicates the coding for me, but I understand the issues. I do need to keep the NULL, when it's present, but I'll have to manage the use of the value as a subscript in another way. hobbs added on 2002-10-22 00:52:21:
Logged In: YES
user_id=72656
I don't think it does complicate things at all, you are just
looking at the wrong kind of object. A "String" is a utf-8
string. You want "ByteArray"s, so string map {String
ByteArray} in your code - the APIs are all there.
mahlonstacy added on 2002-10-22 01:49:23:
Logged In: YES user_id=595029 OK, I converted the offending TclStringObj to TclByteArrayObj, without other changes, and it seems to work OK. (we'll do more testing). FWIW, I interpreted the man page for Tcl_AppendToObj to suggest that the function would properly encode any string passed into it. Guess not. Thanks for clearing this up. dkf added on 2002-10-22 15:30:40:
Logged In: YES user_id=79902 Hmm. I agree that the Tcl_AppendToObj documentation could be much clearer, and might possibly even be wrong; can we *really* take embedded NULs in the "bytes" argument correctly, or do we always need them two-byte encoded if they are not the end-of-string marker? dkf added on 2002-10-22 16:44:32:
Logged In: YES user_id=79902 Review of the code indicates that the manpage is wrong; 'bytes' may not contain NUL bytes (well, it won't cause a memory fault, but strange effects - as seen here - might happen) except as an end-of-string marker when 'length' is -1 (or not long enough to overlap the NUL). Indeed, it is not even documented that 'bytes' is UTF8! Will fix... dkf added on 2002-10-22 19:21:21:
Logged In: YES user_id=79902 Hmm. The documentation was completely out of synch with the implementation; what was written and what was true were quite different things, and had been since Tcl8.1... dgp added on 2002-10-22 21:08:51:
Logged In: YES user_id=80530 Note that this is a general problem. Uncertainty about what encoding is required for the string pointed to by objPtr->bytes. It arose in Tcl Bug 584603 as well. If we are requiring UTF-8, then there's probably additional places in the docs to note this. Also, do we know of any extensions/users of Tcl_Obj's that have used the documented freedom to have non-encoded embedded NULLs in the counted strings pointed to by objPtr->bytes that we are now belatedly declaring illegal? Note in particular that Tcl's own command [encoding convertfrom identity] is now illegal by this documentation change, since it can return a Tcl_Obj with a non-UTF8 string rep. dkf added on 2002-11-16 19:33:11:
Logged In: YES user_id=79902 [encoding convertfrom identity] doesn't fill me with glee I must admit. In fact, I'd be ever so much happier if it didn't exist. IMO, we should distinguish between the way that code-points are encoded as bytes, and the way that characters are mapped to code-points. In particular, [encoding] should be used to alter the mapping between characters and code-points (e.g. code-point 00A4 is the international currency symbol in UNICODE but is the euro symbol in ISO 8859-15). Where I get worried is when we start using it to muck around with the mapping to bytes; I'm of the opinion that the mapping of code-points to bytes should be (in the bytes-part of a Tcl_Obj, and consequently in Tcl_AppendToObj) based on that of UTF-8 (people can also use (possibly lossy) single-byte mappings via byte arrays and double-byte mappings via the string object type.) So what does this all mean? Well, it means that [encoding convertfrom identity] Must Die. There should not be any way to produce non-UTF8 bytes content (using the term broadly as a way of encoding code-points as bytes, and not providing those CPs with an interpretation.) dgp added on 2003-01-27 23:42:48:
Logged In: YES user_id=80530 Note that [tcltest::bytestring] is equivalent to (implemented as) [encoding convertfrom identity]. This is apparently useful to be able to test/compare strings that are not valid UTF-8. dkf added on 2003-01-28 20:23:32:
Logged In: YES user_id=79902 As far as I'm concerned, it's a fault for a Tcl_Obj string rep to contain non-UTF8 data, and the behaviour of a great fraction of the core is undefined in that case. If anyone needs this sort of thing, perhaps they should consider refactoring (perhaps with the aid of the [binary] command to indicate that it is the representation that is being examined...) If I'm wrong in this, please explain exactly how and exactly what is being looked for in code that does this sort of trick. dgp added on 2003-01-29 00:46:47:
Logged In: YES user_id=80530 The [bytestring] command is used in several tests in the parse*.test and utf.test files. For example, the tests utf-1.* appear to be checking for correct UTF-8 encoding. Is there an alternative way to express these tests without allowing non-UTF-8 strings in objPtr->bytes ? dkf added on 2003-02-05 20:40:55:
Logged In: YES user_id=79902 Hmm. Perhaps we need a way to convert between UTF-8 strings and byte-arrays? I would be keen on methods to convert arbitrary byte-arrays into "strings" (i.e. potentially non-UTF8 contents of the bytes member) being restricted to the testing extension, as such things are for testing of the core only and not for general use. hobbs added on 2003-02-06 00:22:01:
Logged In: YES user_id=72656 The way to convert between utf-8 and byte array is already in Tcl - you create a ByteArray obj and ask for a String obj, or vice versa. dgp added on 2003-02-06 00:26:26:
Logged In: YES user_id=80530 OK, then, one example for the dumb guy, please. How should we rewrite test utf-1.2 so that it would not need to use [bytestring] ? hobbs added on 2003-02-06 00:35:21:
Logged In: YES user_id=72656 You would have to go back to the source to figure out the reasoning of that, but it was likely a desire to not rely on the core too much to test itself, instead creating functions that should be the same. Good testing principle. utf-1.2 is peculiar because it exposes the one oddity of Tcl utf- 8 that is non-standard: \x00 == \xC0\x80. It may be that bytestring is the one way to trick this up. You may still be able to use 'binary format' though. dkf added on 2003-02-06 21:03:59:
Logged In: YES user_id=79902 Eh? I think there's confusion abounding here. I think that all the handling of the 'identity' encoding is strange from top to bottom, and that it should be replaced with other things. There should be a way to go from UTF8strings to bytearrays such that the single character string \u0000 gets mapped to a two byte bytearray containing \xc0 and \x80. This would fit quite nicely with the [binary] command IMO. There should also be a way to go the other direction, though as this would have the ability to produce strings that are not well-formed UTF8, it would have to be a testing-only command, and hence part of the tcltest executable only. But we know how to do that (and do so already for testing other features of the core.) That would be sufficient for testing all oddities in the core that we come across, and it would have the advantage of making the bytes member of every Tcl_Obj be defined to contain a UTF8 string at all times (when not NULL and you're not testing, of course.) Such assumptions, which are really already there anyway, would make life easier in other places (as outlined in this Bug.) IOW, Jeff I think you've misunderstood. :^) dkf added on 2003-02-10 23:43:51:
Logged In: YES user_id=79902 I've heard that a reason for not getting rid of the [encoding convertfrom/convertto identity] commands is compatability with existing extensions. Is this really the case? (The other encodings are not a problem, since we can still use UTF8 as a code-point to byte-sequence mapping...) dkf added on 2009-07-29 17:00:42:
BTW, the original problem is now completely gone, as of 8.5. [info exists] is bytecode compiled and the storage format of array keys is now object based. | ||||
