Tcl Library Source Code

Old Tcl scripts running on 8.6.x+ with corresponding Tcllib for crc32 package produce a checksum.
Now with Tcl 9.0.1 and corresponding Tcllib the crc32 package do the same,
but for some file's content an error occur.

Error message from "::crc::crc32" is "integer value too large to represent".
Problem seems to be inside of an operation of
"::crc::Crc32Update" and/or "::crc::Crc32Final".
If You install BAWT 3.0.1 64-Bit Tcl/Tk 9.0.1 for/on Windows 10/11,
You can check at least with the two following smallest files available:
C:\Tcl\lib\poTcllib3.0.0\poExec.tcl      2024-08-16 20:26:04   4108
C:\Tcl\lib\tcllib2.0\valtype\luhn5.tcl   2024-09-13 19:48:12   4121

Both files have correct UTF-8 encoding with simple UNIX-LF.
So there is another reason for this problem than the encoding of the file's contents.
(My first thinking is therefore wrong and I changed this description.)
Can anybody debug deeper into this?

Simple test in Tcl:
package require crc32
::crc::crc32 -filename "C:/Tcl/lib/tcllib2.0/valtype/luhn5.tcl"

I use on ActiveTcl 8.6.4.1 crc32 1.3.2 and found it on BAWT-distribution 3.0.1,
Tcllib 2.0, crc32 1.3.4 thanks.

Best greetings, Thomas [TWu]

Hello together,
some update for Version 9.0.4 (BAWT 3.3.0) with included tcllib2.1 and tcllibc2.1.
The contained crc32 package, version 1.3.4 uses critcl (not trf), see output of "parray ::crc::accel".
Script "package require crc32;crc::crc32 -file {C:\Program Files\Tcl\lib\poTcllib3.3.0\poExec.tcl}" results still in error
>>>integer value too large to represent
    while executing
"Crc32_c $data $sum"
    (procedure "Crc32Update" line 6)
    invoked from within
"Crc32Update $tok [read $opts(-channel) $opts(-chunksize)]"
    (procedure "crc::crc32" line 42)<<<
The value of array variable ::crc::<id-number>(sum) is "-3632003085".
Same for file {C:\Program Files\Tcl\lib\tcllib2.1\\valtype\luhn5.tcl} with sum value "-3152562668".
The internal file handle is left open (memory/resource leak!).
The ::crc::<id-number> arrays are left over too.
The signbit is correct with value "2147483648".
As long as I can't look inside the C code of Crc32_c and others, I can't say anything about the wrong/negative return value.
Maybe a compiler or size problem?
Thanks for all Your work! Best greetings, Thomas.

Hello Harald,
possible differences between systems und distributions? I do not change anything, only install (now) BAWT 3.1.0 with Tcl/Tk 9.0.2 inkl. same crc 1.3.4 and get the same error messages and left open file after teh error. The example is so small, I can not see, where the difference comes from. I'm under Windows 11 Home in 64-Bit. What else I can do to report?
Have wunderfull Advent! Thomas [TWu]

Closing as no reaction

I have tested this with tcllib 2.0 and tcl 2.0 release and tcl 8.6.13 and tcl 9.0.2.

Both give the same result without error:

% lappend auto_path {C:\test\tcllib20\tcllib2.0}
//zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl C:/myprograms/tcl9.0_x64/lib //zipfs:/lib/tk/tk_library //zipfs:/lib/tk/tk_library/ttk {C:\test\tcllib20\tcllib2.0}
% package require crc32
1.3.4
% set p {C:\test\tcllib20\tcllib2.0\valtype\luhn5.tcl}
C:\test\tcllib20\tcllib2.0\valtype\luhn5.tcl
% ::crc::crc32 -filename $p
2493131738

By source code inspection, the relevant line is there:

fconfigure $opts(-channel) -translation binary

This implies the binary (8859-1) encoding.

Sorry, I dont see any issue.

Take care, Harald

@Pat Thoyts or @apnadkarni:
crc32 with option "-file" should use the opened file using "fconfigure -encoding iso8859-1" to prevent problems from binary files.
Loading the files into memory (so them are all in unicodes from 0 to 255) and use crc32 with option "-data" brings the same error.
The only difference for "-file" is the not closing/not releasing the file handle.
This may occur on Tcl 9.0 by reading binary codes in the file which can not decoded.
As Uli E. stated, it may be work-around using the not documented option "-chunksize".
But for really large files/data this is not the solution alone.
The problem occurs on combining the crc32 result for each chunk.
Maybe for Tcl 9.0 there is a need for another sign-handling?
Additionally, with the error "integer value too large to represent" the handle for option -file is leaved open too.
Because both named (examples) files are in correct (strict) encoding,
the calculation is the first part, the correct fconfigure the second part, and
try-catch errors to finally close the opened file the third part to be resolved.
As mentioned, the same may be necessary for crc16, cksum and sum too!

Thanks for the ticket. I have never used this and I am not involved.
I suppose, a CRC is only supported for data with values 0 to 255, so no unicode support.
If unicode should be CRC'd, one should use an encoding.

For example:

::crc::crc32 -data [encoding convertto utf-8 $data]

Take care,
Harald

Hello Harald,
I assign it to You now to get the right person look in here.
Please forward reassign it. Thank You and greetings!
Thomas W.

Additional problem:
while the summing of the block-results goes wrong and is not catched,
the routine fails before closing the opened file.
So there are memory leaks and resource leaks too!
I do not look inside tcl- and c-code - but catch and try-final should be
the solution for this case.

Uli E. stated the same error under Linux (Wine) for Tcl 9.0.2.
He used a 100 kB PNG-file. His work-around is the undocumented switch
"-chunksize". But we can't load really big files completely in memory!
The chunksize my be a good hint for the solution.

Please see https://wiki.tcl-lang.org/page/expr+problems+with+int
and https://wiki.tcl-lang.org/page/Proper+integers+for+Tcl+8.5+or+9.0
Maybe these are points to here as error-message and check-sum is cited there.