Tcl Library Source Code

View Ticket
Login
Bounty program for improvements to Tcl and certain Tcl packages.
Ticket UUID: 3493711
Title: S3: Fix wasteful string conversion of large binary object
Type: Patch Version: None
Submitter: samoconnor Created on: 2012-02-25 04:57:05
Subsystem: amazon-s3 Assigned To: aku
Priority: 5 Medium Severity: Severe
Status: Closed Last Modified: 2014-06-23 18:36:27
Resolution: Fixed Closed By:
    Closed on:
Description:
In our app S3::Get is used to download large binary objects from AWS (many hundreds of MB).
The current code stores the downloaded object in "x" then does: S3::debug "Body: $x"
This causes "x" to be converted to a string, and creates a second huge string "Body: .....".
The patch below passes the large object directly from the binary network channel to the result dict entry.

A small test program that downloads a 5MB file uses 24MB before the patch vs 9MB with the patch.

--- /System/Library/Tcl/tcllib1.12/amazon-s3/S3.tcl.orig2012-02-25 15:32:45.000000000 +1100
+++ /System/Library/Tcl/tcllib1.12/amazon-s3/S3.tcl2012-02-25 15:33:25.000000000 +1100
@@ -580,9 +580,7 @@
 if {[dict exists $thunk orig outchan]} {
     fcopy $s3 [dict get $thunk orig outchan]
 } else {
-    set x [read $s3]
-    dict set thunk outbody $x
-    S3::debug "Body: $x"
+    dict set thunk outbody [read $s3]
 }
 return [S3::nextdo all_done $thunk readable]
     } else {
User Comments: aku added on 2014-06-23 18:36:27:
Fixed contact information on closed/fixed ticket.

aku added on 2014-06-20 18:36:47:
Applied in revision [5e9d8b632f].
Committed.
Pushed.
Thank you.
And apologies for the long delay.

Better fixes to consider:
- Modify the S3::debug to subst its arguments only when debugging is active.
- Replace S3::debug with use of the 'debug' package in Tcllib (which does defer argument expansion to when it is active).

These might be moot, depending on the overall fate of the S3 module and packages.