D 2018-04-01T14:06:53.353
J foundin 1.7.16
J icomment This\sbug\smight\snot\sbe\sin\sTcltls:\sit\smight\sbe\sin\sTcl\sstacked\schannels,\sthe\sTcl\r\n[read]\scommand,\sfileevent\sgeneration\sor\sbuffering.\r\n\r\nThe\sbug\soccurs\swhen\susing\sTcltls\s1.7.16\swith\sthe\sstock\sTcl\s8.6.8\sand\r\nLibreSSL\s2.6.4,\sbut\shas\sexisted\sfor\sa\slong\stime:\sit\salso\soccurs\sin\solder\r\nversions\sat\sleast\sas\sfar\sback\sas\sTcltls\s1.6.7,\sTcl\s8.5.13\sand\sOpenSSL\s1.02d.\r\n\r\nWhen\sa\schannel\sis\sopened\swith\stls::socket,\sa\snon-blocking\s[read]\sdoes\snot\r\nalways\sreturn\sthe\savailable\sdata,\ssometimes\s(if\s-blocksize\sis\s4096)\seven\sif\sit\r\nexceeds\sthe\ssize\srequested;\sinstead\sit\swaits\suntil\sthe\sremote\sserver\scloses\r\nthe\sconnection.\r\n\r\nWhen\susing\shttp::geturl\swith\stls::socket,\sa\sHTTPS\srequest\sfor\san\sentity\sthat\r\nis\slarger\sthan\sthe\s-blocksize\shangs\sif\sthe\sserver\sresponse\suses\sa\spersistent\r\nconnection\sand\sdoes\snot\suse\schunked\sencoding\s(but\sdoes\ssupply\san\saccurate\r\nContent-Length\sheader).\s\sThe\sclient\shas\ssome\ssuccessful\s[reads],\sbut\sthen\r\ndoes\snothing\suntil\sthe\sserver\stimes\sout,\salthough\sthe\sfull\sresponse\shas\sbeen\r\nreceived\sand\sis\sbuffered\ssomewhere.\r\n\r\nThe\srelevant\spart\sof\shttp\sis\sin\sthe\scommand\shttp::Event,\sfor\sunchunked\sdata\r\nwith\sknown\ssize\s$state(totalsize)\sobtained\sfrom\sa\sContent-Length\sheader.\s\sThe\r\nrelevant\spart\sof\sthe\scode\scan\sbe\sabstracted\sas:\r\n\r\nfconfigure\s$sock\s-blocking\soff\s-translation\sbinary\s-buffersize\s$state(-blocksize)\r\nfileevent\s$sock\sreadable\s[list\shttp::Event\s$sock\s$token]\r\n\r\nproc\shttp::Event\s{sock\stoken}\s{\r\n\s\s\s\svariable\s$token\r\n\s\s\s\supvar\s0\s$token\sstate\r\n\r\n\s\s\s\s#\s1.\sHow\smany\sbytes\sto\sask\sfor.\r\n\s\s\s\sset\sreqSize\s$state(-blocksize)\r\n\r\n\s\s\s\s#\s2.\sRead.\r\n\s\s\s\sset\sblock\s[read\s$sock\s$reqSize]\r\n\s\s\s\sset\sn\s[string\slength\s$block]\r\n\s\s\s\sif\s{$n\s>=\s0}\s{\r\n\s\s\s\s\s\s\s\sappend\sstate(body)\s$block\r\n\s\s\s\s\s\s\s\sincr\sstate(currentsize)\s$n\r\n\s\s\s\s}\r\n\r\n\s\s\s\s#\s3.\sCheck\sfor\send\sof\sdata.\r\n\s\s\s\sif\s{$state(currentsize)\s>=\s$state(totalsize)}\s{\r\n\s\s\s\s\s\s\s\sEof\s$token\r\n\s\s\s\s}\r\n\s\s\s\sreturn\r\n}\r\n\r\nThe\sbug\scan\sbe\sworked\saround\sby\sprecisely\sspecifying\sthe\ssize\sof\sthe\r\nuntransferred\sresponse\sbody.\r\n\r\n\s\s\s\s#\s1.\sHow\smany\sbytes\sto\sask\sfor.\r\n\s\s\s\sset\sreqSize\s[expr\s{$state(totalsize)\s-\s$state(currentsize)}]\r\n\r\nThis\sworkaround\sfails\sif\sreqSize\sis\scapped\sat\s$state(-blocksize):\r\n\r\n\s\s\s\s#\s1.\sHow\smany\sbytes\sto\sask\sfor.\r\n\s\s\s\sset\sreqSize\s[expr\s{min($reqSize,\s$state(-blocksize))}]\r\n\r\nWhen\sthe\shanging\soccurs,\smore\sthan\s$state(-blocksize)\sbytes\scan\sbe\savailable,\r\n(if\s-blocksize\sis\s4096)\sbut\sthey\sare\snot\sreturned\sby\s[read]\suntil\stimeout.\r\n\r\nThe\sproperties\sread\sby\s[fconfigure\s$sock]\s(when\sadded\sto\shttp::Event)\sare:\r\n\s\s-blocking\s0\r\n\s\s-buffering\sfull\r\n\s\s-buffersize\s8192\r\n\s\s-encoding\sbinary\r\n\s\s-eofchar\s{{}\s{}}\r\n\s\s-translation\s{lf\scrlf}\r\nThe\slast\sthree\sproperties\scorrespond\s(for\sreading)\sto\sthe\sresults\sof\sthe\scommand\r\n\s\sfconfigure\s$sock\s-translation\sbinary\r\n\r\nread(n)\sdescribes\snon-blocking\smode\swith\stwo\sarguments:\r\n\r\n\s\s\s\s\s\sread\schannelId\snumChars\r\n\r\n\s\sIf\s\schannelId\sis\sin\snonblocking\smode,\sthe\scommand\smay\snot\sread\sas\smany\r\n\s\scharacters\sas\srequested:\sonce\sall\savailable\sinput\shas\sbeen\sread,\sthe\scommand\r\n\s\swill\sreturn\sthe\sdata\sthat\sis\savailable\srather\sthan\sblocking\sfor\smore\sinput.\r\n\s\sIf\sthe\schannel\sis\sconfigured\sto\suse\sa\smulti-byte\sencoding,\sthen\sthere\smay\r\n\s\sactually\sbe\ssome\sbytes\sremaining\sin\sthe\sinternal\sbuffers\s\sthat\sdo\snot\sform\r\n\s\sa\s\scomplete\scharacter.\s\sThese\sbytes\swill\snot\sbe\sreturned\suntil\sa\scomplete\r\n\s\scharacter\sis\savailable\sor\send-of-file\sis\sreached.\s\sThe\s-nonewline\sswitch\sis\r\n\s\signored\sif\sthe\scommand\sreturns\sbefore\sreaching\sthe\send\sof\sthe\sfile.\r\n\r\nThe\shanging\soccurs\seven\swith\sa\sbinary\sresource,\sfetched\swith\s-encoding\sbinary,\r\nso\sthe\smulti-byte\sissue\sdoes\snot\sarise.\s\sThere\sis\sno\sproblem\swith\shttp,\sonly\r\nwith\shttps\swhere\s[tls::socket]\sis\sused\sin\splace\sof\s[socket].\sread(n)\smakes\r\nclear\sthat\sin\snonblocking\smode,\sthe\scommand\swill\snot\swait\sto\sreceive\sthe\snumber\r\nof\scharacters\srequested.\r\n\r\nExample\scode\sthat\sdemonstrates\sthe\sdelay:\r\n\r\n#\sThe\shttp\spackage\ssets\sboth\sthe\s-buffersize\sfor\sthe\ssocket,\sand\sthe\snumber\sof\r\n#\sbytes\srequested\swith\s[read],\sto\sthe\shttp::geturl\soption\s-blocksize,\swhich\shas\r\n#\sdefault\svalue\s8192.\r\n\r\npackage\srequire\shttp\r\n\r\nset\sstart\s[clock\smilliseconds]\r\n\r\nproc\shttp::Log\s{args}\s{\r\n\s\s\s\sset\stime\s[expr\s{[clock\smilliseconds]\s-\s$::start}]\r\n\s\s\s\sputs\sstderr\s[list\s$time\s{*}$args]\r\n\s\s\s\sflush\sstderr\r\n\s\s\s\sreturn\r\n}\r\n\r\npackage\srequire\stls\r\nhttp::register\shttps\s443\s[list\s::tls::socket\s-ssl2\s0\s-ssl3\s0\s-tls1\s1\s-tls1.1\s1\s-tls1.2\s1]\r\n\r\nproc\sGetFromWeb\s{blocksize\surl}\s{\r\n\s\s\s\sset\s::prevSize\s0\r\n\s\s\s\sset\stoken\s\s\s\s[::http::geturl\s$url\s\s\s\s\s\\\r\n\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s-blocksize\s$blocksize\s\s\\\r\n\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s-keepalive\s1\s\s\s\s\s\s\s\s\s\s\s\\\r\n\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s-progress\shttpProgress\s\\\r\n\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s-headers\s{accept-encoding\sunsupported}\s\\\r\n\s\s\s\s]\r\n\s\s\s\sarray\sget\s$token\r\n\s\s\s\s#\sN.B.\sImplicit\sReturn\r\n}\r\n\r\nproc\shttpProgress\s{token\stotal\scurrent}\s{\r\n\s\s\s\supvar\s0\s$token\sstate\r\n\s\s\s\sset\smsg\s{,\sand\stherefore\swill\snot\sdemonstrate\sthe\sbug.}\r\n\r\n\s\s\s\sif\s{\s\s\s\s[info\sexists\sstate(transfer)]\r\n\s\s\s\s\s\s\s\s\s&&\s($state(transfer)\seq\s"chunked")\r\n\s\s\s\s}\s{\r\n\tputs\s"The\sresponse\suses\schunked\stransfer\sencoding,\s$msg"\r\n\s\s\s\s}\selseif\s{$total\s==\s0}\s{\r\n\tputs\s"The\sserver\shas\snot\sreported\sthe\ssize\sof\sthe\sentity,\s$msg"\r\n\s\s\s\s}\r\n\s\s\s\sif\s{$state(connection)\seq\s{close}}\s{\r\n\tputs\s"The\sserver\swill\sclose\sthe\sconnection,\s$msg"\r\n\s\s\s\s}\r\n\s\s\s\sset\stime\s[expr\s{[clock\smilliseconds]\s-\s$::start}]\r\n\s\s\s\sset\sextra\s[expr\s{$current\s-\s$::prevSize}]\r\n\s\s\s\sset\s::prevSize\s$current\r\n\s\s\s\sputs\s"$time\sFetched\s$extra\sbytes\s-\saccumulated\s$current\sbytes\sof\stotal\s$total"\r\n\s\s\s\sflush\sstdout\r\n}\r\n\r\n\r\nset\sblocksize\s8192\r\n\r\n#\sThis\sURL\sserves\s(at\sleast\sin\sthe\sUK)\sa\sresource\sof\sabout\s300kB,\sunchunked,\sand\r\n#\stimes\sout\safter\sapprox\s13s.\r\n\r\nset\surl\shttps://www.bbc.co.uk/\r\n\r\n#\sThese\sURLs\sdemonstrate\sthat\shttp\ssites\s(using\s[socket])\sdo\snot\r\n#\strigger\sthe\sbug.\r\n\r\n#\sset\surl\shttp://news.mit.edu/sites/mit.edu.newsoffice/files/styles/news_article_image_top_slideshow/public/images/2018/MIT-Debris-Disks_0.jpg?itok=kGim01Xz\r\n#\sset\surl\shttp://static01.nyt.com/images/2018/03/30/theater/30threetallwomen2/merlin_135911256_e2911f33-8b35-445d-90ee-d15457c19fae-superJumbo.jpg\r\n\r\n#\sThe\scorresponding\shttps\sURLs\sdo\strigger\sthe\sbug,\sbut\sare\sinconvenient\sto\suse\r\n#\sbecause\sthe\sserver\sdoes\snot\stime\sout\sfor\sabout\s10\sminutes.\r\n\r\n#\sset\surl\shttps://news.mit.edu/sites/mit.edu.newsoffice/files/styles/news_article_image_top_slideshow/public/images/2018/MIT-Debris-Disks_0.jpg?itok=kGim01Xz\r\n#\sset\surl\shttps://static01.nyt.com/images/2018/03/30/theater/30threetallwomen2/merlin_135911256_e2911f33-8b35-445d-90ee-d15457c19fae-superJumbo.jpg\r\n\r\nset\sstart\s[clock\smilliseconds]\r\nset\simg\s[GetFromWeb\s$blocksize\s$url]\r\nset\sNUL\s{}
J login anonymous
J mimetype text/plain
J private_contact a774b1f5653dc3360e29ea7d635a64e94c9f1dab
J severity Important
J status Open
J title Buffering\suntil\stimeout
J type Incident
K 94c6a431fee47acdb590ee3963704ef1d756a5cf
U anonymous
Z 9eddb107e8f1264f0e1ebef61463c431