Tcl Source Code

View Ticket
Login
Ticket UUID: 21b0629c81fbe38ad153cd8fd899626200a66e15
Title: 0-day vulnerability - insufficient escape by exec of batch-files for windows
Type: Bug Version: all (win)
Submitter: sebres Created on: 2018-08-17 13:09:03
Subsystem: 16. Commands A-H Assigned To: sebres
Priority: 9 Immediate Severity: Critical
Status: Closed Last Modified: 2023-11-30 19:19:26
Resolution: Fixed Closed By: oehhar
    Closed on: 2023-11-30 19:19:26
Description:

[UPDATED]
0-day vulnerability in tcl-exec (for win) by calling of batch-files resp. default windows command processor.

% exec my-echo.cmd {test &whoami} 
  "test &whoami"
% exec my-echo.cmd {test"&whoami} 
  test\"&whoami
% exec my-echo.cmd {test&whoami}
  test
  my_domain\sebres
It may be expected in the direct invocation of cmd.exe /c ... as regards it accepts many special meta-chars (so some pre-processing on the part of command-processor may be required). But the same pre-processing by invocation of the batch-files is very dangerous, so is unwanted as vulnerable.

Rather it is a matter the weird windows unescape behavior in case of command processor,
therefore almost everything else beside Tcl is affected by same vulnerability also similar way.

For example python (for win):

>>> subprocess.call(['my-echo.cmd', 'test &whoami'])
  "test &whoami"
>>> subprocess.call(['my-echo.cmd', 'test"&whoami'])
  test\"&whoami
>>> subprocess.call(['my-echo.cmd', 'test&whoami'])
  test
  my_domain\sebres

See github.com/sebres/PoC/SB-0D-001-win-exec for more extensive PoC with additional info (and more lang's).

Thanks to Peter (resuna) for link to discussion

It looks like the arguments containing unpaired quote-chars somewhere will cause processing of the special meta-characters, so it should be extra escaped using tripple circumflex but only for unpaired quote-chars (by paired quotes it remains the meta-char as well as ^^^ sequence).

For more extensive PoC I wrote a small helper dumping arguments (here test-dump.exe) as well as small batch (test-dump.cmd) doing the same:

@pushd %~dp0
@rem echo %*
@test-dump.exe %*
@popd

Instead of test-dump.exe one can use following code as test-dump.tcl (with adjusting the invocation in test-dump.cmd)

catch { puts -nonewline "    `[file tail $::argv0]´ `[join $::argv "´ `"]´"; flush stdout }

And use following tcl-script (test-dump-inv.tcl) to try or validate different variants (to find the "injuries"):

set testinv {test-dump.exe test-dump.CMD}

set tn 0 foreach a { {test"whoami} {test""whoami} {test&whoami} {test|whoami} {"test&whoami} {"test|whoami} {test"&whoami} {test"|whoami} {"test"&whoami} {"test"|whoami} {""test"&whoami} {""test"|whoami} {test&echo "} {test|echo "} {"test&echo "} {"test|echo "} {test"&echo "} {test"|echo "} {"test"&echo "} {"test"|echo "} {""test"&echo "} {""test"|echo "} {test&echo ""} {test|echo ""} {"test&echo ""} {"test|echo ""} {test"&echo ""} {test"|echo ""} {"test"&echo ""} {"test"|echo ""} {""test"&echo ""} {""test"|echo ""} {test>whoami} {test<whoami} {"test>whoami} {"test<whoami} {test">whoami} {test"<whoami} {"test">whoami} {"test"<whoami} {""test">whoami} {""test"<whoami} {test(whoami)} {test(whoami)} {test"(whoami)} {test"(whoami)} {test^whoami} {test^^echo ^^^} {test"^whoami} {test"^^echo ^^^} } { puts [string repeat - 20] unset -nocomplain prevr foreach cmd $testinv { puts -nonewline [format "*%3d) `%s´" \ $tn [join [list test-dump[file extension [lindex $cmd 0]] $a] "´ `"]] if {[catch { set r [exec {*}$cmd $a] } r]} { set r "ERROR: $r" } if {![info exists prevr] || $prevr eq $r} { puts "" } else { puts " -- *VULNERABLE*" } set prevr $r puts [regsub -all -line {^} $r " "] } incr tn }

The results:

> tclsh.exe test-dump-inv.tcl
--------------------
*  0) `test-dump.exe´ `test"whoami´
      `test-dump.exe´ `test"whoami´
*  0) `test-dump.CMD´ `test"whoami´
      `test-dump.exe´ `test"whoami´
--------------------
*  1) `test-dump.exe´ `test""whoami´
      `test-dump.exe´ `test""whoami´
*  1) `test-dump.CMD´ `test""whoami´
      `test-dump.exe´ `test""whoami´
--------------------
*  2) `test-dump.exe´ `test&whoami´
      `test-dump.exe´ `test&whoami´
*  2) `test-dump.CMD´ `test&whoami´ -- *VULNERABLE*
      `test-dump.exe´ `test´my_domain\sebres
--------------------
*  3) `test-dump.exe´ `test|whoami´
      `test-dump.exe´ `test|whoami´
*  3) `test-dump.CMD´ `test|whoami´ -- *VULNERABLE*
  my_domain\sebres
--------------------
*  4) `test-dump.exe´ `"test&whoami´
      `test-dump.exe´ `"test&whoami´
*  4) `test-dump.CMD´ `"test&whoami´
      `test-dump.exe´ `"test&whoami´
--------------------
*  5) `test-dump.exe´ `"test|whoami´
      `test-dump.exe´ `"test|whoami´
*  5) `test-dump.CMD´ `"test|whoami´
      `test-dump.exe´ `"test|whoami´
--------------------
*  6) `test-dump.exe´ `test"&whoami´
      `test-dump.exe´ `test"&whoami´
*  6) `test-dump.CMD´ `test"&whoami´
      `test-dump.exe´ `test"&whoami´
--------------------
*  7) `test-dump.exe´ `test"|whoami´
      `test-dump.exe´ `test"|whoami´
*  7) `test-dump.CMD´ `test"|whoami´
      `test-dump.exe´ `test"|whoami´
--------------------
*  8) `test-dump.exe´ `"test"&whoami´
      `test-dump.exe´ `"test"&whoami´
*  8) `test-dump.CMD´ `"test"&whoami´ -- *VULNERABLE*
      `test-dump.exe´ `"test"´my_domain\sebres
--------------------
*  9) `test-dump.exe´ `"test"|whoami´
      `test-dump.exe´ `"test"|whoami´
*  9) `test-dump.CMD´ `"test"|whoami´ -- *VULNERABLE*
  my_domain\sebres
--------------------
* 10) `test-dump.exe´ `""test"&whoami´
      `test-dump.exe´ `""test"&whoami´
* 10) `test-dump.CMD´ `""test"&whoami´
      `test-dump.exe´ `""test"&whoami´
--------------------
* 11) `test-dump.exe´ `""test"|whoami´
      `test-dump.exe´ `""test"|whoami´
* 11) `test-dump.CMD´ `""test"|whoami´
      `test-dump.exe´ `""test"|whoami´
--------------------
* 12) `test-dump.exe´ `test&echo "´
      `test-dump.exe´ `test&echo "´
* 12) `test-dump.CMD´ `test&echo "´
      `test-dump.exe´ `test&echo "´
--------------------
* 13) `test-dump.exe´ `test|echo "´
      `test-dump.exe´ `test|echo "´
* 13) `test-dump.CMD´ `test|echo "´
      `test-dump.exe´ `test|echo "´
--------------------
* 14) `test-dump.exe´ `"test&echo "´
      `test-dump.exe´ `"test&echo "´
* 14) `test-dump.CMD´ `"test&echo "´ -- *VULNERABLE*
      `test-dump.exe´ `"test´\""
--------------------
* 15) `test-dump.exe´ `"test|echo "´
      `test-dump.exe´ `"test|echo "´
* 15) `test-dump.CMD´ `"test|echo "´ -- *VULNERABLE*
  \""
--------------------
* 16) `test-dump.exe´ `test"&echo "´
      `test-dump.exe´ `test"&echo "´
* 16) `test-dump.CMD´ `test"&echo "´ -- *VULNERABLE*
      `test-dump.exe´ `test"´\""
--------------------
* 17) `test-dump.exe´ `test"|echo "´
      `test-dump.exe´ `test"|echo "´
* 17) `test-dump.CMD´ `test"|echo "´ -- *VULNERABLE*
  \""
--------------------
* 18) `test-dump.exe´ `"test"&echo "´
      `test-dump.exe´ `"test"&echo "´
* 18) `test-dump.CMD´ `"test"&echo "´
      `test-dump.exe´ `"test"&echo "´
--------------------
* 19) `test-dump.exe´ `"test"|echo "´
      `test-dump.exe´ `"test"|echo "´
* 19) `test-dump.CMD´ `"test"|echo "´
      `test-dump.exe´ `"test"|echo "´
--------------------
* 20) `test-dump.exe´ `""test"&echo "´
      `test-dump.exe´ `""test"&echo "´
* 20) `test-dump.CMD´ `""test"&echo "´ -- *VULNERABLE*
      `test-dump.exe´ `""test"´\""
--------------------
* 21) `test-dump.exe´ `""test"|echo "´
      `test-dump.exe´ `""test"|echo "´
* 21) `test-dump.CMD´ `""test"|echo "´ -- *VULNERABLE*
  \""
--------------------
* 22) `test-dump.exe´ `test&echo ""´
      `test-dump.exe´ `test&echo ""´
* 22) `test-dump.CMD´ `test&echo ""´
      `test-dump.exe´ `test&echo ""´
--------------------
* 23) `test-dump.exe´ `test|echo ""´
      `test-dump.exe´ `test|echo ""´
* 23) `test-dump.CMD´ `test|echo ""´
      `test-dump.exe´ `test|echo ""´
--------------------
* 24) `test-dump.exe´ `"test&echo ""´
      `test-dump.exe´ `"test&echo ""´
* 24) `test-dump.CMD´ `"test&echo ""´ -- *VULNERABLE*
      `test-dump.exe´ `"test´\"\""
--------------------
* 25) `test-dump.exe´ `"test|echo ""´
      `test-dump.exe´ `"test|echo ""´
* 25) `test-dump.CMD´ `"test|echo ""´ -- *VULNERABLE*
  \"\""
--------------------
* 26) `test-dump.exe´ `test"&echo ""´
      `test-dump.exe´ `test"&echo ""´
* 26) `test-dump.CMD´ `test"&echo ""´ -- *VULNERABLE*
      `test-dump.exe´ `test"´\"\""
--------------------
* 27) `test-dump.exe´ `test"|echo ""´
      `test-dump.exe´ `test"|echo ""´
* 27) `test-dump.CMD´ `test"|echo ""´ -- *VULNERABLE*
  \"\""
--------------------
* 28) `test-dump.exe´ `"test"&echo ""´
      `test-dump.exe´ `"test"&echo ""´
* 28) `test-dump.CMD´ `"test"&echo ""´
      `test-dump.exe´ `"test"&echo ""´
--------------------
* 29) `test-dump.exe´ `"test"|echo ""´
      `test-dump.exe´ `"test"|echo ""´
* 29) `test-dump.CMD´ `"test"|echo ""´
      `test-dump.exe´ `"test"|echo ""´
--------------------
* 30) `test-dump.exe´ `""test"&echo ""´
      `test-dump.exe´ `""test"&echo ""´
* 30) `test-dump.CMD´ `""test"&echo ""´ -- *VULNERABLE*
      `test-dump.exe´ `""test"´\"\""
--------------------
* 31) `test-dump.exe´ `""test"|echo ""´
      `test-dump.exe´ `""test"|echo ""´
* 31) `test-dump.CMD´ `""test"|echo ""´ -- *VULNERABLE*
  \"\""
--------------------
* 32) `test-dump.exe´ `test>whoami´
      `test-dump.exe´ `test>whoami´
* 32) `test-dump.CMD´ `test>whoami´ -- *VULNERABLE*

-------------------- * 33) `test-dump.exe´ `test<whoami´ `test-dump.exe´ `test<whoami´ * 33) `test-dump.CMD´ `test<whoami´ -- *VULNERABLE* `test-dump.exe´ `test´ -------------------- * 34) `test-dump.exe´ `"test>whoami´ `test-dump.exe´ `"test>whoami´ * 34) `test-dump.CMD´ `"test>whoami´ `test-dump.exe´ `"test>whoami´ -------------------- * 35) `test-dump.exe´ `"test<whoami´ `test-dump.exe´ `"test<whoami´ * 35) `test-dump.CMD´ `"test<whoami´ `test-dump.exe´ `"test<whoami´ -------------------- * 36) `test-dump.exe´ `test">whoami´ `test-dump.exe´ `test">whoami´ * 36) `test-dump.CMD´ `test">whoami´ `test-dump.exe´ `test">whoami´ -------------------- * 37) `test-dump.exe´ `test"<whoami´ `test-dump.exe´ `test"<whoami´ * 37) `test-dump.CMD´ `test"<whoami´ `test-dump.exe´ `test"<whoami´ -------------------- * 38) `test-dump.exe´ `"test">whoami´ `test-dump.exe´ `"test">whoami´ * 38) `test-dump.CMD´ `"test">whoami´ -- *VULNERABLE*

-------------------- * 39) `test-dump.exe´ `"test"<whoami´ `test-dump.exe´ `"test"<whoami´ * 39) `test-dump.CMD´ `"test"<whoami´ -- *VULNERABLE* `test-dump.exe´ `"test"´ -------------------- * 40) `test-dump.exe´ `""test">whoami´ `test-dump.exe´ `""test">whoami´ * 40) `test-dump.CMD´ `""test">whoami´ `test-dump.exe´ `""test">whoami´ -------------------- * 41) `test-dump.exe´ `""test"<whoami´ `test-dump.exe´ `""test"<whoami´ * 41) `test-dump.CMD´ `""test"<whoami´ `test-dump.exe´ `""test"<whoami´ -------------------- * 42) `test-dump.exe´ `test(whoami)´ `test-dump.exe´ `test(whoami)´ * 42) `test-dump.CMD´ `test(whoami)´ `test-dump.exe´ `test(whoami)´ -------------------- * 43) `test-dump.exe´ `test(whoami)´ `test-dump.exe´ `test(whoami)´ * 43) `test-dump.CMD´ `test(whoami)´ `test-dump.exe´ `test(whoami)´ -------------------- * 44) `test-dump.exe´ `test"(whoami)´ `test-dump.exe´ `test"(whoami)´ * 44) `test-dump.CMD´ `test"(whoami)´ `test-dump.exe´ `test"(whoami)´ -------------------- * 45) `test-dump.exe´ `test"(whoami)´ `test-dump.exe´ `test"(whoami)´ * 45) `test-dump.CMD´ `test"(whoami)´ `test-dump.exe´ `test"(whoami)´ -------------------- * 46) `test-dump.exe´ `test^whoami´ `test-dump.exe´ `test^whoami´ * 46) `test-dump.CMD´ `test^whoami´ -- *VULNERABLE* `test-dump.exe´ `testwhoami´ -------------------- * 47) `test-dump.exe´ `test^^echo ^^^´ `test-dump.exe´ `test^^echo ^^^´ * 47) `test-dump.CMD´ `test^^echo ^^^´ `test-dump.exe´ `test^^echo ^^^´ -------------------- * 48) `test-dump.exe´ `test"^whoami´ `test-dump.exe´ `test"^whoami´ * 48) `test-dump.CMD´ `test"^whoami´ `test-dump.exe´ `test"^whoami´ -------------------- * 49) `test-dump.exe´ `test"^^echo ^^^´ `test-dump.exe´ `test"^^echo ^^^´ * 49) `test-dump.CMD´ `test"^^echo ^^^´ -- *VULNERABLE* `test-dump.exe´ `test"echo ´

The major injury is happened by pre-processing of the arguments with special meta-chars (so depends on quotes-count or position), but some tests show that Tcl seems to have here additionally insufficient escaping by some special characters (at least looks like vulnerable arguments-handling, also by execution of some tests without unpaired quotes (after escape).

Fixed in branch 0-day-21b0629c81.

User Comments: oehhar added on 2023-11-30 19:19:26:

Branch [fix-fb2fa9b3f6--percent-subst-regr] was merged with commit [ee7ec33deb]. Only the "%env%" part of ticket [ea3b6d6792] is reverted. The requirement was ticket [fb2fa9b3f6].

Open work for a general multi-mode exec is pending.

Thanks to Sergey !

Harald


oehhar added on 2023-11-16 16:27:56:

Sergey, yes, I also think, we should only revert the special treatment of "%" by individual quoting.

The rest of the patch improves anyway already present quoting rules and fixes issues with them (double quoting).

Please just go on and merge it to 8.6.

Thank you and take care, Harald


sebres added on 2023-11-16 16:04:07:

I meant only revert the special %-escape (so basically merge branch "fix-fb2fa9b3f6--percent-subst-regr"). Because the rest of [21b0629c81] is fully justified in my opinion, as well as more or less backwards compatible (unless one doesn't try to simulate special "raw" invocation by inject into old escape handling in order to get some special command line).

But sure, I can do that, once I'd get the fossil to hand.

Regarding the documentation, one'd then only need to meant that %-chars are not (yet) specially escaped, so by execution of comspec/batch/cmd windows would replace %var% with environment variables, and this way to become safe it could expect special handling by parameter escape, because may become "vulnerable" against double/triple quotes in whole command line.


oehhar added on 2023-11-16 15:17:57:

Dear Sergey, if you are the opinion, that this should be reverted, it would be great to do that soon. I think, this would be a good thing, as the usability for me is higher, than the vulnerability protection.

Could you just act and do the changes?

I may assist and adapt the documentation.

Thank you for your constant effort, great !

Take care, Harald


sebres added on 2023-11-16 14:15:48:

Well, conditionally true... Even main of many script langs are implemented not very compliant to "default" parameters parsing - see the table in spoiler.

But there is CommandLineToArgvW as well as main of msvcrt, ucrt, libstdc (mingw/gcc) and co, that behave in the same way. Even reactos, wine and other "windows-like" systems and libraries implementing the command line parsing in that "standard" way.

Although the more or less correct handling of comspec (cmd/bat/etc) is hard job, and one can't really do that fully correctly - therefore there is a new branch fix-fb2fa9b3f6--percent-subst-regr, reverting parts of this fix, handling the percent char and so restore the backwards compatibility calling comspec with %var%. As already said many times the percent escape was implemented only because otherwise it'd remain a bit vulnerable (double and triple quotes may change during exec), but since comspec remains unsafe by design, one could indeed restore that handling back (using aforementioned branch).

More or less with a future -raw option, one'd be able to implement own escape routines for any kind of executable, no matter how it does the command line parsing.

This fix implements correct escape for CommandLineToArgvW as well as main of msvcrt, ucrt, libstdc (mingw/gcc) and co, so quasi "standard" command line parser. And without merge of fix-fb2fa9b3f6--percent-subst-regr branch also for comspec as good as possible (but as already said, I'm for merge and fix it later using -shell option or in some special handler if -raw option gets implemented).


dkf added on 2023-11-14 16:04:35:

Ultimately, there isn't that much we can do because Windows itself is not consistent. When dealing with the Windows command line, parsing of that command line is done by the receiving program (usually by it's runtime bootstrap code) and that's inconsistent because there are multiple different runtimes in use out there. It's especially bad when dealing with the CMD builtins like START, but that's simply the most easily noticed. (There were, according to David Gravereaux, who worked on this, outright incompatibilities between the MSVC runtime and the Borland runtime that just could never be resolved; each had situations where it demanded quoting and the other required no quoting.)

Tcl has historically taken the view of trying to make things work well with the MSVC runtime (by far the most common) and for the rest... well, you can write stuff into a batch script (with as much safe control as you like) and run that. It's an impossible situation; direct use of the Windows command line to run programs with untrusted arguments is simply not secure. Note that exec and open are never exposed directly in a safe interpreter; there's a reason for this and it's not something we're likely to change soon.

All IIRC. I've not made much use of this knowledge for over a decade (other than to always give START a leading blank argument because it's particularly awful).


oehhar added on 2023-11-10 11:12:02:

Documentation of the current state added in core-8-6-branch and up: [71dd06e857]

To control the quoting of exec is a purpose of a TIP which may come up soon.

Thanks, Sergey, for the help on the documentation, Harald


oehhar added on 2023-11-03 11:35:00:

Also, recent discussion on this topic may be found in ticket [fb2fa9b3f6].

Take care, Harald


oehhar added on 2023-11-03 11:30:32:

Thanks, great !

Proposed documentation for the introduced quoting in commit [7af4466e66] for review.

Review welcome ! Harald


sebres added on 2018-08-30 18:26:26:

So although now it is anyway better as it previously was, the story continues - the handling round about percent character is a bit complex:

% set ::env(X) "simple-X"; set ::env("X") "quoted-X"
% exec test-dump.cmd {%X%} {%"X"%} {"%X%"}
    `test-dump.exe´ `quoted-X´ `%"X"%´ `"quoted-X"´
So the escaped sequence "%"X"%" causes that this time not variable X but variable "X" will be interpolated by the command processor.

Thanks to Eryk Sun (@eryksun) noticed this by the fix for python.

So there are three possible scenarios:

  1. either let it as is currently, so don't change something (the environment variable in quotes is very rare, but anyway it is a bit ugly as "good" solution);
  2. or try to find better escape handling (I assume possible only if it will be escaped differently for invocation of exe and cmd).
  3. if the "proper" escape of %-char will get state as really impossible or very complex (so can be defined as not a bug but "expected" feature), so let it be, but at least it belongs definitely into the documentation with some bold warning.

So I'm by 2 at the moment, WiP.


sebres added on 2018-08-30 13:45:55:
closed within merge [99af12fd19] and the following, for >= 8.5

sebres added on 2018-08-21 19:04:39:

Grrr... amend in [ae46c72447] should fix it now (+ extends test-cases).

Further tests are welcome.


sebres added on 2018-08-21 16:24:32:

Never ending story: found new cases that are still uncovered by current fix:
- `%` char to be escaped (quoted) in any case (regardless pairing flag), otherwise `%username%` will be interpolated as username.
- escape of multiple backslashes before quote is different (as without following quote) in unpaired quote syntax (upaired flag set).
- etc.

WiP.


sebres added on 2018-08-20 18:08:01:

Although this fixed the issue, at the same time it is a bit incompatible if wanted "raw" (not escaped) arguments,
for example by invocation of cmd /c ...something raw, because script...

Search for exec in own (as well as many open source) projects, shows me almost never the usage of "raw" arguments as expected case. In contrary, many invocation's of cmd /c expecting also that arguments are escaped properly.
Let alone real raw data were anyway impossible (because previously partially escaped also).

Therefore (and because this fixed a grave 0-day vulnerability), the default escape should take place as it is implemented in 0-day-21b0629c81.

But perhaps, just to provide the developer a possibility to invoke cmd /c with raw-unescaped scripts (e. g. something enclosed in 2x double-quoted block, like cmd /c ""...raw-script..."" etc.), I suggest to extend "exec" with new option "-raw".

exec -raw cmd /c ""...raw-script-dev-want-escape-independent...""
For *unix-platforms this can be simply ignored, but I assume it will be anyway used with exec cmd only, so already encapsulated into if {$::tcl_platform(platform) eq "windows"} {exec -raw cmd ...}.


sebres added on 2018-08-20 16:20:31:

Fixed in branch 0-day-21b0629c81 (for 8.5).
I'll merge it soon (>= 8.5), if no objections follow.


sebres added on 2018-08-17 19:13:18:

Updated with many other test-cases (with better readability as single-line, etc.).
Additionally I noticed, that previously I had created the tests using already partially fixed tcl-library (so regenerated now).