Check-in [691293448c]

{"tip": {
	"550":{"url":"./tip/550.md","keywords":"Tcl, memory","created":"11-Jun-2019","post-history":"","state":"Draft","tcl-version":"9.0","vote":"Pending","type":"Project","title":"# TIP 550: Garbage Collection for TclOO","author":["Donal K. Fellows <[email protected]>"],"is-jest":false},
	"549":{"url":"./tip/549.md","keywords":"Tcl","created":"7-June-2019","post-history":"","state":"Draft","tcl-branch":"tip-549","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 549: Make configure --enable-64bit the default","discussions-to":"Tcl Core list","author":["Jan Nijtmans <[email protected]>","Jan Nijtmans <[email protected]>"],"is-jest":false},
	"548":{"url":"./tip/548.md","keywords":"Tcl","created":"3-June-2019","post-history":"","state":"Draft","tcl-branch":"tip-548","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 548:Support `wchar_t` conversion functions and deprecate `Tcl_WinUtfToTChar()` and `Tcl_WinTCharToUtf()`","discussions-to":"Tcl Core list","author":["Jan Nijtmans <[email protected]>","Jan Nijtmans <[email protected]>"],"is-jest":false},
	"547":{"url":"./tip/547.md","keywords":"Tcl","created":"31-May-2019","post-history":"","state":"Final","tcl-branch":"tip-547","tcl-version":"8.7","vote":"Done","type":"Project","title":"# TIP 547: New encodings: UTF-16, UCS-2","discussions-to":"Tcl Core list","author":["Jan Nijtmans <[email protected]>","Jan Nijtmans <[email protected]>"],"is-jest":false},
	"546":{"url":"./tip/546.md","keywords":"Tcl, traces","created":"20-May-2019","post-history":"","state":"Draft","vote":"Pending","tcl-version":"8.7","type":"Project","title":"# TIP 546: Typedefs to Support Source-Compatible Migration","author":["Don Porter <[email protected]>"],"is-jest":false},
	"545":{"url":"./tip/545.md","keywords":"Tk, image","created":"19-May-2019","state":"Draft","tk-branch":"tip-545-svg-options","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 545: image photo SVG options: remove -unit and add target height/width option","author":["Harald Oehlmann <[email protected]>"],"is-jest":false},
	"544":{"url":"./tip/544.md","keywords":"Tcl","created":"15-May-2019","post-history":"","state":"Final","tcl-branch":"tip-544","vote":"Done","tcl-version":"8.7","type":"Project","title":"# TIP 544: Export TclGetIntForIndex()","discussions-to":"Tcl Core list","author":["Jan Nijtmans <[email protected]>","François Vogel <[email protected]>"],"is-jest":false},
	"543":{"url":"./tip/543.md","keywords":"Tcl, traces","created":"10-May-2019","post-history":"","state":"Draft","vote":"Pending","tcl-version":"9.0","type":"Project","title":"# TIP 543: Eliminate `TCL_INTERP_DESTROYED` flag value","author":["Don Porter <[email protected]>"],"is-jest":false},
	"542":{"url":"./tip/542.md","keywords":"Tcl","created":"10-May-2019","post-history":"","state":"Draft","tcl-branch":"utf-max","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 542:Support for switchable (Androwish-compatible) Full Unicode support","discussions-to":"Tcl Core list","author":["Jan Nijtmans <[email protected]>","Jan Nijtmans <[email protected]>"],"is-jest":false},
	"541":{"url":"./tip/541.md","keywords":"Tcl","created":"8-May-2019","post-history":"","state":"Final","tcl-branch":"bug-2858503fff","tcl-version":"8.6","vote":"Done","type":"Project","title":"# TIP 541: Add support for 'end' index in ttk::combobox current","author":["Francois Vogel <[email protected]>"],"is-jest":false},

	"4":{"url":"./tip/4.md","created":"26-Oct-2000","post-history":"","state":"Draft","vote":"Pending","type":"Informative","title":"# TIP 4: Tcl Release and Distribution Philosophy","discussions-to":"news:comp.lang.tcl","author":["Brent Welch <[email protected]>","Donal K. Fellows <[email protected]>","Larry W. Virden <[email protected]>","Larry W. Virden <[email protected]>"],"is-jest":false},
	"3":{"url":"./tip/3.md","created":"14-Sep-2000","obsoleted-by":"467","post-history":"","state":"Accepted","vote":"Done","type":"Process","title":"# TIP 3: TIP Format","author":["Andreas Kupries <[email protected]>","Donal K. Fellows <[email protected]>"],"is-jest":false},
	"2":{"url":"./tip/2.md","created":"12-Sep-2000","post-history":"","state":"Draft","vote":"Pending","type":"Process","title":"# TIP 2: TIP Guidelines","author":["Andreas Kupries <[email protected]>","Donal K. Fellows <[email protected]>","Don Porter <[email protected]>","Mo DeJong <[email protected]>","Larry W. Virden <[email protected]>","Kevin Kenny <[email protected]>"],"is-jest":false},
	"1":{"url":"./tip/1.md","created":"14-Sep-2000","obsoleted-by":"386","post-history":"","state":"Withdrawn","vote":"No voting","type":"Informational","title":"# TIP 1: TIP Index","author":["TIP Editor <[email protected]>"],"is-jest":false},
	"0":{"url":"./tip/0.md","created":"11-Dec-2000","post-history":"","state":"Final","vote":"Done","type":"Process","title":"# TIP 0: Tcl Core Team Basic Rules","author":["John Ousterhout <[email protected]>"],"is-jest":false},
	"@min": 0,
	"@max": 550
}, "@timestamp": 1564866578}

<td valign='top'><a href='/tcl/timeline?r=tip-549'>Link</a></td>
</tr>
<tr class='state-draft type-project version-87'>
<td valign='top'><a href='./tip/548.md'>548</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.7</td>
<td valign='top'>Draft</td>
<td valign='top' ><a href='./tip/548.md'>Support `wchar_t` conversion functions and deprecate `Tcl_WinUtfToTChar()` and `Tcl_WinTCharToUtf()`</a></td>
<td valign='top'><a href='/tcl/timeline?r=tip-548'>Link</a></td>
</tr>
<tr class='state-final type-project version-87'>
<td valign='top'><a href='./tip/547.md'>547</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.7</td>
<td valign='top'>Final</td>

# TIP 548:Support `wchar_t` conversion functions and deprecate `Tcl_WinUtfToTChar()` and `Tcl_WinTCharToUtf()`
	Author:         Jan Nijtmans <[email protected]>
	Author:         Jan Nijtmans <[email protected]>
	State:          Draft
	Type:           Project
	Vote:           Pending
	Created:        3-June-2019
	Post-History:   
	Discussions-To: Tcl Core list
	Keywords:       Tcl
	Tcl-Version:    8.7
	Tcl-Branch:     tip-548
-----

# Abstract

This TIP proposes to add `wchar_t` conversion functions and deprecate `Tcl_WinUtfToTChar()` and `Tcl_WinTCharToUtf()`
in favour of those new functions.

# Rationale

The functions `Tcl_WinUtfToTChar()` and `Tcl_WinTCharToUtf()` originally were functions able to do two different
conversions, depending on the runtime platform: On Windows 95/98/ME they performed conversions between UTF-8 and
the Windows default encoding (usually CP1252), on later Windows versions they convert between UTF-8 and UTF-16.
The length parameter of `Tcl_WinTCharToUtf()` always was in bytes, but most other Unicode-related Tcl functions
expect their length in Unicode characters.

Since Windows 95/98/ME are not supported any more, it's time to fix this inconsistency.

Modern systems have a `wchar_t` type which represents a Unicode-like type, which can either be 16 bits
(unsigned short) or 32 bits (int). This TIP proposes 3 additional functions which convert between
`wchar_t`-related types and UTF-8, and 3 more which convert between `unsigned short`-related types (UTF-16)
and UTF-8. The new functions work identically on all platforms, not only Windows.

# Specification

This document proposes:










 * Enhance the `Tcl_UniCharToUtfDString()` function such that the uniLength parameter is allowed to
   have the value -1. In that case, the UniChar string will be read up to the closing /u0000 character.


 * Enhance the `Tcl_UniCharToUtfDString()` and `Tcl_UtfToUniCharDString()` functions such that the src/uniStr

   parameters are allowed to have the value NULL. In that case, the functions return NULL, without doing anything.











 * New functions:

     `Tcl_UtfToWCharDString()`, similar to `Tcl_UtfToUniCharDString()`, but has a `wchar_t` pointer type in its signature.

     `Tcl_WCharToUtfDString()`, similar to `Tcl_UniCharToUtfDString()`, but has a `wchar_t` pointer type in its signature

     `Tcl_UtfToWChar()`, similar to `Tcl_UtfToUniChar()`, but has a `wchar_t` pointer in its signature.
     
     On Windows, `wchar_t` is the same type as `unsigned short`, but on other platforms `wchar_t` might be a 32-bit type ('int' usually).
     These functions map to either the 32-bit or the 16-bit versions, depending on the size of `wchar_t`, automatically.

     `Tcl_UtfToChar16DString()`, similar to `Tcl_UtfToUniCharDString()`, but has an `unsigned short` pointer type in its signature.

     `Tcl_Char16ToUtfDString()`, similar to `Tcl_UniCharToUtfDString()`, but has an `unsigned short` pointer type in its signature

     `Tcl_UtfToChar16()`, similar to `Tcl_UtfToUniChar()`, but has an `unsigned short` pointer in its signature.

 * Deprecate the following functions:

     `Tcl_WinUtfToTChar()`, in favour of `Tcl_UtfToWCharDString()`

     `Tcl_WinTCharToUtf()`, in favour of `Tcl_WCharToUtfDString()`

   If Tcl is compiled with either -DTCL\_UTF\_MAX=6 (which is not officially supported) or -DTCL\_NO\_DEPRECATED, those functions will
   become macro's, which do exactly the same thing. In Tcl 9.0, the 2 deprecated functions will be removed from the stub tables,
   but the replacement macro's will still be there. So, the functions can still be used in extensions, they will be replaced
   with the new functions automatically.

 * The windows dde and registry extensions (tclWinDde.c and tclWinReg.c) are updated to use the new functions `Tcl_UtfToWCharDString()`
   and `Tcl_WCharToUtfDString()`, serving as proof/demonstration that the functions in this TIP actually work.

# How to upgrade.

No need to do anything. In Tcl 9.0, the two deprecated functions are replaced by macro's which do the same thing.
But if you want to prevent a (future) deprecation warning, you can do the following:

In your extension, replace the call:

     `Tcl_WinUtfToTChar(....., bufPtr)`;
     
with the following two lines:

     `Tcl_DStringInit(bufPtr);`
     `Tcl_UtfToWCharDString(....., bufPtr);`

And also, replace:

     `Tcl_WinTCharToUtf(....., bufPtr);`
     
with the following two lines:

     `Tcl_DStringInit(bufPtr);`
     `Tcl_WCharToUtfDString(....., bufPtr);`

If the `Tcl_WinTCharToUtf()` call originally had a "length" parameter not equal to -1, divide it by 2 (or ... don't multiply it by 2 any more).


# Compatibility

This is fully upwards compatible.