This was discussed on the chat yesterday.  Errors in negotiation can be received 
by [tls::handshake], but a bgerror is still produced.  This short example 
demonstrates:

package require tls

tls::init -cadir /etc/ssl/certs -require true -ssl2 0 -ssl3 0 -tls1 0 -tls1.1 0

proc bgerror {err} {
    puts "bgerror: $err"
}

proc do_connect {} {
    set chan [tls::socket irc.choopa.net 9999]

    # ensure handshake completes
    try {
        while {![tls::handshake $chan]} {puts handshake}
    } on error {err opts} {
        puts "handshake error: $err"
        return false
    }

    return true
}

do_connect
puts "synchronous code done"
update

Output:

handshake error: handshake failed: certificate verify failed
synchronous code done
bgerror: SSL channel "sock556449741e30": error: certificate verify failed

Note that without the call to tls::handshake, the same error will be produced 
on first attempt to read or write the socket.

There are calls to Tcl_BackgroundError() in both Tls_Error() and
VerifyCallback().  The Tls_Error() invocations all come through stacked channel 
machinery, so it *may* be possible to instead pass the error back to Tcl.  Note 
the twapi ticket says this isn't an option there, but refers to reflected 
channels.  I'm not sure if transchans will propagate errors from callbacks
VerifyCallback() is called by openssl so probably not so simple.

If extra plumbing is needed, one suggestion was to use a hash table keyed by 
the channel ID to mark the presence of a synchronous operation that can return 
the error.  This may need mutex protection.

Another question when looking at the code:  it seems like tcltls requires/assumes 
that the tls channel is the top of a channel stack.  I'm not sure this is 
correct, since it might make sense to push another transform on top of tls.  It 
may be worth investigating this.