Tcl Source Code

View Ticket
Login
Bounty program for improvements to Tcl and certain Tcl packages.
Ticket UUID: 659131
Title: development targets: Complete pthreads?
Type: RFE Version: None
Submitter: elfring Created on: 2002-12-27 20:19:09
Subsystem: 80. Thread Package Assigned To: vasiljevic
Priority: 6 Severity:
Status: Open Last Modified: 2004-08-13 15:48:53
Resolution: None Closed By: vasiljevic
    Closed on: 2003-08-24 15:42:47
Description:
I want to continue our communication 
(https://sourceforge.net/tracker/?group_id=10894&atid=360894&func=detail&aid=577014) 
because the "Thread package 
2.5" is available. The package has 
been improved in severall ways.

1. It seems that some things 
that are offered by the pthreads library can still be done in the 
future.
- add missing lock types 
(http://www.humanfactor.com/pthreads/pthread-code-
snippets.html)

2. more "convenience" functions for thread 
shared variables
"string", "regexp" or "keyed list" from the TclX 
extension for example

3. Will a thread that was not preserved 
("thread::create" without option "-preserved") exit 
immediately?

4. A sub function "names" is available for some 
commands. I think that it would be needed for "thread::mutex" and 
"thread::cond", too.

5. When will it be possible to query the 
mutexes and condition variables by which threads they are 
used?

6. The description for "thread::vwait" is missing. 
(http://cvs.sourceforge.net/cgi-
bin/viewcvs.cgi/*checkout*/tcl/thread/doc/html/thread.html)

7. 
The function "tsv::lock" is available. How do you think about a similar 
function "thread::lock"?
User Comments: elfring added on 2004-08-13 15:48:53:
Logged In: YES 
user_id=572001

Is a file appendix useful for version 2.6?

elfring added on 2004-03-20 03:23:32:
Logged In: YES 
user_id=572001

Are there any chances that the files that I appended as a suggestion in November will be reviewed?

elfring added on 2003-12-24 04:42:19:
Logged In: YES 
user_id=572001

Can the following links give you more ideas for the TIP?

- The C10K problem
  http://www.kegel.com/c10k.html

- The Filament Runtime System
  http://www.cs.arizona.edu/people/filament/fil.html

elfring added on 2003-11-27 21:20:43:
Logged In: YES 
user_id=572001

1. Yes - I want to add a couple of things.

more introspection calls:
Tcl_GetThreadSpecificDataKeys(keyIdsPtr);
Tcl_GetMutexLockWaiters(mutexPtr, threadIdsPtr);
...

I am unsure if names can help with the housekeeping. I think that pointers are enough for debugging.


2. Check naming conventions
Tcl_MutexTryLock(mutexPtr);
   or
Tcl_TryLockMutex(mutexPtr);
   or
Tcl_MutexLockTry(mutexPtr);


3. Please answer some more questions from the previous communication.
I repeat the topic "spin locks" and "attributes" for example.

vasiljevic added on 2003-11-27 01:23:37:
Logged In: YES 
user_id=95086

As promised, here is the small list of things 
I'd like to TIP. 
 
Add new calls: 
 Tcl_SetThreadSpecificData(keyPtr, data); 
 Tcl_GetThreadSpecificData(keyPtr); 
 Tcl_CreateThreadSpecificData(keyPtr, cleanupCallback); 
 
The idea is to introduce better handling of thread 
specific data for cleanup purposes. The current Tcl 
implementation with simple exit callbacks shows some 
limitations. 
 
Add new calls: 
 Tcl_MutexTryLock(mutexPtr); 
 Tcl_SetMutexName(mutexPtr, mutexName); 
 Tcl_GetMutexName(mutexPtr); 
 Tcl_GetMutexLockAttempts(mutexPtr); 
 Tcl_GetMutexLockWaits(mutexPtr); 
 
The idea here is to extend the mutex structure 
to include some housekeeping counters. This 
would allow for monitoring locking hotspots. 
 
Add new calls: 
 Tcl_ConditionSignal(condPtr) 
 Tcl_ConditionWaitEx(condPtr, mutexPtr, timePtr) 
 
The first one is simply missing. The second one 
would return the cause of the exit (timer expired, 
or variable signalled) which is not reported by 
the current Tcl_ConditionWait() implementation. 
 
Add new calls: 
 Tcl_RWLockWriteLock(rwlockPtr); 
 Tcl_RWLockReadLock(rwlockPtr); 
 Tcl_RWLockUnlock(rwlockPtr); 
 
Not generally needed since could be implemented 
by existing primitives, but simplifies work for 
some people. This is a goodie only. 
For the Tcl extension, this can be done in the 
extension itself, so no core changes would be 
needed. 
 
 
Now, the reference implementation is pending. 
I hope to get some time in next couple of weeks 
and get it (finally) done. 
Is there anything you might add to this?

elfring added on 2003-11-20 18:08:28:
Logged In: YES 
user_id=572001

Thanks for this anwers.

Which Win32 functions are you missing in the POSIX API?
Shall we try to find more solutions together?

vasiljevic added on 2003-11-20 02:37:57:
Logged In: YES 
user_id=95086

OK. The reader/writer locks are really piece  
of cake. I do not generaly think those are good idea 
for many reasons (starvation of threads) but here/there 
they might be ok. 
Yes, thread pools are a great vehicle for building 
complex server suff things in Tcl alone. 
 
For Win threads: there are places, for example, where 
you can impersonate a thread for a specific user. 
This is crucial for MT-servers (fileservers for example) 
to enforce protection schemes. Posix does not allow 
for this. 
But I would not like to make this a two-person flame. 
I do not program on Win. I even dont like it. I just 
miss some things from Posix and I find them in Win api. 
 
For shared tcl objects I mean the shared Tcl_Obj. 
I went a long way to make it somehow "shared" in 
tsv, but this is only a poor aproximation, not really 
the "right thing". 
 
Numerous other plans? Well... do tsv monitors 
(aka traces) so people can build up simple notification 
system when a tsv variable changes. 
Then, add shared-memory persistence to tsv, so people can 
share data in across processes as they can do across 
threads. 
Then, abstract threads/processes under one clean simple 
API so you can create pools of processes or threads or 
combined (this can be a very powerful thingy) out of Tcl. 
 
Well, this is enough for the moment, right? :) 
If you have some more ideas in that direction 
(simple, easy to understand/use but powerful) 
please step-out with them. I do not think that 
implementing Tcl API for semaphores will make 
peoples life easier ;)

elfring added on 2003-11-20 01:36:22:
Logged In: YES 
user_id=572001

The RWL thing is an important goody. We've got different views for the pieces that should come next. Would like to publish anything about your "numerous other plans" to increase the user's patience?
The pools make your package very attractive.

There are a lot of people that would completely disagree with "superior" Windows threads. I think that every implementation like yours that offers condition variables is better than it.
Which parts do you like from the Win32 programming interface?

Do you mean data structures with "shared Tcl objects" that do not need to use the tsv interface?

How are the chances to get more points from 1. to 13. answered?

vasiljevic added on 2003-11-19 22:30:44:
Logged In: YES 
user_id=95086

I understand. The thing is that we already have a  
good API and we might improve on that by adding 
more goodies (reader-writer locks for example). 
But I doubt that by fully implementing whatever API 
we will attract more people. I think we will attract 
more people if things become simpler to use. 
Threads are already complicated enough, per-se. 
This is where I want to expand. Therefore, I did  
threadpools. Now I'm thinking abut simpler cross-thread 
interp syncing and tsv variable traces (more to come). 
I will add the reader-writer locks (yes, they are already 
in AOLserver) to the threading extension, plus support 
for lset and new dictionary datatype. This will all give 
you more tools in your toolbox. I have numerous other  
plans, but one after another :) 
 
As far as underlying OS-API is concerned, the Win 
threads are in some respect superior to Posix. 
Tcl has already using Win threads, so I see no benefit. 
The Tcl thread API isolates you from the OS one and  
it does this pretty good. It can be improved though 
which is, as I understand, the topic of our discussion. 
 
As for the Tcl object system, I'm thinking about  
possibilities to introduce locking structs in Tcl  
objects so they can be optionally shared across 
threads (which is not the case now). This is however 
far-reaching backwardly incompatible change which won't 
see the light of day before 9.0 release or such.

elfring added on 2003-11-19 22:09:22:
Logged In: YES 
user_id=572001

I just want that your package will be completed. I hope that it will become more attractive to other developers if nothing is missing.

The pthread interface that is used by the TCL library can be run on all systems that support it. The implementation "http://sources.redhat.com/pthreads-win32/" can be used for Windows platforms. Can this one "be better" than the TCL approach so far?

The AOLserver can be improved with the support for reader and writer locks.

Do you want to turn the synchronization primtivites into TCL objects?
Would you like to provide synchronized data structures in the TCL object system?
Or is this a similar thing like the command "thread::eval"?

vasiljevic added on 2003-11-19 14:31:21:
Logged In: YES 
user_id=95086

Thanks. Now we're getting somewhere... 
I did skip thru the distro and there, immediately, 
one *big* question pops-up which you might be able 
to answer. 
 
What it the purpose of your proposal? 
 
Currently, Tcl offers (incomplete) but practicaly 
very usable set of API calls to build just about 
any MT-app you can think of, by *using* Posix pthreads 
library on Unix. On Windows, it uses native Windows 
threads library. What can be better than that? 
 
There are some *dusty* corners about what Posix  
and/or Win API's offer which are of no great 
practical use (I never needed the spin mutex 
in my life, for example). Although I think we 
should eventually close these holes, the  
implementation is good as it is.  
I have compiled a wish-list some time ago, mostly 
based on AOLserver thread compatibility library 
about features missing in the Tcl API. This should 
be the base for the improvement which I wanted to  
TIP, but never got the time to make the reference 
implementation. I might send you this if I find it 
somewhere on my home computer. 
The idea was to junk the AOLserver thread lib in  
favour of Tcl API. One this has been done, Tcl  
would have the most complete, useful and thoroughly 
tested API not found in any other script environment.  
 
So, again, what *exactly* is what are you trying 
to achieve? I see that you put a lot of effort in 
this direction, and this is very encouraging.  
Yet, I'd like to have some more background info. 
 
Apart from that, there is one *huge* issue in Tcl 
design that we ought to pay much more attention. 
This is the Tcl object system and protected access 
to Tcl objects accross threads. This is the part 
that deserves more time than wrapping the very 
last bit of Posix and/or Win thread API.

elfring added on 2003-11-19 05:26:00:
File Added - 67927: tclPosixThread3.tar.gz

Logged In: YES 
user_id=572001

another file package...

14. Should be more macros used?
Examples:
- EXTERN
- VOID
- CONST
- _ANSI_ARGS_

vasiljevic added on 2003-11-18 03:49:02:
Logged In: YES 
user_id=95086

sorry but I can't unzip the uploaded file. 
Somehow it tells me I need zip version 2.0 
or something. I don't have this on my 
Suse 9.0 system. Can you pack the files with 
plain tar ?

elfring added on 2003-11-18 02:38:55:
File Added - 67784: tclPosixThread2.zip

Logged In: YES 
user_id=572001

7. I reorganized the source code to use inlined functions. (See also "https://sourceforge.net/tracker/?func=detail&aid=838147&group_id=10894&atid=360894".)
Is the switch "INLINE_SUPPORTED" okay? Or must it replaced by the condition "defined(HAVE_C_INLINE) || defined(__cplusplus)"?
I'm curious if you can agree with this suggestion.

8. I added the missing attributes and synchronization primitives.

9. I leave out the function "pthread_rwlock_timed(rd|wr)lock" at the moment because I assume that you want the time calculations like it is done in the function "Tcl_ConditionWait".

10. files "tclPosixAttr.h" and "tclPosixSync.h":
10.1 I am unsure if the "MASTER_LOCK" will be needed by the code.
10.2 Do the generated names fit to your guidelines?

11. file "generic/tclThread.c":
11.1 Are you going to add the functions "TclRememberSpin" and "TclRememberRWLock"?
Or can the function "RememberSyncObject" directly be called?
11.2 Will any more empty functions be added?
11.3 What will happen with the "stub table"?

12. Please answer some "TO DO" questions in the code.

13. How are the chances now to get anybody to create a TIP if this is still needed?

vasiljevic added on 2003-11-06 17:11:16:
Logged In: YES 
user_id=95086

Thank you for your efforts.  
I will look at the code you've posted. 
Upfront just a couple of anwers: 
 
TclRememberMutex is defined in generic/tclThread.c 
and is just a wrapper for RememberSyncObject. 
 
Self-initialization of thread sync primitives means 
there is no explicit creator function. The primitive 
is auto-initialized on the first use. However, you 
need to do Tcl_MutexFinalize and/or Tcl_ConditionFinalize 
when not needed any more. Take care: this can be tricky. 
 
Tcl_Mutex mutex; 
Tcl_MutexLock(&mutex); /* will create the mutex */ 
/*....*/ 
Tcl_MutexUnlock(&mutex); 
Tcl_MutexFinalize(&mutex); /* will destroy it */ 
 
Errors from system calls should be set using  
Tcl_SetErrno(errno).

elfring added on 2003-11-05 06:06:28:
File Added - 66487: tclPosixThread.zip

elfring added on 2003-11-05 06:06:27:
Logged In: YES 
user_id=572001

1. I reorganized the source code for the files "tclUnixThrd.h" and "tclUnixThrd.c" of TCL 8.4.4 to get a better understanding.
I refactored the code into separate files. I added a few functions. Can my approch help in our discussion?

2. Does it fit to the expected coding style?

3. How do you think that attributes should be added to the library?

4. I am looking for documentation of the function "TclRememberMutex". - How is the data allocated?

5. I am unsure about the statement "All of these synchronization objects are self initializing." (http://tcl.tk/man/tcl8.4/TclLib/Thread.htm).

6. Should error values be forwared to the variable "errno"?

vasiljevic added on 2003-08-26 22:33:26:
Logged In: YES 
user_id=95086

1. As far as I'm concerned, everybody is welcome to submit the 
whatever kind of TIP he/she likes. It is open source development, 
so anybody is more than welcome to contribute.  
 
2. I do not have any work done in that direction yet. I wanted to get 
Tcl thread API to the point that we can shred the AOLserver's 
custom pthread wrapper, but I'm very short on time lately. 
 
3. Everybody would benefit from some marketing :)

elfring added on 2003-08-25 17:58:50:
Logged In: YES 
user_id=572001

1. Can we start with an informational TIP?
   It will be transformed to a 
project TIP later.

2. Have you got wrapper templates or 
classes?
   Can an example show how easy the missing functions can 
be added?

3. Would a little "advertisment" help to get more 
developers to copy and write the code?

vasiljevic added on 2003-08-24 22:42:47:
Logged In: YES 
user_id=95086

I know. Chances of getting the TIP accepted are much  
higher if there is a ref implementation sitting somewhere. 
In this case, I have very little concern about adoption  
of the TIP. It is the implementation which troubles me. 
Not because of the complexity (it is just adding new  
pthreads C-wrappers here and there) but because of  
the lack of time. 
So, lets see if I'll be able to manage that :)

dkf added on 2003-08-24 22:28:38:
Logged In: YES 
user_id=79902

Not strictly true, and certainly not to submit it (getting a
vote to accept is slightly different; we've had problems in
the past with TIPs where the author hasn't been able to
arrange implementation effort.) TIPs do require a commitment
of time and effort from *someone* though, as our engineering
standards are rather high.

vasiljevic added on 2003-08-23 21:56:19:
Logged In: YES 
user_id=95086

To submit the TIP one needs to have a reference implementation, 
otherwise nothing will happen. 
I had very little time to catch up with that so it just keeps  
sleeping. Hopefully, times will get better.

elfring added on 2003-08-23 20:31:02:
Logged In: YES 
user_id=572001

When will this Tcl Improvement Proposal (http://www.tcl.tk/cgi-
bin/tct/tip/2.html) be started?

elfring added on 2003-08-16 16:43:52:
Logged In: YES 
user_id=572001

It needs some time to get the function interface 
"http://www.tcl.tk/man/tcl8.4/TclLib/Thread.htm" up to date ...

vasiljevic added on 2003-05-30 23:44:35:
Logged In: YES 
user_id=95086

We'll soon make a TIP in order to get more of pthreads available 
on the C-API so everybody can take advantage of that.

elfring added on 2003-01-13 01:00:58:
Logged In: YES 
user_id=572001

Normative 
documentation:
http://www.opengroup.org/onlinepubs/007904975/basedefs/pthread.h.html

elfring added on 2003-01-13 00:12:18:
Logged In: YES 
user_id=572001

Can anything from the class library "Jsync - collection of synchronization 
classes for Java" (http://www.garret.ru/~knizhnik/jsync-1.03/package-
jsync.html) be used for the TIP?

elfring added on 2003-01-08 18:08:25:
Logged In: YES 
user_id=572001

Are the comments from the discussion "library proposal" 
(http://groups.google.com/groups?selm=IhBS9.18%
24LJ1.422974%40news.cpqcorp.net) also applicable to the 
threads package?

elfring added on 2003-01-05 17:47:52:
Logged In: YES 
user_id=572001

Is cooperative multitasking (like "http://www.gnu.org/software/pth/pth-
manual.html#the compromise of pth") or preemptive multithreading 
provided by the internal TCL threads library?
Does this library support 
simultanous execution on several processors?

elfring added on 2003-01-04 19:33:34:
Logged In: YES 
user_id=572001

I point to another good description:
Guide to the POSIX Threads Library - 
http://www.tru64unix.compaq.com/docs/base_doc/DOCUMENTATION/V51_HTML/ARH9RBTE/TITLE.HTM

newsgroup: 
comp.programming.threads

elfring added on 2002-12-31 04:14:34:
Logged In: YES 
user_id=572001

1. I add some links to helpful information about algorithms and 
technologies.
1.1 Can the API be improved so that an implementation of 
the CORBA standard "Concurrency Service" 
(http://www.omg.org/technology/documents/corbaservices_spec_catalog.htm, 
http://fpx.de/Combat/) would be possible with TCL?
1.2 Can the 
package profit from "Communicating Sequential Processes" 
(http://www.cs.ukc.ac.uk/research/groups/crg/research_interests.html)

elfring added on 2002-12-29 05:40:56:
Logged In: YES 
user_id=572001

1. Interesting...
You mentioned "pthreads". - So I made suggestions in 
this direction.
I did not know that the programming interface should 
improve on the C level before further enhancements can be made. I would 
prefer that all functionality of the underlying thread function or class library 
will be also provided on the TCL level.
The functions that you are not 
convinced to exist because they are needed. The frequency for their usage 
patterns is different, of course.

5. I'm curious about that. Please put 
the ideas into the TIP.

6. I would prefer an other wording than "vwait". 
Would a "waiting on a thread variable" be easier to understand?

vasiljevic added on 2002-12-29 01:29:12:
Logged In: YES 
user_id=95086

1. Basicaly, the threading extension exposes the Tcl API.  
It does *not* add to the API.  
Speaking of the API, there are some API corners that need  
to be improved and I'm working on that. There will be a  
new TIP on this soon.  
Yes, all the stuff you've described are ok, but do you  
really need them on the script level? I think I might 
wrap the rwlocks, although I'm not really convinced that 
they are a good thing to use.  
 
3. Sure. 
 
5. You'd need to make your own wrappers arround  
pthread_mutex and friends and provide a kind of 
debugging mutex implementation where you track  
who (which thread) is holding the mutex and be  
able to kill the process on deadlock attempts. 
I might be proposing this in the upcomming TIP. 
The pthread does not do this for you, but you 
can make one yourself. 
 
6. Nowhere. It is an idiom one uses frequently  
when entering the event loop forever, like in 
some server applications, for example. 
 
8. Message passing == script passing. It is basically 
the same thing. 
 
10. Ok, the doc is little bit terse on that. I might  
get put it more clear.

elfring added on 2002-12-29 01:06:31:
Logged In: YES 
user_id=572001

9. It is good that the TCL behaviour for the function "exec" is different from that 
what can be read under "http://docs.sun.com/db/doc/806-
6867/6jfpgdcn4?a=view".

elfring added on 2002-12-29 00:39:24:
Logged In: YES 
user_id=572001

1.8 attr_... - http://docs.sun.com/db/doc/806-6867/6jfpgdcmi?a=view

elfring added on 2002-12-29 00:25:27:
Logged In: YES 
user_id=572001

1. I know that all things that are not provided by the package yet can be done by 
own additional programming.
But I read the available documentation in 
this way that "semaphores" do more (They can be shared between threads 
and processes.) and are a synchronization technique with other uses than 
mutex and condition variables alone. The following items are also part of the 
POSIX standard 1003.1c and may be offered in a future version of the thread 
package.
It is good that the thread pool functions were implemented 
without locking.
1.1 mutexattr... - http://docs.sun.com/db/doc/806-
6867/6jfpgdcmn?a=view
1.2 mutex_consistent_np, mutex_trylock - 
http://docs.sun.com/db/doc/806-6867/6jfpgdcmo?a=view
1.3 
condattr... - http://docs.sun.com/db/doc/806-
6867/6jfpgdcmp?a=view
1.4 cond_wait (I assume that the function 
"pthread_cond_timedwait" is used by the package.), 
cond_reltimedwait_np - http://docs.sun.com/db/doc/806-
6867/6jfpgdcmq?a=view
1.5 sem_... - 
http://docs.sun.com/db/doc/806-6867/6jfpgdcmr?a=view
1.6 
rwlockattr... - http://docs.sun.com/db/doc/806-
6867/6jfpgdcms?a=view
1.7 rwlock... - 
http://docs.sun.com/db/doc/806-6867/6jfpgdcmt?a=view

2. 
great

3. Will you add this explanation to the 
documentation?

5. Do you know a function library that will help to get 
that information?

6. In which package is the call "vwait 
forever"?
(You called it "this little bastard" once.)

7. Okay - 
Evaluation in a locked mode...

8. What you you mean with "message 
passing"?

9. Well, then your implemenation does not need any 
special treatment in this case like other threading libraries.

10. Your 
answer...?


Thanks

vasiljevic added on 2002-12-28 21:06:47:
Logged In: YES 
user_id=95086

1.  
Semaphores are ok. So are critical sections and recursive 
locks. But, what I want to achieve is *simplicity*. Thread 
programming has become pretty simple thanks to the Tcl 
compartment thread model. If you look at the tpool.tcl source 
you won't find a single condition variable or a mutex. Yet, 
relatively complex implementation of threadpool is done with 
Tcl alone and it is only about 20% slower than the C-level 
implementation. Furthermore, the Tcl implementation costed 
about 2 days development. The C-one took 10 days. So, 
economically speaking, the Tcl version is the clear winner. 
 
Needles to say that a semaphore can be done in pure Tcl 
with a mutex and a condition variable, which the threading 
extension already supplies.  
 
2. I will include the keyed-list datastructure in 2.6. 
 
3. It will just have the refcount of 0 meaning that caller of 
such "non-preserved" thread may never know wether the 
thread is still there for the next operation. But, it will not  
exit w/o being told so with a "thread::release" 
 
4. I will shortly revamp the cond/mutex Tcl API support. Watch 
for the new TIP. This way the API will be more powerful and  
you can get many new info/statistics about the mutex etc. 
 
5. Inherently impossible by pthread alone. 
 
6. There is no such command in the thread package. There 
is however thread::wait and it is described. 
 
7. There is already thread::eval which evaluates a script under 
the mutex, either implicit or explicit one. 
   
8. I think that the message passing, together with channel 
transfer and detach/attach allows one to implement almost 
any task.  
 
9. As a script programmer, you can safely do the "exec"  
from the MT Tcl shell. The underlying implementation will 
do the right thing.

elfring added on 2002-12-28 20:57:38:
Logged In: YES 
user_id=572001

10. Can the explanation for "spurious thread wake-ups" with the function 
"thread::cond wait" become more comprehensible?
(Interrupted 
Waits on Condition Variables - http://docs.sun.com/db/doc/806-
6867/6jfpgdcn9?a=view)

elfring added on 2002-12-28 20:33:42:
Logged In: YES 
user_id=572001

9. Is there anything to be careful about forking a new 
process?
(http://www.gnu.org/software/pth/pth-
manual.html#process forking)

elfring added on 2002-12-28 20:13:25:
Logged In: YES 
user_id=572001

8. Is the technique "sending TCL scripts" the only inter-thread 
communication mechanism with the thread package?
Will "message 
ports" (http://www.gnu.org/software/pth/pth-manual.html#message port 
communication) be applicable?

elfring added on 2002-12-28 04:10:32:
Logged In: YES 
user_id=572001

1. Another POSIX item
- semaphores (Comparing Primitives - 
http://docs.sun.com/db/doc/806-
6867/6jfpgdcn0?l=de&a=view)
Will this be offered, too?

Attachments: