|Title:||development targets: Complete pthreads?|
|Submitter:||elfring||Created on:||2002-12-27 20:19:09|
|Subsystem:||80. Thread Package||Assigned To:||vasiljevic|
|Status:||Open||Last Modified:||2004-08-13 15:48:53|
|Closed on:||2003-08-24 15:42:47|
I want to continue our communication (https://sourceforge.net/tracker/?group_id=10894&atid=360894&func=detail&aid=577014) because the "Thread package 2.5" is available. The package has been improved in severall ways. 1. It seems that some things that are offered by the pthreads library can still be done in the future. - add missing lock types (http://www.humanfactor.com/pthreads/pthread-code- snippets.html) 2. more "convenience" functions for thread shared variables "string", "regexp" or "keyed list" from the TclX extension for example 3. Will a thread that was not preserved ("thread::create" without option "-preserved") exit immediately? 4. A sub function "names" is available for some commands. I think that it would be needed for "thread::mutex" and "thread::cond", too. 5. When will it be possible to query the mutexes and condition variables by which threads they are used? 6. The description for "thread::vwait" is missing. (http://cvs.sourceforge.net/cgi- bin/viewcvs.cgi/*checkout*/tcl/thread/doc/html/thread.html) 7. The function "tsv::lock" is available. How do you think about a similar function "thread::lock"?
elfring added on 2004-08-13 15:48:53:
Logged In: YES user_id=572001 Is a file appendix useful for version 2.6?
elfring added on 2004-03-20 03:23:32:
Logged In: YES user_id=572001 Are there any chances that the files that I appended as a suggestion in November will be reviewed?
elfring added on 2003-12-24 04:42:19:
Logged In: YES user_id=572001 Can the following links give you more ideas for the TIP? - The C10K problem http://www.kegel.com/c10k.html - The Filament Runtime System http://www.cs.arizona.edu/people/filament/fil.html
elfring added on 2003-11-27 21:20:43:
Logged In: YES user_id=572001 1. Yes - I want to add a couple of things. more introspection calls: Tcl_GetThreadSpecificDataKeys(keyIdsPtr); Tcl_GetMutexLockWaiters(mutexPtr, threadIdsPtr); ... I am unsure if names can help with the housekeeping. I think that pointers are enough for debugging. 2. Check naming conventions Tcl_MutexTryLock(mutexPtr); or Tcl_TryLockMutex(mutexPtr); or Tcl_MutexLockTry(mutexPtr); 3. Please answer some more questions from the previous communication. I repeat the topic "spin locks" and "attributes" for example.
vasiljevic added on 2003-11-27 01:23:37:
Logged In: YES user_id=95086 As promised, here is the small list of things I'd like to TIP. Add new calls: Tcl_SetThreadSpecificData(keyPtr, data); Tcl_GetThreadSpecificData(keyPtr); Tcl_CreateThreadSpecificData(keyPtr, cleanupCallback); The idea is to introduce better handling of thread specific data for cleanup purposes. The current Tcl implementation with simple exit callbacks shows some limitations. Add new calls: Tcl_MutexTryLock(mutexPtr); Tcl_SetMutexName(mutexPtr, mutexName); Tcl_GetMutexName(mutexPtr); Tcl_GetMutexLockAttempts(mutexPtr); Tcl_GetMutexLockWaits(mutexPtr); The idea here is to extend the mutex structure to include some housekeeping counters. This would allow for monitoring locking hotspots. Add new calls: Tcl_ConditionSignal(condPtr) Tcl_ConditionWaitEx(condPtr, mutexPtr, timePtr) The first one is simply missing. The second one would return the cause of the exit (timer expired, or variable signalled) which is not reported by the current Tcl_ConditionWait() implementation. Add new calls: Tcl_RWLockWriteLock(rwlockPtr); Tcl_RWLockReadLock(rwlockPtr); Tcl_RWLockUnlock(rwlockPtr); Not generally needed since could be implemented by existing primitives, but simplifies work for some people. This is a goodie only. For the Tcl extension, this can be done in the extension itself, so no core changes would be needed. Now, the reference implementation is pending. I hope to get some time in next couple of weeks and get it (finally) done. Is there anything you might add to this?
elfring added on 2003-11-20 18:08:28:
Logged In: YES user_id=572001 Thanks for this anwers. Which Win32 functions are you missing in the POSIX API? Shall we try to find more solutions together?
vasiljevic added on 2003-11-20 02:37:57:
Logged In: YES user_id=95086 OK. The reader/writer locks are really piece of cake. I do not generaly think those are good idea for many reasons (starvation of threads) but here/there they might be ok. Yes, thread pools are a great vehicle for building complex server suff things in Tcl alone. For Win threads: there are places, for example, where you can impersonate a thread for a specific user. This is crucial for MT-servers (fileservers for example) to enforce protection schemes. Posix does not allow for this. But I would not like to make this a two-person flame. I do not program on Win. I even dont like it. I just miss some things from Posix and I find them in Win api. For shared tcl objects I mean the shared Tcl_Obj. I went a long way to make it somehow "shared" in tsv, but this is only a poor aproximation, not really the "right thing". Numerous other plans? Well... do tsv monitors (aka traces) so people can build up simple notification system when a tsv variable changes. Then, add shared-memory persistence to tsv, so people can share data in across processes as they can do across threads. Then, abstract threads/processes under one clean simple API so you can create pools of processes or threads or combined (this can be a very powerful thingy) out of Tcl. Well, this is enough for the moment, right? :) If you have some more ideas in that direction (simple, easy to understand/use but powerful) please step-out with them. I do not think that implementing Tcl API for semaphores will make peoples life easier ;)
elfring added on 2003-11-20 01:36:22:
Logged In: YES user_id=572001 The RWL thing is an important goody. We've got different views for the pieces that should come next. Would like to publish anything about your "numerous other plans" to increase the user's patience? The pools make your package very attractive. There are a lot of people that would completely disagree with "superior" Windows threads. I think that every implementation like yours that offers condition variables is better than it. Which parts do you like from the Win32 programming interface? Do you mean data structures with "shared Tcl objects" that do not need to use the tsv interface? How are the chances to get more points from 1. to 13. answered?
vasiljevic added on 2003-11-19 22:30:44:
Logged In: YES user_id=95086 I understand. The thing is that we already have a good API and we might improve on that by adding more goodies (reader-writer locks for example). But I doubt that by fully implementing whatever API we will attract more people. I think we will attract more people if things become simpler to use. Threads are already complicated enough, per-se. This is where I want to expand. Therefore, I did threadpools. Now I'm thinking abut simpler cross-thread interp syncing and tsv variable traces (more to come). I will add the reader-writer locks (yes, they are already in AOLserver) to the threading extension, plus support for lset and new dictionary datatype. This will all give you more tools in your toolbox. I have numerous other plans, but one after another :) As far as underlying OS-API is concerned, the Win threads are in some respect superior to Posix. Tcl has already using Win threads, so I see no benefit. The Tcl thread API isolates you from the OS one and it does this pretty good. It can be improved though which is, as I understand, the topic of our discussion. As for the Tcl object system, I'm thinking about possibilities to introduce locking structs in Tcl objects so they can be optionally shared across threads (which is not the case now). This is however far-reaching backwardly incompatible change which won't see the light of day before 9.0 release or such.
elfring added on 2003-11-19 22:09:22:
Logged In: YES user_id=572001 I just want that your package will be completed. I hope that it will become more attractive to other developers if nothing is missing. The pthread interface that is used by the TCL library can be run on all systems that support it. The implementation "http://sources.redhat.com/pthreads-win32/" can be used for Windows platforms. Can this one "be better" than the TCL approach so far? The AOLserver can be improved with the support for reader and writer locks. Do you want to turn the synchronization primtivites into TCL objects? Would you like to provide synchronized data structures in the TCL object system? Or is this a similar thing like the command "thread::eval"?
vasiljevic added on 2003-11-19 14:31:21:
Logged In: YES user_id=95086 Thanks. Now we're getting somewhere... I did skip thru the distro and there, immediately, one *big* question pops-up which you might be able to answer. What it the purpose of your proposal? Currently, Tcl offers (incomplete) but practicaly very usable set of API calls to build just about any MT-app you can think of, by *using* Posix pthreads library on Unix. On Windows, it uses native Windows threads library. What can be better than that? There are some *dusty* corners about what Posix and/or Win API's offer which are of no great practical use (I never needed the spin mutex in my life, for example). Although I think we should eventually close these holes, the implementation is good as it is. I have compiled a wish-list some time ago, mostly based on AOLserver thread compatibility library about features missing in the Tcl API. This should be the base for the improvement which I wanted to TIP, but never got the time to make the reference implementation. I might send you this if I find it somewhere on my home computer. The idea was to junk the AOLserver thread lib in favour of Tcl API. One this has been done, Tcl would have the most complete, useful and thoroughly tested API not found in any other script environment. So, again, what *exactly* is what are you trying to achieve? I see that you put a lot of effort in this direction, and this is very encouraging. Yet, I'd like to have some more background info. Apart from that, there is one *huge* issue in Tcl design that we ought to pay much more attention. This is the Tcl object system and protected access to Tcl objects accross threads. This is the part that deserves more time than wrapping the very last bit of Posix and/or Win thread API.
elfring added on 2003-11-19 05:26:00:
File Added - 67927: tclPosixThread3.tar.gz Logged In: YES user_id=572001 another file package... 14. Should be more macros used? Examples: - EXTERN - VOID - CONST - _ANSI_ARGS_
vasiljevic added on 2003-11-18 03:49:02:
Logged In: YES user_id=95086 sorry but I can't unzip the uploaded file. Somehow it tells me I need zip version 2.0 or something. I don't have this on my Suse 9.0 system. Can you pack the files with plain tar ?
elfring added on 2003-11-18 02:38:55:
File Added - 67784: tclPosixThread2.zip Logged In: YES user_id=572001 7. I reorganized the source code to use inlined functions. (See also "https://sourceforge.net/tracker/?func=detail&aid=838147&group_id=10894&atid=360894".) Is the switch "INLINE_SUPPORTED" okay? Or must it replaced by the condition "defined(HAVE_C_INLINE) || defined(__cplusplus)"? I'm curious if you can agree with this suggestion. 8. I added the missing attributes and synchronization primitives. 9. I leave out the function "pthread_rwlock_timed(rd|wr)lock" at the moment because I assume that you want the time calculations like it is done in the function "Tcl_ConditionWait". 10. files "tclPosixAttr.h" and "tclPosixSync.h": 10.1 I am unsure if the "MASTER_LOCK" will be needed by the code. 10.2 Do the generated names fit to your guidelines? 11. file "generic/tclThread.c": 11.1 Are you going to add the functions "TclRememberSpin" and "TclRememberRWLock"? Or can the function "RememberSyncObject" directly be called? 11.2 Will any more empty functions be added? 11.3 What will happen with the "stub table"? 12. Please answer some "TO DO" questions in the code. 13. How are the chances now to get anybody to create a TIP if this is still needed?
vasiljevic added on 2003-11-06 17:11:16:
Logged In: YES user_id=95086 Thank you for your efforts. I will look at the code you've posted. Upfront just a couple of anwers: TclRememberMutex is defined in generic/tclThread.c and is just a wrapper for RememberSyncObject. Self-initialization of thread sync primitives means there is no explicit creator function. The primitive is auto-initialized on the first use. However, you need to do Tcl_MutexFinalize and/or Tcl_ConditionFinalize when not needed any more. Take care: this can be tricky. Tcl_Mutex mutex; Tcl_MutexLock(&mutex); /* will create the mutex */ /*....*/ Tcl_MutexUnlock(&mutex); Tcl_MutexFinalize(&mutex); /* will destroy it */ Errors from system calls should be set using Tcl_SetErrno(errno).
elfring added on 2003-11-05 06:06:28:
File Added - 66487: tclPosixThread.zip
elfring added on 2003-11-05 06:06:27:
Logged In: YES user_id=572001 1. I reorganized the source code for the files "tclUnixThrd.h" and "tclUnixThrd.c" of TCL 8.4.4 to get a better understanding. I refactored the code into separate files. I added a few functions. Can my approch help in our discussion? 2. Does it fit to the expected coding style? 3. How do you think that attributes should be added to the library? 4. I am looking for documentation of the function "TclRememberMutex". - How is the data allocated? 5. I am unsure about the statement "All of these synchronization objects are self initializing." (http://tcl.tk/man/tcl8.4/TclLib/Thread.htm). 6. Should error values be forwared to the variable "errno"?
vasiljevic added on 2003-08-26 22:33:26:
Logged In: YES user_id=95086 1. As far as I'm concerned, everybody is welcome to submit the whatever kind of TIP he/she likes. It is open source development, so anybody is more than welcome to contribute. 2. I do not have any work done in that direction yet. I wanted to get Tcl thread API to the point that we can shred the AOLserver's custom pthread wrapper, but I'm very short on time lately. 3. Everybody would benefit from some marketing :)
elfring added on 2003-08-25 17:58:50:
Logged In: YES user_id=572001 1. Can we start with an informational TIP? It will be transformed to a project TIP later. 2. Have you got wrapper templates or classes? Can an example show how easy the missing functions can be added? 3. Would a little "advertisment" help to get more developers to copy and write the code?
vasiljevic added on 2003-08-24 22:42:47:
Logged In: YES user_id=95086 I know. Chances of getting the TIP accepted are much higher if there is a ref implementation sitting somewhere. In this case, I have very little concern about adoption of the TIP. It is the implementation which troubles me. Not because of the complexity (it is just adding new pthreads C-wrappers here and there) but because of the lack of time. So, lets see if I'll be able to manage that :)
dkf added on 2003-08-24 22:28:38:
Logged In: YES user_id=79902 Not strictly true, and certainly not to submit it (getting a vote to accept is slightly different; we've had problems in the past with TIPs where the author hasn't been able to arrange implementation effort.) TIPs do require a commitment of time and effort from *someone* though, as our engineering standards are rather high.
vasiljevic added on 2003-08-23 21:56:19:
Logged In: YES user_id=95086 To submit the TIP one needs to have a reference implementation, otherwise nothing will happen. I had very little time to catch up with that so it just keeps sleeping. Hopefully, times will get better.
elfring added on 2003-08-23 20:31:02:
Logged In: YES user_id=572001 When will this Tcl Improvement Proposal (http://www.tcl.tk/cgi- bin/tct/tip/2.html) be started?
elfring added on 2003-08-16 16:43:52:
Logged In: YES user_id=572001 It needs some time to get the function interface "http://www.tcl.tk/man/tcl8.4/TclLib/Thread.htm" up to date ...
vasiljevic added on 2003-05-30 23:44:35:
Logged In: YES user_id=95086 We'll soon make a TIP in order to get more of pthreads available on the C-API so everybody can take advantage of that.
elfring added on 2003-01-13 01:00:58:
Logged In: YES user_id=572001 Normative documentation: http://www.opengroup.org/onlinepubs/007904975/basedefs/pthread.h.html
elfring added on 2003-01-13 00:12:18:
Logged In: YES user_id=572001 Can anything from the class library "Jsync - collection of synchronization classes for Java" (http://www.garret.ru/~knizhnik/jsync-1.03/package- jsync.html) be used for the TIP?
elfring added on 2003-01-08 18:08:25:
Logged In: YES user_id=572001 Are the comments from the discussion "library proposal" (http://groups.google.com/groups?selm=IhBS9.18% 24LJ1.422974%40news.cpqcorp.net) also applicable to the threads package?
elfring added on 2003-01-05 17:47:52:
Logged In: YES user_id=572001 Is cooperative multitasking (like "http://www.gnu.org/software/pth/pth- manual.html#the compromise of pth") or preemptive multithreading provided by the internal TCL threads library? Does this library support simultanous execution on several processors?
elfring added on 2003-01-04 19:33:34:
Logged In: YES user_id=572001 I point to another good description: Guide to the POSIX Threads Library - http://www.tru64unix.compaq.com/docs/base_doc/DOCUMENTATION/V51_HTML/ARH9RBTE/TITLE.HTM newsgroup: comp.programming.threads
elfring added on 2002-12-31 04:14:34:
Logged In: YES user_id=572001 1. I add some links to helpful information about algorithms and technologies. 1.1 Can the API be improved so that an implementation of the CORBA standard "Concurrency Service" (http://www.omg.org/technology/documents/corbaservices_spec_catalog.htm, http://fpx.de/Combat/) would be possible with TCL? 1.2 Can the package profit from "Communicating Sequential Processes" (http://www.cs.ukc.ac.uk/research/groups/crg/research_interests.html)
elfring added on 2002-12-29 05:40:56:
Logged In: YES user_id=572001 1. Interesting... You mentioned "pthreads". - So I made suggestions in this direction. I did not know that the programming interface should improve on the C level before further enhancements can be made. I would prefer that all functionality of the underlying thread function or class library will be also provided on the TCL level. The functions that you are not convinced to exist because they are needed. The frequency for their usage patterns is different, of course. 5. I'm curious about that. Please put the ideas into the TIP. 6. I would prefer an other wording than "vwait". Would a "waiting on a thread variable" be easier to understand?
vasiljevic added on 2002-12-29 01:29:12:
Logged In: YES user_id=95086 1. Basicaly, the threading extension exposes the Tcl API. It does *not* add to the API. Speaking of the API, there are some API corners that need to be improved and I'm working on that. There will be a new TIP on this soon. Yes, all the stuff you've described are ok, but do you really need them on the script level? I think I might wrap the rwlocks, although I'm not really convinced that they are a good thing to use. 3. Sure. 5. You'd need to make your own wrappers arround pthread_mutex and friends and provide a kind of debugging mutex implementation where you track who (which thread) is holding the mutex and be able to kill the process on deadlock attempts. I might be proposing this in the upcomming TIP. The pthread does not do this for you, but you can make one yourself. 6. Nowhere. It is an idiom one uses frequently when entering the event loop forever, like in some server applications, for example. 8. Message passing == script passing. It is basically the same thing. 10. Ok, the doc is little bit terse on that. I might get put it more clear.
elfring added on 2002-12-29 01:06:31:
Logged In: YES user_id=572001 9. It is good that the TCL behaviour for the function "exec" is different from that what can be read under "http://docs.sun.com/db/doc/806- 6867/6jfpgdcn4?a=view".
elfring added on 2002-12-29 00:39:24:
Logged In: YES user_id=572001 1.8 attr_... - http://docs.sun.com/db/doc/806-6867/6jfpgdcmi?a=view
elfring added on 2002-12-29 00:25:27:
Logged In: YES user_id=572001 1. I know that all things that are not provided by the package yet can be done by own additional programming. But I read the available documentation in this way that "semaphores" do more (They can be shared between threads and processes.) and are a synchronization technique with other uses than mutex and condition variables alone. The following items are also part of the POSIX standard 1003.1c and may be offered in a future version of the thread package. It is good that the thread pool functions were implemented without locking. 1.1 mutexattr... - http://docs.sun.com/db/doc/806- 6867/6jfpgdcmn?a=view 1.2 mutex_consistent_np, mutex_trylock - http://docs.sun.com/db/doc/806-6867/6jfpgdcmo?a=view 1.3 condattr... - http://docs.sun.com/db/doc/806- 6867/6jfpgdcmp?a=view 1.4 cond_wait (I assume that the function "pthread_cond_timedwait" is used by the package.), cond_reltimedwait_np - http://docs.sun.com/db/doc/806- 6867/6jfpgdcmq?a=view 1.5 sem_... - http://docs.sun.com/db/doc/806-6867/6jfpgdcmr?a=view 1.6 rwlockattr... - http://docs.sun.com/db/doc/806- 6867/6jfpgdcms?a=view 1.7 rwlock... - http://docs.sun.com/db/doc/806-6867/6jfpgdcmt?a=view 2. great 3. Will you add this explanation to the documentation? 5. Do you know a function library that will help to get that information? 6. In which package is the call "vwait forever"? (You called it "this little bastard" once.) 7. Okay - Evaluation in a locked mode... 8. What you you mean with "message passing"? 9. Well, then your implemenation does not need any special treatment in this case like other threading libraries. 10. Your answer...? Thanks
vasiljevic added on 2002-12-28 21:06:47:
Logged In: YES user_id=95086 1. Semaphores are ok. So are critical sections and recursive locks. But, what I want to achieve is *simplicity*. Thread programming has become pretty simple thanks to the Tcl compartment thread model. If you look at the tpool.tcl source you won't find a single condition variable or a mutex. Yet, relatively complex implementation of threadpool is done with Tcl alone and it is only about 20% slower than the C-level implementation. Furthermore, the Tcl implementation costed about 2 days development. The C-one took 10 days. So, economically speaking, the Tcl version is the clear winner. Needles to say that a semaphore can be done in pure Tcl with a mutex and a condition variable, which the threading extension already supplies. 2. I will include the keyed-list datastructure in 2.6. 3. It will just have the refcount of 0 meaning that caller of such "non-preserved" thread may never know wether the thread is still there for the next operation. But, it will not exit w/o being told so with a "thread::release" 4. I will shortly revamp the cond/mutex Tcl API support. Watch for the new TIP. This way the API will be more powerful and you can get many new info/statistics about the mutex etc. 5. Inherently impossible by pthread alone. 6. There is no such command in the thread package. There is however thread::wait and it is described. 7. There is already thread::eval which evaluates a script under the mutex, either implicit or explicit one. 8. I think that the message passing, together with channel transfer and detach/attach allows one to implement almost any task. 9. As a script programmer, you can safely do the "exec" from the MT Tcl shell. The underlying implementation will do the right thing.
elfring added on 2002-12-28 20:57:38:
Logged In: YES user_id=572001 10. Can the explanation for "spurious thread wake-ups" with the function "thread::cond wait" become more comprehensible? (Interrupted Waits on Condition Variables - http://docs.sun.com/db/doc/806- 6867/6jfpgdcn9?a=view)
elfring added on 2002-12-28 20:33:42:
Logged In: YES user_id=572001 9. Is there anything to be careful about forking a new process? (http://www.gnu.org/software/pth/pth- manual.html#process forking)
elfring added on 2002-12-28 20:13:25:
Logged In: YES user_id=572001 8. Is the technique "sending TCL scripts" the only inter-thread communication mechanism with the thread package? Will "message ports" (http://www.gnu.org/software/pth/pth-manual.html#message port communication) be applicable?
elfring added on 2002-12-28 04:10:32:
Logged In: YES user_id=572001 1. Another POSIX item - semaphores (Comparing Primitives - http://docs.sun.com/db/doc/806- 6867/6jfpgdcn0?l=de&a=view) Will this be offered, too?