Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
107 check-ins using file tests/results/mail/reply/default/wrongargs-notenough version 17e010dcec
|
2024-02-17
| ||
| 17:16 | tweak: github description derived from url Leaf check-in: 4e61dd2645 user: aku tags: trunk | |
| 17:16 | fix: missing variable initializations check-in: 2faf0907e5 user: aku tags: trunk | |
| 16:56 | fix: whitespace check-in: ae0f59b109 user: aku tags: trunk | |
| 16:55 | fix: typos, and whitespace check-in: 297762a29b user: aku tags: trunk | |
|
2022-07-28
| ||
| 21:04 | fix: broken list of updates, due to the new `last` field colliding with loop variable fix: comparison of changed time vs cycle, was string, now proper epoch fix: remove direct calls into ops reporting from exec. indirect through callback fix: mishandling of TLS edge cases by Tcl `httpget`+`tls` packages. switched `url` utils to call on external `curl` instead tweak: website - renamed the pages for repo tables, page titles feat: website - added repo tables sorted by commits, forks. added columns for same data feat: website - added project tables sorted by #repos, #stores. sort order descending tweak: website - sorting for size, commits, forks descending tweak: website - reduce simple repo lists for projects to show only primaries chore: website - added '#forks' column to these lists, removed fork indicator from tags Leaf check-in: d131c978bf user: aku tags: vcs-plugged | |
|
2022-07-24
| ||
| 19:44 | fix: handling of `git://*` urls in the main vcs layer. cannot be url checked, bypass that action. leave check to the git commands invoked by the git vcs driver fix: weird insertion of empty line in `update` output from time to time. fix: export of state, skip projects without repository attached to them. this can happen when a phantom is removed after failure to complete feat: extended git VCS driver to invoke garbage collection on repositories after their updates. I.e. keep disk size in check feat: new chirurgy command `insert`. splice repository into the system, by url, project name, vs, and storage data. reject only if url is already present. no other checking. in other words, this can be used to add a broken, unreachable url. needed to recover when a broken repository was accidentally removed feat: new chirurgy command `stats`. actually not dangerous. regenerates size and commit data for referenced repositories. good to use after an `insert` feat: new output options for command `list`: --csv, --tsv, --pretty. pretty is default and pre-existing formatted table in terminal. csv, tsv generate respectively formatted data - they report the __entire__ query result, i.e. all columns (even if not shown by pretty), and all rows (--offset, --limit are void) feat: new sort options `--by-time` and `-by-commits` I.e. added ability to sort list results by time (last duration) and number of commits. both sort orders impose filter clauses to suppress repositories which do not have the relevant data (.i.e. phantoms) chore: extended `repository` table with dedicated column for duration of last setup/update. easier to pull out for display, and makes it available for use in ordering chore: tweaked `update` output, to align VCS names as well chore: removed various unsed/disabled pieces of code check-in: 049db6a63d user: aku tags: vcs-plugged | |
|
2022-07-22
| ||
| 16:17 | chore: whitespace cleanup chore: code cleanup, removed some unused procs, and gone exports tweak: help section indenting tweak: dropped ability to auto-regenerate site after changes - nice only for small setups does not scale to large setups where site generation can take minutes feat: hack command to change the vcs of a repository and its store check-in: 6dd242455a user: aku tags: vcs-plugged | |
| 12:39 | fix/todo: do not create phantoms getting blocked at all no leaving around of disabled+hidden phantoms check-in: 2d19cf68c5 user: aku tags: vcs-plugged | |
| 11:13 | now this one ran away from me a tiny bit ... - on the viewing side completely revamped the internals for getting lists of repositories. collapsed the large set of similar functions into a single function holding the core sql (returned fields, tables, base clauses to join the tables), plus a single function configuring the core with more clauses (filter, ordering, offset/limit) The various list commands then fed into that at first, and then got (mostly) subsumed into a massively expanded `list` command with lots more options to choose the filters, ordering, etc removed command `issues`. use `list --issues` instead removed command `disabled`. use `list --disabled` instead feat: Many new options for `list` feat: `list` substring search extended, accept multiple strings refactored and tweaked the processing from repo series to table as well feat: Added storage of lower-case project names and url. done for proper case-insensitive search with the move of the match process into sql [1]] - the fork tracking got redone a third time now the main change is that the immediate checks of found forks for validity/reachability is defered to happen when trying to complete the phantoms, i.e. actually setting up the store for the fork when update reaches it. this reduces the load on the remote servers significantly the only place where a series of validiity checks can still happen are imports. fix: modified `m url ok` to recognize status `429 too many requests`, and retry with a delay (using the `Retry-After` response. And a default of 60 seconds if no specific request was made forks who fail creation are removed. Of course, on the next update of the primary they will be recreated if they are reported again. however note that private forks are reported despite being inaccessible for cloning. To prevent such from bouncing between created as phantom and removed on failed completion a new tracking table counts such bounces and phantoms which exceed a configurable threshold are taken offline immediately when re-created the next time [[2]]. at that point bouncing stops feat: new command `config block` - view/modify the block threshold for phantoms feat: new option `--blocks` for `statistics`, to show block table feat: new commands `track`, `untrack`, `hide`, `publish` to view/modify the new flags for fork tracking and private repos removed `--track-forks` from `add`. default off, and use new `track`, `untrack` commands instead refactored the innards of `add`, `submission accept` and `import` for the above tweak: redid import/export as well, with proper long names for the commands feat: export/import now saves/uses tracking and privacy state flags tweak: removed any mentions of phantoms from the website, be it links or counts. for the site these simply do not exist required some reworking of statistics internals to handle requests for either with and without phantoms - general feat: new debug/test command `colors` to show the terminal colors feat: new debug/test command `url-ok` for direct access of the url validation functionality tweak: changed layout of config and statistics displays. separated into panels, shown left to right tweak: modified the messaging of command `update` to look like a table for normal operation (no failures or warnings), and mostly like it when interrupted by such fix: handling of `--extend` in `add`. validator default generated bogus data for it fix: mirror-vcs communication and import/export - flatten project names to single line, encode/decode linebreaks fix: mirror-vcs communication, read ops log even for failure, to get the detailed error message(s) fix: rolodex cleanup when encountering invaild ids left in it fix: fork retrieval for github. `git hub forks` delivers only the first 100. replaced with explicit `curl` [[3]] chore: removed a number of internal, now superfluous commands notes [[1]] todo: check if sqlite collating allows for the same [[2]] todo: do not create them at all. do not leave an (unused) phantom around [[3]] todo: check if Tcl http package is suitable check-in: a785918096 user: aku tags: vcs-plugged | |
|
2022-07-12
| ||
| 19:44 | fix: missing commit of vcs changes supporting [1dbe8e5f9c], and other lost changes fix: non-path safe base64 encoding for various store files fix: missing skip of unused stores in site generation, ditto in site db synch feat: show page generation time (extended footer) in site generation chore: rework the per-vcs store/repo count display in site fix: edge cases in cleanup of lost stores (no store directory), lost code feat: attempting to enable a disabled repository now validates the url fix: status display for stores with issues chore: reworked fork handling. added delays to reduce request load. stop after a fixed number of adds (defer more to future updates) note: `git hub forks` retrieves only first hundred forks, even if asked for `--all` note: doc reading shows that direct use of the api via `curl` or http package should be feasible note: given the extreme number of forks seen thinking on redesign to make fork management easier (better info where possible, auto-track off by default, knowing the number of forks (even if not tracked)) check-in: e9475f891f user: aku tags: vcs-plugged | |
|
2022-07-11
| ||
| 19:06 | fix: internals of store setup - defered init means validate again at actual setup time fix: missing removal of debris (partially init store) when adding a store fails fix: inverted condition for updating store attributes after update fix: bad column names when saving updated store attributes fix: table style for come-around-mail. go back to plain ASCII for this (box drawing chars may not display properly) feat: show counts of removed/same/possibly added forks after update feat: show phantom count, and per vcs store counts in statistics (cli, web) feat: as part of the previous, get per vcs repository counts feat: show count of pages when (re)generating the site check-in: 1dbe8e5f9c user: aku tags: vcs-plugged | |
|
2022-07-09
| ||
| 21:38 | chore: comment formatting tweaks (fill-column 90), removed some disabled left-over old code chore: removed various left-over unused project name arguments in repo/store/vcs commands feat: new command `archive` to extract stores from the system, various modifiers feat: split `site make` from `site on`, make the former a dedicated one-shot. feat: implemented repositories to function without a store - phantoms use: new forks from setup or update are added as phantoms -> time is spent only on url check - still linear in #forks, - yet much less than full store setup, i.e. cloning -> time for setup of highly forked repo is kept limited, - distributed across follow-on updates followup updates incrementally initialize the missing stores lots of places modified to deal with a missing store fix: calculation of statistics for an empty database (no projects, repositories, stores) fix: missing default of current repos for commands `enable`, `disable`, `move`, `remove` also handle an empty database fix: missing update of `repo_pending` when adding/removing repositories check-in: 01925ddb24 user: aku tags: vcs-plugged | |
| 12:42 | chore: general whitespace cleanup. check-in: dc6bac384f user: aku tags: vcs-plugged | |
| 12:39 | chore: reworked the structure of store directories, simplified callers (no need to rename stores anymore) chore: documented the new store state files chore: stores/vcs now discard leftover v2 state files in stores check-in: 98b9508198 user: aku tags: vcs-plugged | |
|
2022-07-08
| ||
| 22:58 | feat: extended command `add` with option `--extend` (add + merge semantics) feat: new command `move` feat: new command `statistics` feat: new commands `project`, and `projects` for project details and listing feat: tweaked table formatting, now using the box drawing characters feat: new site pages for statistics, projects, and project details chore: added code to github driver to clean up v2 state files (fork management) chore: db setup tweak, removed the auto-increment modifiers from all tables still using this unrecommended thing chore: redone page generation internals for easier reading of emitter code chore: redone store pages, account for multi-project possibility chore: reworked the guts of commands `export` and `import` for new semantics, and shared stores chore: reworked the guts of commands `merge` & `split` for the new semantics (store ops on repos, not projects) chore: tweaked index pages, removed small statistics, issues, disabled - all in the navbar chore: various tweaks to command messaging check-in: 039ce288e8 user: aku tags: vcs-plugged | |
|
2022-07-04
| ||
| 07:12 | chore: reworked merge & split for the new semantics (store ops on repos, not projects) check-in: 25f69ccbde user: aku tags: vcs-plugged | |
| 07:11 | Update system introduction to match changes. check-in: d482cec0ee user: aku tags: vcs-plugged | |
|
2022-07-03
| ||
| 18:29 | feat: list projects feat: extended project rename to allow merging (= bulk move of repos in a project) fix: cleanup of still present old command names (mset -> project) fix: bad debug tags in new code fix: comment typo check-in: 636c7d468e user: aku tags: vcs-plugged | |
|
2022-07-02
| ||
| 17:09 | chore: extended tracing for checking urls, show TLS activity check-in: 202c8a6889 user: aku tags: vcs-plugged | |
| 13:36 | feat: support for linking to a related web site check-in: 90fe27baa6 user: aku tags: vcs-plugged | |
|
2022-07-01
| ||
| 19:20 | database: extended notes about dropping of table `name`. check-in: 670408821a user: aku tags: vcs-plugged | |
| 19:14 | submission accept: Extended ability of user to override submission data. submission accept: New options alias -V (--vcs) submission accept: New option --url, alias -U check-in: 9e0a0cb23c user: aku tags: vcs-plugged | |
| 15:31 | Tweaked the formatting of avg update times (show window size, elements going into the result) Bugfix: Normalize the project name before putting it into lists. Flatten to single line, remove superfluous whitespace. Look at the description of https://github.com/luaforge/tclua. The multi-line broke MD formatting of the table) Added run time stamps to site generator (only at the major places) check-in: b6d8811f29 user: work tags: vcs-plugged | |
| 14:21 | 1. db migration, main and site 1. mirror set renamed to project. 1. updated cmdr spec, most commands 1. several commands switch to take repos instead of projects 1. completed the conversion of common vcs to call out to the backoffice code 1. reworked repo, store, vcs handling for the new schema 1. iterated over commands to check for issues and fix them 1. previous not complete, see merge, split, web pages 1. even so, the majority of the functionality looks to be working again 1. especially add/update of repos, i.e. the core. 1. and yet, still more testing needed to validate handling of issues with network and/or repo servers. 1. issues should be more localised however, as access (especially to github) should now use smaller requests overall 1. website, touch up of generated pages (lists, store details, new list of forks for > 5 forks of a repo) 1. see TODO.md for general things outside of fixes check-in: 5c2d6c24ac user: work tags: vcs-plugged | |
|
2022-06-28
| ||
| 15:19 | Fix v2/v3 repository migration check-in: 0f1fc429ce user: work tags: vcs-plugged | |
| 14:20 | Fix forgotten removal of outdated tables from v2/v3 migration. check-in: 666134b711 user: work tags: vcs-plugged | |
| 14:12 | Brought v2 schema docs into line with code. Writeup of issues. Writeup of v3. Added migration step from v2 to v3 schema. __BEWARE__ At this point migration works, and everything else will be broken, as it is assuming the v2 schema. check-in: 74724d9d0c user: work tags: vcs-plugged | |
| 14:06 | Fix eols in scratch area check-in: e0637d5770 user: work tags: vcs-plugged | |
| 14:04 | Blind work in vcs layer have more commands call on the backoffice code. Untested. Likely broken. check-in: 00532db491 user: work tags: vcs-plugged | |
|
2019-10-30
| ||
| 17:30 | `m vcs *`: - Converted `url-to-name` (was `name-from-url`) - Converted `update`. - Removed `revs`, `remotes`. - Still kept: `detect`. - Added tests for the converted commands. check-in: ce42f93675 user: aku tags: vcs-plugged | |
|
2019-10-18
| ||
| 05:09 | `m ops client`: - Fixed `fork` vs `forks` mismatch. - Ensured proper exit status for failure, regardless of failure cause. `m exec`: - Added `get+route` to the `m exec` variants (Allows re-routing of lines from actual out/err to logical out/err before acting on them, i.e. re-routing affects the progress reporting done, and exit state). - Further added method `err-last-get` to retrieve error state of last `get+route` call. `m vcs *`: - Conversion of `setup`, `mergable?`, `merge`, and `split`. - Extended `setup` to return commit count and size (kb) for store. - Moved the command determining store size out of VCS into `m exec`. Lower-level, shared. - Added tests for the converted commands. check-in: 5ac541ea09 user: aku tags: vcs-plugged | |
|
2019-10-17
| ||
| 07:02 | Removed `vcs-op` from main app (cmdr, glue). Going with the separate `mirror-vcs` app. Converted `export` over to new system. Reworked `m::ops::client` interna, avoid use of `::exit`. Added tests for `mirror-vcs` and the converted commands. check-in: 543460f8c4 user: aku tags: vcs-plugged | |
|
2019-10-16
| ||
| 05:21 | Scratch data for dev check-in: cddcfce63a user: aku tags: vcs-plugged | |
|
2019-10-15
| ||
| 19:36 | Start on moving VCS operations into separate executable, with main executable calling out. First step in making VCS more pluggable, extensible without changing the core. So far ops `version` and `cleanup` are moved/redone. New utility package for generic client handling. Added exec variants to specifically handle the plugin comm protocol (stdout, ops log, ...). check-in: 66213af5e1 user: aku tags: vcs-plugged | |
| 19:31 | Updated schema docs to current setup. check-in: 1f4544326b user: aku tags: trunk | |
| 19:29 | Moved processing of debug flags (and EV) into utility package. Added use to the support apps. check-in: 6856986b92 user: aku tags: trunk | |
|
2019-10-07
| ||
| 20:55 | Brought test suite back to spec and matching the code for all but new commands. To come, a commit adding tests for all the commands without and testable. Commands with side effects (primarily sending mail) not yet testable. Requires a mock smtp server for capture. check-in: 3220094340 user: aku tags: trunk | |
|
2019-10-04
| ||
| 06:01 | Larger schema reorg. Dropped table `name` and folded its information into the single user, `mirror_set`. check-in: ed5ddd6c2f user: aku tags: trunk | |
| 05:50 | Fixed typos in sql invokation check-in: f96408ad63 user: aku tags: trunk | |
| 04:14 | Fixed breakage to cycle mail done by previous commit. Tweaked the content of the table (reordered, removed previous data) check-in: a6ea53adbb user: aku tags: trunk | |
|
2019-10-03
| ||
| 19:37 | Added mail configuration `width` (state element `mail-width`). Determines the width of tables placed into generated mails. Currently the only application is in the generation of the cycle report mail. New command `mail width`. Affected commands `updates` and `debug cycle-mail`. check-in: 4e7a29301d user: aku tags: trunk | |
| 00:58 | Exposed SMTP narrative tracing to mail config commands (show, set). check-in: ea95a59325 user: aku tags: trunk | |
|
2019-10-02
| ||
| 19:32 | - `debug cycle-mail` Extended to send as mail, configured or specific destination. Plus backend work for that. Fixes to the mail sending code which prevented this from working during updates. Error was swallowed. - `update` Fixed backend wrongly still using inactive/disabled repositories. - vcs backends. A bit more narrative tracing. check-in: 119798879d user: aku tags: trunk | |
| 02:02 | Changes - `config show` : Show pending and total mirror sets with take per cycle. - `pending` : Show number of pending and total mirror sets before table. - `update` : Show start of current cycle in log of current run. Show more logging in the hook called when the cycle turns. Show take, and number of pending and total mirror sets. - mset backend : Added method to return count of pending mirror sets. - Changed variable names for all the generated mails, and unfolded lines. Enables dev to see what is generated at each location without requiring an editor. check-in: 5a2d58e068 user: aku tags: trunk | |
|
2019-02-07
| ||
| 07:36 | Updated README to note svn support. check-in: 44fd555cd3 user: aku tags: trunk | |
| 07:27 | Fixed issue with cycle reporting in site, for unchanged repos. Also influenced by inverted condition on which pages to get insertions. check-in: 307c378610 user: aku tags: trunk | |
| 07:08 | Tkt [667f777f17]. Added support for SVN. Fixed issue with forks handling for vcs which do not support such. check-in: f4c6f041b8 user: aku tags: trunk | |
| 06:56 | Fake out SForge's handling of git repositories. check-in: 53d0011159 user: aku tags: trunk | |
| 06:55 | Added tracking of the cycle roll over, with sending of mail if a destination is configured. New config command (`report`) for same. New debug command (`cycle-mail`) to check mail generation for cycle roll over. The start of the current cycle is stored in a new state key, `start-of-current-cycle`. The start of the cycle before is stored under `start-of-previous-cycle`. Cycle information is reported with the configuration, in `updates`, and all the list-based pages of the generated site. check-in: b03bdbfd3f user: aku tags: trunk | |
|
2019-02-05
| ||
| 18:25 | Fixed bad management of the tracking window. Now properly extending it when not at configured max length. check-in: 351703b3b7 user: aku tags: trunk | |
|
2019-01-25
| ||
| 20:14 | Ticket [7a4a72b413]. Added caching of commit counts to database. Added caching of github fork counts to database. Added tracking of size/commit changes (current, previous). Added tracking of update times (min, max, last N). N is configurable via new command `config window`. Default N = 10. Extended command `details` to show the new information (commits, changes, update time stats (moving average)). Same for web site generation. Extended commands `add` and `import` to show setup time for store, and github fork counts. Warn at higher fork counts, stop at very high fork counts. Extended command `update` to show update time for store, and (new) github fork counts. Extended command `updates` to show size changes, commits and commit changes, and time spent on last update. check-in: 46ca952cd0 user: aku tags: trunk | |
|
2019-01-22
| ||
| 23:54 | Added test command `debug mail-setup`. Generates and sends a fixed test mail using the current mail configuration. Extended the `details` command to show the store path. check-in: f6d37ebcae user: aku tags: trunk | |
| 04:41 | Multiple icons for state information in website. Commands `enable` and `disabled` extended to accept multiple repository references. Added command `disabled` to show table of disabled repositories. Extended website with table of disabled repositories as well. Modified the database backend for tables of issues to exclude fully disabled stored. Affects website and command `issues`. Extended all store lists with issue flag. Save fork information in sorted order for easier comparison. check-in: d5d8695d63 user: aku tags: trunk | |
|
2019-01-21
| ||
| 19:59 | Ticket [0fe3495c2f]: Modified the handling of stores with issues: - Implemented user command `issues` to see stores with problems in the last update. - Cache information about existence problems (%stderr not empty) into the database, for easier access and filtering. - Extended the database schema of `store_times` to record this flag. - Added migration code initializing the flag cache from the primary information (%stderr logs). - Extended the store update logic to manage the flag cache directly. - Added a low-level database function retrieving the list of troubled stores, with filtering on the new flag. - Changes to web site generation. - Simplified. Now uses the new database function. - Always generate the page, even if empty. check-in: 833e635b29 user: aku tags: trunk | |
| 19:12 | Fixed message bug in `enable`/`disable` core. Extended `update` to show primary repository when printing timing. check-in: e88284510b user: aku tags: trunk | |
| 19:08 | Added automatic adaption of tables to terminal size (mostly width, some height). Extended `limit` for `list`/`rewind` to allow `auto` in support of the latter (height adaption). check-in: 72663b138e user: aku tags: trunk | |
| 18:47 | Provide terminal size information through global state parameters. Needed for upcoming adaptive tables. check-in: a7265d6957 user: aku tags: trunk | |
| 18:41 | Tweaked some command names slightly. Reorganization of the help section structure. check-in: fb0ace30b1 user: aku tags: trunk | |
| 18:33 | Stop crashes in update, capture and save to stderr log. check-in: cc538d9628 user: aku tags: trunk | |
| 18:32 | Save fork information (local, remote) into files for help with post-mortems check-in: c3f02e8700 user: aku tags: trunk | |
| 18:31 | More stderr cleanup, around notes about packing and garbage collection check-in: 8657898c2f user: aku tags: trunk | |
|
2019-01-11
| ||
| 21:29 | Tkt [f565633195]. Implemented a `details` command to inspect the status of specific repos/stores from the command line. check-in: 20c51d38ee user: aku tags: trunk | |
| 20:47 | Handle connection abort in url validation as OK. I.e. urls exists, got connection to server, and the protocol expected something non-standard. check-in: c64c1c09a5 user: aku tags: trunk | |
| 00:34 | Added early re-validation of store urls in the high-level VCS layer. Avoid calling on the vcs client if the repo is not reachable. Print information into the std logs for cli output, and website. check-in: 02907658c8 user: aku tags: trunk | |
|
2019-01-10
| ||
| 19:11 | Added code measuring and printing the time needed to perform an update, per store. check-in: 2472bbe909 user: aku tags: trunk | |
| 16:15 | Extended submission ids with `%c` for a `current` submission. Makes automating `submission accept` with a loop easier when there is nothing to reject, no need to know all the id numbers. Trap and slience child status errors, i.e. `exit 1` for `git hub forks` and use empty list as result. Until `git hub` has handled the issue with some repos reacting bad to the redirection of their output in this manner. check-in: 59e366f72c user: aku tags: trunk | |
| 04:31 | Fix bad references to main database in submission helper. - Two direct references for transactions - One indirect references for VCS names and codes. - Extended site database to hold VCS names and codes. - Extended sync to re-fill the new VCS table. - Added messaging already used in the local setup. Force definition for TERM if missing from environment. More narrative tracing in the exec utilities, and vcs management. VCS github driver - Reorder handling of new and removed forks. Remove first, then add. Required for rename operations, the old definition must be moved out of the way first before the new one can be made. check-in: e13be6dfcd user: aku tags: trunk | |
|
2018-12-06
| ||
| 06:23 | Capture possibility that a github project does not have a description. check-in: 03700b698f user: aku tags: trunk | |
| 03:38 | Undo 'partial clear', not fully working at the moment. Fixed a return of FixUrl missing its result, breaking the caller. check-in: 3048bfbc2c user: aku tags: trunk | |
|
2018-12-05
| ||
| 05:11 | Ticket [f3f481018b] Integrated support for web submission via CGI app. check-in: 10abc1e94a user: aku tags: trunk | |
| 05:02 | Submission sync - Implemented `site sync` command for testing just the sync operation. - Fixed issues with sync operation - Fixed bad time marker in db setup preventing run of operation. - Fixed typos in description variable preventing sync of same into main. Placed submission CGI helper into the site navigation bar, exposing it to users. Closed-Leaf check-in: 39430f6ffa user: aku tags: f3f481018b-cgi-submission-support | |
| 00:39 | Reorganized the cli command structure - Centralized all four configuration commands (`show`, `take`, `store`, `limit`) under `config`. `config show` is default. - Centralized all five submission management commands (`submissions` (-> `list`), `submit` (-> `enter`), `accept`, `reject`, `rejected`) under `submission`. The renamed commands (Note `->`) have aliases under their old names. `submission list` is default. A new alias is `rejections`, for `submission rejected`. Added new command `drop` to remove entries from the rejection table (tests and other cruft). logic/submissions Changed `accept` and `reject` to handle not only the specific submission they are invoked with, but also all its duplicates, in terms of the referenced url. Support for a validator of rejections. New validator for rejections, used in command `drop`. check-in: 5472a887e3 user: andreask tags: f3f481018b-cgi-submission-support | |
|
2018-12-04
| ||
| 21:37 | main database Added table to help with syncing of submissions. Records submissions processed by the cli (accepted, rejected). IOW submissions we are done with. site database Dropped table `sync`, not used by the sync implementation. logic/submission Extended to use the new table web/site Implemented sync operation for submissions. Plus more docs on the relevant dataflow. Note: Sync and modified submission ops are not tested. check-in: c109adadcf user: andreask tags: f3f481018b-cgi-submission-support | |
| 09:36 | mirror-search Meta data fixes From `mirror-submit` Fake TERM setup stderr redirection mirror-submit Reworked to use the site database for all information needed, and as the sole destination for any submissions. Reorganized to have all database ops in a single transaction before starting to emit any page elements. Using new tables to cache url validation results, and url conversions into descriptions. Both ops reach out to external sites, making the submission script an indirect attack vector. Caching the results, even if only for a limited time (30 minutes fixed at the moment) will reduce the rate of access for any url going through the submission script to at most once per cache expiry interval. Add `clear` button to the form to reset url, vcs type, and description. It keeps session id, submitter name and email. site database Added two new tables for caching of url validation, and conversion to descriptions. Further fixed the bogus unique attribute on urls in submissions. Unique is url + session, not url alone. vcs github Tweak generation of description somewhat check-in: 969a4c87aa user: aku tags: f3f481018b-cgi-submission-support | |
|
2018-12-03
| ||
| 04:32 | vcs github: Another attempt at fixing the issues with fork handling. Query the local store directly for the known remotes, not the cache file. Now nothing should really be able to get out of sync anymore. check-in: 4d01e57f61 user: aku tags: f3f481018b-cgi-submission-support | |
|
2018-12-02
| ||
| 21:42 | mirror-search: Fix status icon link. vcs git: Move gitlab warning to regular log vcs github: Changed handling of remotes to keep working in case of troubles desyncing the saved list vs what is known to the git store. site generation: Fix inversion of guard in NameFill stripping all names instead of filling the missing. check-in: 355499b0f2 user: aku tags: f3f481018b-cgi-submission-support | |
| 06:23 | Site generation changes: - Created a secondary database for use specifically by the CGI helper applications. - Reduced schema to contain only the data needed. - Updated as part of the site generation process. Mostly push from main (search index, rejected) Push/pull for submission information Note - Exposes less data to possible attack through the CGI helper. - Keeps locking windows short (cli needs secondary only during an update, which is short, and the CGI helpers also make only short transactions. - The cli now fully resolves symlinks in the database location to place the secondary database truly beside the primary instead of beside a symlink. Search helper changed to use the new database and schema. check-in: f5e3712fad user: aku tags: f3f481018b-cgi-submission-support | |
| 01:37 | Site generation changes: - Added a site a directory, ibling to web and web_out, which represents the served site. - web_out is now not only the destination of the generator, but also the staged site. - Added commands to perform a fast changeover of the currently served site to the newly generated (directory moves). Generating the new site does not interrupt the serving of the old state anymore. check-in: b73174868d user: aku tags: f3f481018b-cgi-submission-support | |
|
2018-11-30
| ||
| 06:30 | CGI submission helper: Define a fake environment variable TERM if missing, to make `git hub user` work properly. The variable is not set when the web server is launched from cron, with CGI inheriting. One time too many getting into trouble because of VCS plugins overriding `split`. Now renamed to `cleave`. Further axed any local definitions of `list` as well, renamed to `all`. Get ahead of the possible trouble. Fixed a few issues in the code managing the bootstrap header and footer. check-in: 9b6ff41618 user: aku tags: f3f481018b-cgi-submission-support | |
|
2018-11-29
| ||
| 08:48 | CGI submission helper: Rewrite homepage link in navbar of bootstrap header to absolute. Still something weird going on with the logo image. check-in: 1fd4c85986 user: aku tags: f3f481018b-cgi-submission-support | |
| 08:26 | Site generation: ensure necessary package Mail reply management: Changes do not affect the site, do not regenerate. check-in: b6765a3f76 user: aku tags: f3f481018b-cgi-submission-support | |
| 08:12 | Fixed CGI submission helper Added robustness against internal troubles. Catch all and write to stderr with a marker. To do: Use the mail packages used by the accept/reject cli commands to automatically send a mail about the issue to self. check-in: 06f53183ae user: aku tags: f3f481018b-cgi-submission-support | |
| 08:06 | Made m::url validator more robust against bogus urls. Catch all socket troubles and treat as failure. check-in: 89a2686e09 user: aku tags: f3f481018b-cgi-submission-support | |
| 08:04 | VCS plugins: - Fixed missing cmdr::color requirement - Fixed color command invokations - Replaced a number of vcs cli invokations used to check for their availability with `auto_execok`, i.e. mostly internal search of the PATH. `git hub` is left. An optional sub command of `git` it is not in the regular PATH. check-in: 9e27eb6d03 user: aku tags: f3f481018b-cgi-submission-support | |
| 06:16 | Submission management work - New app: `mirror-submit`. CGI helper. - Database changes Table `submission` New columns - vcode Capture VCS information - description Capture description = name of mirror set to be - session Generated Extended `mirror submit` to capture vcs and description, optional. When not specified it now auto-detects and generates this information like `mirror add` does. Extended `mirror accept` to enable override of vcs and description. When not specified falls back to the information coming from `mirror submit`, or fully generated like `mirror add`. Refactored the mail generation in `mirror accept/reject` a bit, and moved the common header/footer text into separate generator procedures. Moved the generation of repository description / mirror set name into the VCS plugins. Extended the same in the github plugin to query github for user name and project description. Extended the submission logic to take the session id into account. Resubmission of an url in the same session overwrites the existing submission. This enables editing vcs, description, etc of a submission without creating many near-duplicates of it. The session of submissions made before the database change is `:lock:`. As this is not created by neither cli nor cgi these entries cannot be edited. The CLI generates session ids of the form `cli.<day>/host/user`. This enables a user on a specific host to edit all submissions from the current day. The next day they are locked and new entries are made under the new session. The CGI generates a session id of the form `(clicks)`, whenever the form is freshly entered. While reposting the form the session id is kept. Added a new svg to the site generator, for the submit action of the CGI helper. check-in: 27c99c873f user: aku tags: f3f481018b-cgi-submission-support | |
|
2018-11-28
| ||
| 05:52 | CGI search: Add logo to generated table of results check-in: c688ee887b user: aku tags: trunk | |
| 05:52 | cli: Require package `try` iff needed check-in: 9563ee79a6 user: aku tags: trunk | |
| 05:50 | Site gen: Fix location retrieval for CGI helpers. Added vcs logos to site, and generator check-in: 9fb5e85a32 user: aku tags: trunk | |
| 05:27 | Submissions: Extended table to hold mset description and vcs type. Updated logic and glue to take these into account where needed (submit, accept, reject). check-in: 6761d82387 user: aku tags: trunk | |
| 05:23 | VCS: Moved derivation of name from url into the vcs plugins. Extended github plugin to query for owner name and repository description. check-in: b659e7220b user: aku tags: trunk | |
|
2018-11-27
| ||
| 06:38 | mirror-search. New CGI helper app. Substring search over the mirror set names. Wapp-based. Integrated with the site generator. New utility package `m::format`. check-in: 81eb7281d6 user: aku tags: trunk | |
|
2018-11-26
| ||
| 17:40 | gh-pull Fixed processing, dropped list commands, may not be a proper list. Now using a basic regexp to split an entry into the pieces we need. gh-follow Removed from install, tclapp does not handle shell script properly, of course. Index pages - Mention sort order in the title. VCS git - Trim count information, contained trailing newlines. Import - Show number of repos getting imported. check-in: 5dd41eaa0f user: aku tags: trunk | |
|
2018-11-23
| ||
| 21:18 | Integrated removal of the last place where `curl` was used. check-in: 35d75f3062 user: andreask tags: trunk | |
| 21:14 | Add better feedback to import (when it is verifying the provided urls). Too silent otherwise. Fixed promised and missing fallback to stdin. check-in: b58983f633 user: andreask tags: trunk | |
| 20:30 | Drop remaining use of `curl`, replace with `m url ok`. Closed-Leaf check-in: efdc601f5f user: andreask tags: drop-last-curl | |
| 07:24 | README: Fix typos, tweak some markup. check-in: 9e01442c98 user: aku tags: trunk | |
| 07:19 | Added location using this tool as an example section check-in: 1f1b25d76e user: aku tags: trunk | |
| 07:07 | Redone the codeblocks of the README check-in: 213425da9f user: aku tags: trunk | |
| 06:49 | Added README with intro and basic operation. Modified the VCS plugins to have all `detect` methods check for the necessary DVCS client and ignore the url if it is not present. check-in: 02fb6961ce user: aku tags: trunk | |
|
2018-11-22
| ||
| 08:34 | Second github helper `gh-follow`. A level above `gh-pull` it explores the outbound neighbourhood of the specified user U, generating repository lists for all the people U is following. Makes use of `git hub` to script the github API, like `gh-pull`. Both helpers are now installed with the main mirror applications. check-in: 35dcdd2947 user: aku tags: trunk | |
| 07:03 | Added helper application `gh-pull`. Given a github user name, and (optionally) real name it pulls the list of repositories for that user, and writes it as mirror import spec to stdout, with description and language information. Forks are resolved to their root repository. After human review and filtering (removal of unwanted repositories, changing names, descriptions, etc) this can then be directly imported into a mirror instance. check-in: 4f871a923d user: aku tags: trunk | |
| 04:04 | Tkt [be3b04b8ef] Support for hg/mercurial -- Integrated check-in: 89ba2a759b user: aku tags: trunk | |
| 04:04 | Tkt [be3b04b8ef] Support for hg/mercurial Fixes to command options and detection patterns. Closed-Leaf check-in: 6cf270621d user: aku tags: tkt-be3b04b8ef-hg-support | |
| 03:36 | Implemented Tkt [706a587698]. Show statistics in generated site. check-in: 0c5afaf7cc user: aku tags: trunk | |
|
2018-11-21
| ||
| 23:58 | Draft support for hg/mercurial check-in: cc19739120 user: andreask tags: tkt-be3b04b8ef-hg-support | |
| 07:51 | Implemented Tkt [21dc22ab06]. Allow repos to be disabled. Reworked producers and consumers of data series. Series now mostly list of dicts. Easier to extend, easier to ignore new columns where irrelevant. Followup on Tkt [ce4b5f9f0f]. The change by commit [d645475321] was buggy, leaving mset and repository behind, without a store. Fixed. check-in: a59e41fcb0 user: aku tags: trunk | |
| 04:29 | 1. Implemented tkt [6e4f3bbe05]. Search repo/mset by substring over name + url. 2. Implemented tkt [19a54cac60]. Import: Allow empty lines and #-based comments in specs. 3. Fixed tkt [59b33037a4]. Import: Wrongly aborts on merge rejection. 4. Fixed tkt [ce4b5f9f0f]. Import: Wrongly aborts on any repo setup issue. (Ad 4) Issues are now reported without breaking the import. The relevant repository is simply skipped, as is the mirror set if all its remotes fail. In case of errors all captured command output is shown, even without --verbose. Additional guard: Skip duplicate repositories. As part of the above more changes: - New utility package `m::url`. Validate url, resolve redirections. New url validator package on top. Tests added. - Extended the `add` and `submit` commands to perform url validation, redirections are followed. Extended `import` to perform the same. Tests updated. - New command `show` to show the system config (store, limit, take). Using option `--all` shows site and mail config as well. Tests added. Site config now uses colorization to highlight missing mandatory elements. Mail config highlights elements which should be changed. check-in: d645475321 user: aku tags: trunk | |