Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
50 most recent check-ins
|
2020-07-20
| ||
| 04:13 | `lspace::index::property::string` - New index support package. Extends an atom manager with a table to link to things. This gives each linked thing a string-valued property. `lspace::index::property::uint` - New index support package. Links things to unsigned integer values. This gives each linked thing an unsigned integer-valued property. Leaf check-in: e7e5e11309 user: aku tags: x-redo-stores-indexing-tags | |
| 01:23 | `lspace::index::atom` - API tweaks. Rationalized method names, documented in-code. Updated tests. `lspace::index::attribute` - API tweaks. Rationalized method names, documented in-code. Updated tests. `lspace::index::bag` - New index support package. Management of thing bags. I.e. properties with no value attached to them beyond being present or net. This handled as set memebership. `lspace::index::gate` - New index support package. Encapsulates defering the processing of things dependent on other things until such dependencies become available. I.e. internal capture and re-ordering ensuring that dependencies are handling before dependents regardless of actual order of entry into the index. check-in: 145fbcc56c user: aku tags: x-redo-stores-indexing-tags | |
|
2020-07-19
| ||
| 04:08 | Comment fixups check-in: d29a5207ee user: aku tags: x-redo-stores-indexing-tags | |
|
2020-07-16
| ||
| 14:48 | Added new index support package `lspace::index::attribute-manager`. Plus testsuite. Built on top of `lspace::index::attribute`, this handles an entire set of attributes instead of a single one. check-in: 0ea6265c6d user: aku tags: x-redo-stores-indexing-tags | |
| 14:44 | Extended database setup with user args which are passed to the migration commands. Essentially client data. This is used by lspace::index(atom,things,attribute) to pass in the prefix for table names. check-in: 60293ac0cd user: aku tags: x-redo-stores-indexing-tags | |
| 14:41 | lspace::index::attribute - Fixed db schema handling. Each instance needs its own schema id. Not doing so prevents second instance from setting itself up properly after the first is done, wrongly thinking that it is already done. - Tweaked internal docs (comments), leaked copy-pasta nature of the initial work. - Tweaked internal the atom sub-database, now named `value`. - Tweaked history schema. (Re-)added the `end` field for time intervals, with attendant code maintaining it (Insert, + Truncate). This enables finding the interval for a timestamp without having to `LIMIT` the evaluation, a necessary thing when searching across documents. As part of the suppport, added an index on (end, start). - New methods: documents, get:exact, get:prefix, get:substring - Updated tests: New methods, and fixed a few bogus descriptions. check-in: b4e8210410 user: aku tags: x-redo-stores-indexing-tags | |
| 14:26 | lspace::index::things - Fixed db schema handling. Each instance needs its own schema id. Not doing so prevents second instance from setting itself up properly after the first is done, wrongly thinking that it is already done. check-in: 86975b9580 user: aku tags: x-redo-stores-indexing-tags | |
| 14:22 | lspace::index::atom - Fixed db schema handling. Each instance needs its own schema id. Not doing so prevents second instance from setting itself up properly after the first is done, wrongly thinking that it is already done. - New methods `has`, `iter:map`, `get:map`, `query:map`. - Updated testsuite: New methods, and fixed some bogus descriptions. check-in: e4cbf5b2ec user: aku tags: x-redo-stores-indexing-tags | |
|
2020-07-12
| ||
| 07:22 | New index support package `lspace::index::attribute`. Manages the history of a single attribute, for all documents. Makes use of atoms and things. Main internal complexities are: (1) A representation enabling easy retrieval of the values valid at a specific point in time. (2) Ensuring that the final history is the same regardless of the order the changes are entered into the system. For the representation the atom and things managers were important, as they made it possible to efficiently store the set of values associated with the time intervals started by each change. Both base values and their (semi)sets are stored compressed, without duplication. check-in: 83cdd1d211 user: aku tags: x-redo-stores-indexing-tags | |
| 07:19 | `lspace::index::atom` - Changed API for queries. Dropped the methods handling a pattern spec and dispatching to the specific matchers. Only the specific matchers are left. The user is responsible for pattern syntax and dispatch. - Further added accessor method converting atom id to string. - Corrected schema data stamp. - Updated testsuite. `lspace::index::thing` - Corrected schema data stamp - Updated testsuite. check-in: 74f5de1fb5 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-07-10
| ||
| 06:31 | Added two support database packages: - `lspace::index::atom` - Management of interned strings Efficient substring search via table of suffices. (A substring is a prefix of a suffix). - `lspace::index::things` - Management of semi-sets of things. (Semi because they are ordered. Sets of the same elements in different order are two different semi-sets). check-in: fb1d4afd71 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-05-12
| ||
| 04:27 | Extended all stores with methods to determine the disk space taken up by an archive. Together with the nominal size of the stored documents we can calculate the compression ratio. check-in: befa43a9da user: aku tags: x-redo-stores-indexing-tags | |
|
2020-05-07
| ||
| 04:07 | Helper package importing all known schemata. Implies registration as well. check-in: 57bac04bc9 user: aku tags: x-redo-stores-indexing-tags | |
| 04:04 | New schema packages to handle higher-level mutable documents: Core representing the document + time-ordered change artifacts check-in: 7bb364a0fb user: aku tags: x-redo-stores-indexing-tags | |
| 04:02 | Document core: Allow `store-to` method to clobber destination, on request. check-in: ab855bb0b7 user: aku tags: x-redo-stores-indexing-tags | |
| 04:01 | FS schema: Switch to the new time utilities. check-in: da45546d99 user: aku tags: x-redo-stores-indexing-tags | |
| 04:01 | Small fill-in of tracing. check-in: 8bcec6e9f6 user: aku tags: x-redo-stores-indexing-tags | |
| 04:00 | time utilities for epoch/is8601 conversion (fixed, free form) check-in: 2171b1f814 user: aku tags: x-redo-stores-indexing-tags | |
| 03:59 | db setup: New method `setup_for` enabling dynamic table/.. creation outside of initial setup. check-in: 501f4865b7 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-05-04
| ||
| 07:20 | Extended indexer: content/schema linkage, image geometry png/jpeg. check-in: 06aa5071ca user: aku tags: x-redo-stores-indexing-tags | |
| 06:21 | Moved factory out of store, now handles `index` as well. Updated tests. New `index` pseudo-store to hold indexing information for documents. Partial querying (single operators only, no and/or/not, no attribute retrieval) check-in: c46a71df84 user: aku tags: x-redo-stores-indexing-tags | |
| 06:14 | Extended `schema::core` with attribute retrieval methods (get single, get all) check-in: 9fd1bac462 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-05-01
| ||
| 04:04 | Extended `select schema` to accept the relation operators used for config output. Extended store core and all stores to save factory information (class, arguments) which allows construction of a store just from the information found in its database. Note, this links stores by their paths. The arguments are typed making special-case processing during construction easy. New package implementing a `store factory`. This package uses the new information (see above) to create proper in-memory store instances just from the store path, automatically resolving backend dependencies (paths too). Tests updated and extended. check-in: 4ea9d1a648 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-30
| ||
| 08:22 | New store `archive`. Encapsulates processing pipeline cdc -> zstd -> ccsds -> fspack. check-in: ec511eaf0c user: aku tags: x-redo-stores-indexing-tags | |
| 06:39 | Store `ccsds`: Switched to in-memory documents, no temp files. check-in: 9224baf77c user: aku tags: x-redo-stores-indexing-tags | |
| 06:39 | Store `sqlite`: Added missing method `content-path`. Fixed retrieval of string data. check-in: 0c97beeb92 user: aku tags: x-redo-stores-indexing-tags | |
| 05:36 | First schema documents. Inodes, files, directories, and symlinks. Filesystem scanner. check-in: 913b96de14 user: aku tags: x-redo-stores-indexing-tags | |
| 05:35 | Reworked document core to handle an in-memory rep as well as the old path rep. Convert as needed. Updated and extended test suites. Point is to avoid the filesystem as much as possible during the various processing stages of a document. Even with an SSD a filesystem is likely slower than in-memory. check-in: 007a1c12aa user: aku tags: x-redo-stores-indexing-tags | |
| 05:31 | futil: Fix varname typo in `chan-to` check-in: 3f501323b0 user: aku tags: x-redo-stores-indexing-tags | |
| 05:30 | Fix document leaks in various stores (mostly in case of errors, and one unconditional (cdc index)). check-in: 16877279bb user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-22
| ||
| 03:35 | New packages: Low-level `zstd` (de)compression. And store based on it, analogous to `zip`. Tests added. check-in: 40713c91f6 user: aku tags: x-redo-stores-indexing-tags | |
| 03:33 | Modified store `zip` to use max compression. Further detect when compression expands a document, and pass such through instead. check-in: d4aef58cbe user: aku tags: x-redo-stores-indexing-tags | |
| 03:32 | Test numbering fix. Tests for zlib core (de)compression operation. check-in: a335a3224f user: aku tags: x-redo-stores-indexing-tags | |
| 03:30 | Moved `SIZE` helper to common code. Updated users. Zip store: new test for coming change check-in: 867aa59894 user: aku tags: x-redo-stores-indexing-tags | |
| 03:28 | New store `fs-flat`, flat directory storage. check-in: faf77ab45d user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-21
| ||
| 07:56 | Added support for special `schema` documents providing indexable meta information about data documents. Core manipulation, parsing, detection, registry of known schemata. check-in: c23ad20bde user: aku tags: x-redo-stores-indexing-tags | |
| 07:53 | Added transformer store `ccsds` using the new error correction package to encode incoming documents into error-resistant forms for the backend. General structure that of the `zip` transformer store. check-in: 62aba5ce21 user: aku tags: x-redo-stores-indexing-tags | |
| 07:51 | Added low-level package `lspace::correct` for error correction. Provides CCSDS (255,223) reed-solomon coding based on [email protected]:quiet/libcorrect (srcs embedded). check-in: 4b5fc89b29 user: aku tags: x-redo-stores-indexing-tags | |
| 07:45 | Typo fix. Tweak temp file names for id. Inflate in-memory until zlib fixed in tcl core. check-in: b35a499a62 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-17
| ||
| 02:49 | Extended document API required by schema trialing. check-in: bb458918b2 user: aku tags: x-redo-stores-indexing-tags | |
| 02:47 | Compare the iter implementation options, and choose. Updated the code comments. check-in: 042d745e23 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-09
| ||
| 07:06 | Fix leak of temp documents when zipping. check-in: a17e5cbb6b user: aku tags: x-redo-stores-indexing-tags | |
| 01:30 | New store `cdc` breaking large files into chunks before storing them in a backend store. Uses the `lspace::cdc` package for chunk generation. Required modifications of the `catalog` API (`del` has to return size information), and updates to the existing stores to match that change. Plus testsuite. Small cleanups in test suites, and deeper inspection of tables for zip. check-in: f31587b01d user: aku tags: x-redo-stores-indexing-tags | |
| 01:19 | Added low-level package `lspace::cdc` to perform content defined chunking on channels using Rabin-fingerprinting. Essentially a critcl wrapper of `[email protected]:fd0/rabin-cdc` (BSD licensed). Plus testsuite. check-in: ee41fed7b1 user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-07
| ||
| 18:34 | Added proper identification to temp files used during (un)zipping. check-in: 4d25a2a3a4 user: aku tags: x-redo-stores-indexing-tags | |
| 18:33 | Fixed missing size resolution in config validation causing bogus mismatches. check-in: c2147558cb user: aku tags: x-redo-stores-indexing-tags | |
| 01:17 | Store `section`: Fix typo, remove outdated information. check-in: 47fb2ffa97 user: aku tags: x-redo-stores-indexing-tags | |
| 01:16 | New store `zip`. Transformer. Compresses incoming documents and stores the results into a backend. Vice versa retrieves from backend and decompresses. Assumes to have exclusive access to the backend. New testsuite. check-in: e0811790ac user: aku tags: x-redo-stores-indexing-tags | |
|
2020-04-06
| ||
| 23:45 | Modified `statistics` package to use the config database of the store instead of a separate file. Like `packs`, `sqlite`, `section`, etc. are all doing already. Updated `fs-core` to match, as well as all associated tests. check-in: f7d7929a13 user: aku tags: x-redo-stores-indexing-tags | |
| 20:22 | Converted `fspack`, `fslog` to the new packer, strong simplification of their internals. Required changes in fs API and catalog (access to the unique internal id per uuid, ...). Required updates of `fs`, `fs2`, and `section` to match. Updated tests. check-in: b272cec865 user: aku tags: x-redo-stores-indexing-tags | |