Configuration Storage
If you are looking for the description of the current configuration-storage mechanisms, skip down to the current storage description.
Context and History
magic-folder
originally was extracted from Tahoe-LAFs where it was a
sub-command / sub-system. Thus, it inherited Tahoe’s method of
configuration. This is a menagerie of various files in a “node
directory” including a tahoe.cfg
INI-style file. There is a
private/
sub-directory which often has “extra-private” information
(like secret keys).
Although a magic-folder is necessarily tied to a Tahoe-LAFS client we don’t need to store our configuration there.
We have decided that a user upgrading from a tahoe magic-folder
style Magic Folder to a stand-alone Magic Folder will run
magic-folder migrate
. This allows us the freedom to store our
configuration in any way we like.
Legacy Configuration and State
In versions of Tahoe-LAFS that had magic-folder enabled, the following configuration could be used:
enabled
(bool) fromtahoe.cfg [magic_folder]
local.directory
(fromtahoe.cfg [magic_folder]
)poll_interval
(fromtahoe.cfg [magic_folder]
)“magic folders” dict from
<node_dir>/private/magic_folders.yaml
. This uses magic-folder “names” as keys with the following valid keys as sub-dicts: -directory
: local path to the “magic folder” -poll_interval
: how often to poll (in seconds) -collective_dircap
: read-cap of the shared directory (list of users) -upload_dircap
: write-cap of our mutable directory -umask
: the umask to use for downloaded files<node_dir>/private/magicfolder_<name>.sqlite
the local sqlite database for tracking state
Related, Tahoe-LAFS has a token in <node_dir>/private/api_auth_token
which is required to access parts of the Web API. This isn’t directly
part of magic-folder’s configuration (however, standalone magic-folder
will also use such a token).
What Needs to be Stored
As we are currently re-designing the storage mechanism, some of this might change. However, based on current understanding the state we WANT to store is:
for each magic-folder (indexed by “name”): - local directory path (the “magic folder”) - an author:
name
private_key (32 bytes, a NaCl signing key)
a write-cap of our mutable directory
some state (see below)
the node-directory of our Tahoe-LAFS client - we may need the
private/api_auth_token
- we will need thenode.url
(at least) - (it MAY be sufficient to only store the API endpoint (the contentsof
node.url
), not node-directory. If it’s feasible to only store the API endpoint, we should do that instead of caring about the node-directory. It would probably still be convenient to refer to a Tahoe client by node-directory (but we can just pull out the Web-API endpoint). If we only store the endpoint and the Tahoe config changes, the user would also have to change their magic-folder config.our own API endpoint (a Twisted server endpoint-string, e.g.
tcp:1234
orunix:/tmp/foo
)
State that we need to store:
a secret token that protects the API (only users who can read the file can access the endpoint)
list of other participants (per magic-folder) (see note)
local cache of remote snapshots (probably per-magic folder)
local copy of snapshots we haven’t yet uploaded (probably per-magic folder) - metadata details of the snapshot - “stash directory” for file-data of the snapshot
NOTE: Currently the “list of other participants” comes from the “collective dircap” but in the Leif Design each user decides whom they wish to pull updates from. So, the “list of other participants” above might be “a readcap of a directory” (and we would determine the actual list at runtime by reading that capability).
UI to Change the Config
Writing configuration files is hard, error-prone and not a great interface. It also makes it hard (or impossible) to change where or how configuration is stored. All configuration shall be changed by an HTTP API. There will also be CLI commands that are thin wrappers of the HTTP API. GUI front-ends could use the command-line or the HTTP API. Nothing except the HTTP API should be touching the actual stored configuration.
(Of course we can’t stop users from editing the config files themselves, but this should be discouraged).
Where and How To Store the Configuration
Somewhat mirroring Tahoe-LAFS’s design, configuration shall all be found in a single directory. This directory can be specified via command-line option. (Should we have a default? Seems sensible, most OSes have a reasonably well-defined place for programs to store configuration data).
All global configuration and state shall be stored in an SQLite3 database. This database will have tables for global configuration and state.
The “configuration directory” will look like this:
confdir
(e.g.~/.config/magic-folder
) -global.sqlite
-api_token
: 32 bytes of binary data -magic_folder.pid
: contains the PID of the currently-running process
The global.sqlite
database will have the following tables:
“version” with a single row containing “1”
“magic_folders” - columns: -
name
is a unique unicode string the user provides to identify this folder -location
is a local path for that folder’s configuration“config” - columns: -
api_endpoint
is a Twisted server string description -tahoe_node_directory
is the node-directory of our Tahoe-LAFSclient
Note: Ideally, we’d only need the Web-API URI however that can be configured in tahoe to be randomly-assigned on startup and so we should read the URI from
node.url
. There may be other configuration we’ve neglected (since magic-folder used to be inside Tahoe) so this also allows us to fix that later too.
Each “magic folder” location is a directory containing the stash-directory and state for that magic-folder. It will look like this:
<magic folder location>/
-README
: a file containing information about magic-folder -state.sqlite
: the state database -api_client_endpoint
: a Twisted endpoint-string describing how to connect to the HTTP API -api_token
: the token required to authenticate to the HTTP API -folder-<hash>
: one subdirectory for each magic-folder in the daemonstash/
: the stash-directory containing LocalSnapshot contentstate.sqlite
: state database for this magic-folder
The state.sqlite
for each magic-folder shall contain the following
tables:
“version” (will always contain 1 row) - column: - “version” is an int, currently
1
version-dependent additional tables
See the SQL statements creating additional tables in config.py
.
Each version may add or change tables.
The configuration should only be read and changed via the `http-api`_ and not via this database.