Fatal errors with monotone 0.48 and SQLite 3.7.3

If you encounter errors like this recently

$ mtn pull
mtn: fatal: error: null result in query: …

this might because you’ve updated your SQLite version to 3.7.3 or later. The SQLite guys fixed a bug in their API which we have worked around for ages. [This patch](http://pastebin.ca/1970723) fixes the issue and another minor nitpick. Its unfortunately not part of the recently released 0.48.1 and must therefor be applied manually.

If you can’t recompile, the best thing for now is to downgrade SQLite to 3.7.2 or earlier. You may also want to bug your packager of choice to
include the above fix until we release 0.99 – the next version which includes the above fix as well – in a couple of days.

Sorry for the inconveniences!

monotone 0.48.1 released, please update your servers! (Updated)

We, the monotone team, have just released version 0.48.1 of our version control system.

This release contains an important security bugfix amongst minor other changes: monotone servers that have remote command execution enabled can be crashed if the client sends an empty command string to the server. If you have this feature enabled, we urge you to update your servers to the new version. Distributions should hopefully provide new packages soon as well.

If you cannot update for some reason and do not want to deactivate the feature completely, you can also simply place the following workaround in your server’s monotonerc:

function get_remote_automate_permitted(key, command, opts)
if command[1] == “” then
return false
end

— remaining configuration …

end

The new version can be downloaded at the usual place. Binaries are posted as they come in.

Update: monotone’s NEWS entry doesn’t make it clear which versions are affected by this vulnerability. Since remote command execution became available in 0.46, the affected versions are 0.46, 0.47 and 0.48.

Moving forward

Due to an outage of our old server yesterday, we’ve been forced to switch monotone’s main site, wiki, documentation and the remaining infrastructure earlier than expected to the new server. While not every automatism is in place for the new server, we think that it is actually already usable and decided to not switch back to the old one.

This has one important implication though: The monotone instance running on the new server does no longer allow connections without a path, i.e. pulling like this

$ mtn pull ‘montone.ca’ ‘net.venge.monotone’

will fail. Instead, you have to use a slightly changed URI syntax for monotone clients up to 0.48:

$ mtn pull ‘mtn://monotone.ca/monotone’ \
‘net.venge.monotone’

or, if `mtn` is not in your `/etc/services` and you cannot add

mtn 4691/tcp # Monotone Network Protocol
mtn 4691/udp # Monotone Network Protocol

to it either, use

$ mtn pull ‘monotone://monotone.ca/monotone’ \
‘net.venge.monotone’

because there are a couple of URI parsing bugs in this version which prevent an action with the standard URI syntax unfortunately.

If you don’t want to go through this hassle, you’re also free to push / pull just from the now back-to-life old server:

$ mtn pull ‘old.monotone.ca’ ‘net.venge.monotone’

The **leading server** is however the new one and since both aren’t yet synchronized, it might take a little longer until we see changes which are only pushed to the old one.

Please also note that the new monotone instance separates projects from each other, so if you don’t find a branch at the monotone instance, you might want to try a different one. A list of projects and all of their served branches is available via code.monotone.ca: select a project and click the “Source” tab, you’ll see the branches then listed on the left.

If you are a monotone committer, ensure that you also **do not push unrelated branches** into project databases which do not belong there. (It has proven to be helpful to use different local databases for different projects…) We have no way of enforcing that on the server level as of now, but are working on a solution.

Thanks for reading!

Server keys of the code forge changed

If you previously pulled from code.monotone.ca, you’ll probably get a SERVER IDENTIFICATION warning the next time you sync with it. The problem was that I issued keys with wrong key names previously (which pointed to the temporary DNS) and I had to drop and re-add new keys with the proper DNS name.

Please execute therefor

$ mtn unset known-servers mtn://code.monotone.ca/

e.g. for the monotone project

$ mtn unset known-servers mtn://code.monotone.ca/monotone

on the particular local databases and you’ll be able to pull / sync again.

Sorry for the inconvenience.

Monotone's new project forge officially online

I’m proud to announce that we’re almost done with one big entry in our TODO list for the ongoing server migration: Our indefero forge instance, a web-based project management software, is now set up and online under code.monotone.ca!

The forge contains several monotone-related projects, each with their own bug tracker, wiki, code browser and review module. Here is monotone’s one, and here the one for guitone. We’re planning to add more projects as the need arises and people demand them.

Each of the current and future setups also includes automatic access management for the underlying monotone repository which is served over usher – our proxy. Repositories can simply be accessed by their project name – so if you want to pull from the monotone repository, just trigger

$ mtn pull ‘mtn://code.monotone.ca/monotone?*’

Please note that with the advent of indefero, the main bug tracker of the monotone project is no longer Savannah, but Indefero. All the of the existing open bugs have been imported into monotone’s indefero project, so Savannah’s bug tracker should no longer be used (we’ll leave it online for reference for a little longer, though).

So I said “almost done” – whats missing? A few things actually:

* automatic mirroring of selected repositories with the designated mirrors
* push notifications for IRC / mailing list updates
* a better visual integration into the rest of our setups

While the first two things are on Richard’s TODO list, the last one is on my own schedule, though I want to wait a little longer until indefero’s redesign is through (expected sometime this fall).

And finally, the indefero monotone plugin needed a couple of extensions to core monotone, which I’ll likely blog later about, so stay tuned. If you’re interested in hosting your own monotone-related project on our forge, just drop me a note.

Thanks for reading.

mtn support for indefero got merged

It took us quite a while, but Loic d’Anterroches finally merged my monotone support fork back into indefero. Its expected that the upcoming 1.1 release of indefero is the first release which ships with mtn support.

Most of the things I talked about in the earlier blog post have now been implemented, i.e. you are now able to use remote_stdio to access remote databases or databases behind an usher instance. Furthermore, a forge-level control panel for usher and monotone public key support have been added.

The documentation has also been improved, there is a separate README file which lets you get started and the documentation on the new IDF configuration variables is also exhaustive.

What is kind of missing right now are support scripts for the forge-level setup of monotone. I plan to add these and even more documentation when I set up monotone’s new home server in the next couple of weeks. Yes, you’re reading right, monotone gets a different address, a fancier bug tracker and also a nicer revision browser (thanks to indefero) and if everything goes right, we’ll even offer monotone hosting support for selected projects.

Stay tuned for more updates!

Monotone options rework

As I promised earlier I’ll continue my little series of noteworthy changes and additions in the upcoming monotone release. What I’ll blog about today may sound as if it is merely “syntactic” sugar for the command line end user, but serves a greater purpose when its put in perspective: The introduction of overwritable and negatable options.

Lets start with a simple example: monotone creates a special type of certs if a user explicitely calls the `suspend` command to mark a particular branch – actually its head revision(s) – as end-of-life, or uninteresting. This is useful in case a certain feature branch has been merged back into the main branch and the feature branch name should no longer confuse other people. Since monotone cannot physically delete these no longer needed data due to its distributed nature, the internal revision machinery now simply ignores branches and revisions which have these kind of certs and basically hides them from the user, so they aren’t picked as update candidates when `update` is called and they also no longer show up in the list of branches (`list branches`).

Of course people might still want to see at some point of time what suspended branches are actually available, sometimes maybe to revive a dead development line or for other historical purposes. For this use case monotone has a global option `–ignore-suspend-certs`, which simply deactivates the automatic hiding and all commands behave as if no suspend certs exist at all. So long so good, but there was a nitty gritty use case problem with this (and other similar) functionality:

If a user permanently decided to ignore suspend certs, for example by adding a specific section in his global `get_default_command_options`, he could not easily deactivate or overwrite this setting again via command line. The same applied to long-running processes, like `automate stdio`: once a client triggered the list branches command there with the `–ignore-suspend-certs` option, the client was unable to switch this flag off in case he wanted to query only active branches with the next invocation.

Up until 0.48 only ugly workarounds existed, such as restarting the stdio process, deactivating the loading of hooks or temporarily commenting out the particular section in the lua file, but with the recent merge of the options branch, it became possible. Every revertable, boolean flag gained a corresponding cancel flag, which is most of the time just named `–no-`:

$ mtn ls branches --ignore-suspend-certs --no-ignore-suspend-certs

(Can you guess what this command will now actually do? Little hint: The last occurrence counts…)

But as I mentioned above, this is only half of the truth: Not all cancel flags are just prefixed with a “no-“, in some cases they also look completly different and in some cases they even made us rename the original option, to keep the flow of the UI syntax and to prevent the invention of ugly pseudo options (or do you think `–no-norc` is a good name for the cancel option of `–norc`?).

Before you scream “Oh my god, will I ever get used to this new options?! You broke monotone for me!” let me give you a few words of relief:

  1. For common options which have been changed or removed, there is a new deprecation functionality which points you at the new option syntax.
  2. monotone’s inline command help will not only show you the full syntax of the original option, but also of the cancel option name.
  3. Just as partial command names are completed, option names are now completed as well. If a certain prefix has multiple expansions, all possible options are listed with a short description. (For my little example above, the smallest unique prefix for the long `–ignore-suspend-certs` option is `–ignore-`, as there is also an option named `–ignored` available, for a completely different use case though.)
  4. Finally, if you’re still puzzled by all the new and changed options and general calling syntax and you want a single page which you can just skim / search over for the thing you’re looking for, I’m pleased to tell you that the next monotone version will again have a manual page, but this time its auto-generated from the internal command tree and options. Maybe I’ll tell you a little bit more about this in one of the next blog posts, if not, just try it out with `mtn manpage` as soon as 0.99 hits the streets.

I hope we’ll get a lot of feedback on this for the 0.99 release, as I firmly believe that this overhaul will make the functional and consistent foundations of monotone even stronger.

I also hope that we do not scare too many people away; we did this with purpose for 0.99 and not 1.0 or 1.1, so trust me, we don’t plan to mess around with the UI to this extent again anytime soon 🙂

Thanks for reading so far. Your comments are welcome on irc and via mail.

On monotone selectors

This is the first post in a small series of posts which will show off some of the new functionality you can expect in the next major version of monotone. While there is no fixed release date set for it yet, we plan to release it in fall this year. If you look at the roadmap you see that most things have already been implemented and merged into mainline, so we’re definitely on plan 🙂

Anyways, lets begin this little series with the selector rewrite Tim merged a couple of weeks ago. Selectors are one of the main concepts in monotone to pick revisions other than by their 40 byte long hash id and are therefor very useful to “navigate” between different development lines.

Monotone up until 0.48 knows already many selectors – you can select revisions by tag, by branch, by author, by custom cert values and so on. Selectors can be combined to calculate the intersection between two single sets, like “show me all revisions from author ‘Jon’ on branch ‘my.project'” which would essentially look like this:

$ mtn automate select "a:jon/b:my.project"

The syntax for these selectors is all nice and simple – each selector is prefixed with a unique character and multiple selectors are concatenated with a single slash. While these old-style selectors solved many use cases, some however kept unresolved in the past and users from other DVCS like Darcs had a rather hard time figuring out how to accomplish a certain selection in monotone.

A particular good example is “how can I easily view the changes of a development branch since the last merge point?”. Up until now you either had to figure out manually the revision of the merge point by looking at the output of log or use some scary construct like the following:

$ mtn au common_ancestors $(mtn au select h:main.branch) \
    $(mtn au select h:) | mtn au erase_ancestors -@-

Enter selector functions

Luckily, you don’t have to write these things anymore starting from 0.99 onwards. Give the new selector functions a warm applause!

$ mtn au select "lca(h:main.branch;h:feature.branch)"

In this example “lca” stands for the “least common ancestors” function which takes two arguments, i.e. two other selectors. The syntax is extra short in a workspace where an empty head selector h: defaults to the branch recorded in the workspace options, so if you’re in the feature.branch workspace, just type:

$ mtn au select "lca(h:main.branch;h:)"

Quite convenient, eh? This is not only short, but up to five times faster than the above complex command line. Of course the selector can be directly used in a call to diff or log, like so:

$ mtn diff -r "lca(h:main.branch;h:)"
$ mtn log --to children(lca(h:main.branch;h:))"

But huh, whats that nested children call you ask? Well, the lca function picks the merge point in the _main branch_ and if the revision graph goes around that, log would otherwise happily log more parents (earlier revisions) on the feature branch. The call to children ensures that we pick the merge revision in the feature branch and therefor really stop logging at this revision.

Test drive

There are many more of these selector functions and explaining them all in detail is out of scope here, please have a look at “composite selectors” in the nightly built manual.
And if you want to have an early look at this and play around without having to compile it yourself – at least if you’re on openSUSE or Fedora – just download the binaries from our nightly builds.

New local pre-commit hook in monotone

There was only one hook in monotone until now which could be “reused” to interact with the commit process and validate the changeset that should be committed, the `validate_commit_message` hook. But this was a bit clumsy as it was actually designed to validate the commit message (as the name suggests) and not the changeset, thus the hook was called _after_ the commit message was entered in the editor (or was given with `–message` or `–message-file`).

Now monotone (from 0.99 onwards) gained a new commit hook which is called before the commit message processing takes place, but after the logic validated the changeset and branch to which it should be committed. Its named simply `validate_changes` and takes two parameters, the revision to be committed as full text (parsable in the hook via `parse_basic_io`) as the first and the name of the branch to which the revision should be committed as the second. Just as `validate_commit_message`, it is expected to return a tupel containing a boolean which denotes if the change is valid or not and an optional string which explains the reason if not and which is displayed to the committer afterwards.

With this new installment, it should feel natural e.g. to create a pre-commit hook which ensures that none of the patched or added sources contains Windows line endings:

function validate_changes(revdata, branchname)
  local parsed = parse_basic_io(revdata)
  for _,stanza in ipairs(parsed) do
    if stanza.name == "add_file" or
       stanza.name == "patch" then
      local file = stanza.values[1]
      if not guess_binary_file_contents(file) then
        local fp = assert(io.open(file, "r"))
        local contents = fp:read("*all")
        fp:close()
        if string.find(contents, "\r\n") then
          return false, "CRLF detected"
        end
      end
    end
  end
  return true, ""
end

Unfortunately its not yet possible to call `mtn_automate`, the lua interface to monotone’s automation commands, from hooks like this. Then we could have saved the `read(“*all”)` call and would only have to scan the output of `automate content_diff`, which should be a little faster than doing a full string search in lua for bigger files. We, i.e. the monotone devs, are aware of the problem though and will come up with a solution sooner or later.

I hope this new hook will still be useful for some of you until then.