Batch-remove empty lines at the end of many Confluence pages

In a customer project we’ve decided to collaboratively write a bigger bunch of documentation in Atlassians Confluence and export that with Scroll Office, a third-party Confluence plugin, into Word.

That worked fine so far, but soon we figured that we’ve been kind of sloppy with empty lines at the end of each page, which were obviously taken over into the final document. So instead of going over each and every page and remove the empty lines there, I thought it might be easier to directly do this on the database, in our case MySQL.

The query was quickly developed, but then I realized that MySQL had no PREG_REPLACE function built-in, so I needed to install a UDF, a user-defined function first. Luckily, this UDF worked out of the box and so the query could be finalized:

UPDATE BODYCONTENT
JOIN CONTENT ON CONTENT.CONTENTID=BODYCONTENT.CONTENTID
AND CONTENTTYPE LIKE "PAGE" AND PREVVER IS NULL
SET BODY=PREG_REPLACE("/(

<.p>)+$/", "", BODY)
WHERE BODY LIKE "%

";

This query updates all current pages (no old versions) from all spaces that end with at least one empty line – this is Confluence’s internal markup for that – and removes all of these empty lines from all matches pages.

This was tested with MySQL 5.5.35, lib_mysqludf_preg 1.2-rc2 and Confluence 5.4.2.

I don’t need to mention that it is – of course – highly recommended that you backup your database before you execute this query on your server, right?

Runtime-replace implementations with Roboguice in functional tests

At work we’re heavily depending on Unit and Functional Testing for our current Android application. For Unit testing we’ve set up a pure Java-based project that runs on [Robolectric](http://pivotal.github.com/robolectric/) to provide a functional Android environment and we also added [Mockito](http://code.google.com/p/mockito/) to the mix to ease some code paths with spied-on or completely mocked dependencies. [Moritz Post wrote a comprehensive article](http://eclipsesource.com/blogs/2012/09/25/advanced-android-testing-with-roboguice-and-robolectric/) how to setup this – if you have some time, this is really worth a read.

Now our functional tests are based on what the Android SDK offers us – just that we’re using [Robotium](http://code.google.com/p/robotium/) as a nice wrapper around the raw instrumentation API – and until recently I thought it would not be possible to screw around much with an unaltered, but instrumented application on runtime. But while I was reading through [the Android Testing Fundamentals](http://developer.android.com/tools/testing/testing_android.html) I stumbled upon one interesting piece:

With Android instrumentation […] you can invoke callback methods in your test code. […] Also, instrumentation can load both a test package and the application under test into the same process. Since the application components and their tests are in the same process, the tests can invoke methods in the components, and modify and examine fields in the components.

Hrm… couldn’t that be used to just mock out the implementation of this one REST service our application uses? Yes, it could! Given the following implementation

@ContextSingleton
public class RequestManager {
    ...
    public <I, O> O run(Request<I, O> request) throws Exception {
        ...
    }
}

(where the `Request` object basically encapsulates the needed request data and Input / Output type information)

it was easy to create a custom implementation that would return predefined answers:

public class MockedRequestManager extends RequestManager {
    private Map<Request, Object> responses = new HashMap<Request, Object>();
    ...
    public <I, O> O run(Request<I, O> request) throws Exception {
        Object response = findResponseFor(request);
        if (response instanceof Exception) {
            throw (Exception) response;
        }
        return (O) response;
    }
    ...
    public void addResponse(Request request, Object response) {
        responses.put(request, response);
    }
}

Now that this was in place, the only missing piece was to inject this implementation instead of the original implementation. For that I created a new base test class and overwrote the `setUp()` and `tearDown()` methods like this:

public class MockedRequestTestBase extends ActivityInstrumentationTestCase2 {
    protected Solo solo;
    protected MockedRequestManager mockedRequestManager = new MockedRequestManager();
    ...
    private class MockedRequestManagerModule extends AbstractModule {
        @Override
        protected void configure() {
            bind(RequestManager.class).toInstance(mockedRequestManager);
        }
    }
    ...
    public MockedRequestTest() {
        super(MyActivity.class);
    }
    ...
    @Override
    protected void setUp() throws Exception {
        super.setUp();
        Application app = (Application) getInstrumentation()
            .getTargetContext().getApplicationContext();
        RoboGuice.setBaseApplicationInjector(
            app, RoboGuice.DEFAULT_STAGE,
            Modules.override(RoboGuice.newDefaultRoboModule(app))
                .with(new MockedRequestManagerModule()));
        solo = new Solo(getInstrumentation(), getActivity());
    }
    ...
    @Override
    protected void tearDown() throws Exception {
        super.tearDown();
        Roboguice.util.reset();
    }
}

It is important to note here that the module overriding has to happen _before_ `getActivity()` is called, because this starts up the application and will initialize the default implementations as they’re needed / lazily loaded by RoboGuice. Since we explicitely create a specific implementation of the `RequestManager` class before, the application code will skip the initialization of the actual implementation and will use our mocked version.

Now its time to actually write a test:

public class TestFileNotFoundException extends MockedRequestTestBase {
    public void testFileNotFoundMessage()
    {
        Request request = new FooRequest();
        mockedRequestManager.addResponse(
            request, 
            new FileNotFoundException("The resource /foo/1 was not found")
        );
        solo.clickOnView("request first foo");
        assertTrue(solo.waitForText("The resource /foo/1 was not found"));
    }
}

Thats it. Now one could probably also add Mockito to the mix, by injecting a spied / completely mocked version of the original RequestManager, but I’ll leave that as an exercise for the reader…

Have fun!

Debugging with MacPorts PHP binaries and Eclipse PDT 3.0

You know the times, when things should really go fast and easy, but you fall from one nightmare into another? Tonight was such a night… but lets start from the beginning.

To debug PHP you usually install the excellent [XDebug](http://xdebug.org/) and so did I with the port command `sudo port install php5-xdebug`. After that `php -v` greeted me friendly on the command line already:

PHP 5.3.8 (cli) (built: Sep 22 2011 11:42:56)
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
with Xdebug v2.1.1, Copyright (c) 2002-2011, by Derick Rethans

Eclipse Indigo and Eclipse PDT 3 was already installed, so I thought it should be easy to set up the XDebug debugging option in Eclipse. Under “PHP > PHP Executables” I therefor selected `/opt/local/bin/php` as my CLI version and selected “xdebug” as debugging option.

A first test however showed me that the execution of a test script did not load any module into the PHP interpreter beforehand (for reasons I could only guess, because Eclipse error log kept quite). Looking at the output of `phpinfo()` from my test script and `php -i` from command line showed me the difference: The PHP option “Scan this dir for additional .ini files” was empty when PHP ran inside Eclipse, but was properly set when PHP ran from command line (or in an Apache context).

Asking aunt Google brought up [this issue](https://bugs.php.net/bug.php?id=45114) that shed some light into my darkness: The directory where additional modules reside is configured as a compile time option in PHP and defaults to `/opt/local/var/db/php5` on MacPorts and exactly this can be overridden by either calling PHP with `-n -c` options or by setting the `PHP_INI_SCAN_DIR` environment variable.

Having no access to the actual PHP call from inside Eclipse I tried to go down the environment route, but that did not lead to any success. While the variable was recognized as it should on the normal command line (e.g. `PHP_INI_SCAN_DIR= php -i` disabled the load of additional modules), in Eclipse’ run configuration dialog, in the section environment variables, this was not recognized at all. I tried a little harder and configured the variable inside `~/.MacOSX/environment.plist`, logged out and in again, restarted Eclipse obviously, but had no luck either.

The only viable solution I came up with was to place all the single `extension=` and `zend_extension=` entries directly into my `php.ini` and disable the individual `module.ini` files altogether. At least I can now run and debug properly, but this solution is of course far from being ideal – as soon as I add a new PHP module or want to remove an existing, I have to remember to edit the `php.ini` myself.

By the way, I also tried to use Zend’s debugger (and PDT plugin) as an alternative. [While somebody else](http://www.85qm.de/archives/727-Zend-Debugger-for-PHP-5.3.html) already ranted about that the Zend guys have been unable to provide the Zend Debugger for PHP 5.3 as a standalone download (which hasn’t changed to date), PHP 5.2 debugging worked nicely with [the old Zend PDT plugin](http://www.zend.com/en/community/pdt).

Of course, none of my needed PHP modules were loaded and I really needed PHP 5.3 support, so I had to follow the same route the other guy did and downloaded all of the ZendServer glory (a 137MB download, yay) just to get the right `ZendDebugger.so`. After extracting the `.pax.gz` archive from the installer package I quickly found it underneath `usr/local/zend/lib/debugger/php-5.3.x/`, copied it to my extension directory and added an ini file to load that one instead, just to find out shortly afterwards that the Zend binary was `i386` only and MacPorts of course compiled everything nicely as `x86_64` – php was of course unable to load such a module.

Well, the moral of the story is – go for Xdebug and don’t loose the track. And, let us all hope that Eclipse PDT is developed further, so the remaining glitches like the one above are fixed.

Exception chaining in Java

If you catch and rethrow exceptions in Java, you probably know about exception chaining already: You simply give the exception you “wrap” as second argument to your Exception like this

try { … }
catch (Exception e) {
throw new CustomException(“something went wrong”, e);
}

and if you look at the stack trace of the newly thrown exception, the original one is listed as “Caused by:”. Now today I had the rather “usual” use case of cleanup up a failing action and the cleanup itself was able to throw as well. So I had two causing exceptions and I wanted to conserve both of them, including their complete cause chain, in a new exception. Consider the following example:

try { … }
catch (Exception e1) {
try { … }
catch (Exception e2) {
// how to transport e1 and e2 in a new exception here?!
}
throw e1;
}

My idea here was to somehow tack the exception chain of `e1` onto the exception chain of `e2`, but Java offered no solution for this. So I hunted for my own one:

public static class ChainedException extends Exception {
public ChainedException(String msg, Throwable cause) {
super(msg, cause);
}
public void appendRootCause(Throwable cause) {
Throwable parent = this;
while (parent.getCause() != null) {
parent = parent.getCause();
}
parent.initCause(cause);
}
}

Now I only had to base the exceptions I actually want to chain on `ChainedException` and was able to do this (in fact I based all of them on this class):

try { … }
catch (ChainedException e1) {
try { … }
catch (ChainedException e2) {
e2.appendRootCause(e1);
throw new ChainedException(“cleanup failed”, e2);
}
throw e1;
}

Try it out yourself – you’ll see the trace of `e1` at the bottom of the cause chain of `e2`. Quite nice, eh?

Access the Android menu in VirtualBox on a Mac host

If you’re desperately trying to get the `Menu` button in an [Android x86](http://www.android-x86.org) installation working under [VirtualBox](http://www.virtualbox.org) on a Mac OS X host – whose keyboard of course doesn’t have this “context” / “menu” key Windows keyboards have on the right – you might find the touch-only-device mode in Android x86 handy:

1. Click on the clock in the status bar to enable / disable this mode altogether
2. A swipe from the left to the right emulates the `Menu` button function
3. A swipe from right to left emulates the `Back` button function
4. Simply clicking on the status bar brings you to the `Home` screen

([Source](http://www.android-x86.org/documents/touch-only-device-howto))

ACLs on Mac OS X

This is a short follow-up of an earlier post where I explained how one can set ACLs on FreeBSD. Today now I’ll do the same on another BSD variant, namely Mac OS X, and guess what, the guys from Cupertino implemented ACL management in a completely different manner.

The first thing to notice is that starting with Mac OS X 10.6 (Snow Leopard) ACLs are always enabled and cannot be disabled as in earlier versions. All ACL commands are baked into the `chmod` command and parsed from a string you give it with the ‘+a’ option. The basic syntax here is

$ chmod (+|-|=)a#? ‘ (allow|deny)

Since there is no `setfacl` on Mac OS X, there is no `getfacl` either, so ACLs are instead queried by the special option `-e` of `ls`:

$ ls -le .
[…]
-rw-r–r–+ 1 john users 175 5 Jun 00:23 foo
0: user:dave allow write

Permissions include the usual `read`, `write`, `delete`, `add_file`, and `add_subdirectory` as well as more exotic ones like `{read,write}extattr`, `{read,write}writesecurity` and `chown`. (Read up `chmod`’s man page what these are for.)

There are, however, two more important ones to notice, namely `file_inherit` and `directory_inherit`. These two let you spread your permissions nicely to sub objects and thus let you for example set up a directory, in which a pool of users is allowed to access, modify and delete each other’s files:

$ chmod +a ‘john allow read,write,delete,add_file,add_subdirectory,file_inherit,directory_inherit’ /data
$ chmod +a ‘dave allow read,write,delete,add_file,add_subdirectory,file_inherit,directory_inherit’ /data

The above example gives `john` and `dave` inherited read, write and delete permissions to all file objects underneath `/data`.

Since ACLs are executed in order, they can also be set in an ordered manner. `chmod` has the `+a#` option for that, where `#` is the position into which the ACL should be added. Similarily, existing ACLs can be edited with `=a#`, where again `#` marks the position of the ACL to edit, and deleted with `-a#`.

Finally, if one wants to get rid of all ACLs of a specific node, `chmod -N ` will do the job.

Thats it, have fun playing with ACLs on Mac OS X!

SQLite to MySQL

Migrating a database dump from SQLite to MySQL can be somewhat of a hassle. After various trials I came up with the following that worked for me quite well:

echo .dump | sqlite3 mydb.db |\
egrep -v “\b(BEGIN TRANSACTION|COMMIT|PRAGMA|sqlite_sequence)\b” |\
perl -pe ‘s/^([^’\””]*)”([^”]+)”/\1`\2`/’ |\
perl -pe ‘s/\bautoincrement\b/auto_increment/’ |\
mysql -uroot mydb

The most important line is probably line 3: SQLite encapsulates all table and column names in double quotes and this is not understood by MySQL, so we’re replacing them with backticks. One must be careful here, though, because data INSERTs might also contain double quotes and to avoid having these wrongly replaced as well we only replace them until we encounter the first single quote (these are used in SQLite to encapsulate strings).

Line 3 might also have to be called several times in a row in case multi-column indexes are part of the schema as well, as it will only replace the very first, but no subsequent matches. (I couldn’t get zero-width lookbehind to work properly to remedy this problem; help is welcome.)

Search and replace multiple lines across many files

sed is usually my favourite tool to search and replace things from the command line, but sometimes Perl’s regexes are far more convenient to use. Recently I found out another reason why Perls -pi -e is superior over plain sed: when you want to change multiple lines in a document!

Imagine you have hundreds of source code files where somebody once had the great idea to add a ___version___ property into each class:

public class Foo
{
    private static final String ___version___ = "$Version:$";
    
    // other stuff
}

With Perl the line in question is easy to remove:

$ for file in $(find . -name "*.java"); do \
   cp $file $file.bkp; perl -pi -e \
      "s/\s*public.+___version___.+\n//g" \
   < $file.bkp > $file; rm $file.bkp; done

But, there is one problem: Perl processes each line of the file separately when it slurps in the file, which results in unwanted empty lines:

public class Foo
{
    
    // other stuff
}

Then I stumbled upon this article and the solution is to set a special input separator to let Perl slurp in the file as a whole:

$ for file in $(find . -name "*.java"); do \
   cp $file $file.bkp; perl -p0777i -e \
     "s/\s*public.+___version___.+\n(\s*\n)*/\n/g" \
   < $file.bkp > $file; rm $file.bkp; done

and voila, we get what we want:

public class Foo
{
    // other stuff
}

Digging a little deeper what -0777 actually means leads us to perlrun(1):

The special value 00 will cause Perl to slurp files in paragraph mode. The value 0777 will cause Perl to slurp files whole because there is no legal byte with that value.

Another day saved – thanks to Perl!

And while we’re at it, have a look at Rakudo Star, the best Perl 6 compiler which was released just recently. Perl 6 is in my humble opinion one of the well-designed languages I’ve came across so far, so if you find some time, go over and read the last christmas special, its really worth it!

Tip: Enforce specific key usage for a single SSH connection

In case you have to access a very restricted SSH server which only accepts a single key (ie. the one which is set up in `~/.ssh/authorized_keys`) and otherwise fails, its the easiest to set the specific key in your local `~/.ssh/config` file as follows:

Host very.secure.server
    IdentityFile ~/.ssh/id_dsa
    IdentitiesOnly true

The second entry, `IdentitiesOnly`, forces SSH to only use known identity files and not look for more available identities f.e. from a running `ssh_agent` instance (which are always tried in first instance as it seems).

Tip: Logging with Symfony >= 1.2

Imagine you have a business method in your model which needs to be accessed by two environments: once from a symfony task and once from the web. So far so good, now what if this business method should be able to log contents somewhere visibly, in case of the command line task to console and to a file and in case of the web application to the default logging mechanisms used there?

Getting the logger in web context is easy, all you have to do is

$logger = sfContext::getInstance()->getLogger();

but its a little harder to do for the command line task.

By default no symfony context is created for a command line task and even if it is created, the above call returns an instance of sfNoLogger. Logging in command applications happens through the sfTask::logSection() method, which basically throws an event at the created dispatcher in SYMFONYDIR/lib/command/cli.php. There you can also see that an instance of sfCommandLogger is created, but there is no way to get your fingers at this instance, because its purely local.

So what can we do? Parametricizing the business method with the sfTask instance and using the logSection() is obviously no solution, because this would break in web context where no such sfTask instance exists…

My solution was a bit more straight forward – I simply decided to not use the task-supplied logging schema at all, but created my own logger like this:

$dispatcher = new sfEventDispatcher();
$logger = new sfAggregateLogger($dispatcher);
$logger->addLogger(new sfCommandLogger($dispatcher));
// optionally add another file logger
if ($logToFile)
{
    $logger->addLogger(
        new sfFileLogger($this->dispatcher, ...)
    );
}

Hope this helps somebody.