Uri.Builder.build() works quite well with normal URLs, but it fails with port number support.
The easiest way that I discovered to make it support port numbers was to make it parse a given URL first then work with it.
Uri.Builder b = Uri.parse("http://www.yoursite.com:12345").buildUpon();
b.path("/path/to/something/");
b.appendQueryParameter("arg1", String.valueOf(42));
if (username != "") {
b.appendQueryParameter("username", username);
}
String url = b.build().toString();
And there you have it, a customisable URL with support for port number and queries.