Predicting a Mobile Phone Revolution
Mobiles are in a bit of a crisis. There is huge demand for new phones even though they offer much the same features as the previous phones. They have to really, because behind the scenes they all work on standards. So demand is driven by innovation in the industrial design of the handsets and the user interface software, and to a lesser extent by the additional services layered on top of them. Consumer excitement can be replaced by disappointment though, when they discover that the phone isn’t life changing, isn’t really as wonderful as they thought it would be, or a newer model appears.
The industry has become hugely competitive and has seen former big names fall by the wayside as they are not able to keep up with the pace and quality of their competitors. Also, margins have been eroded by the competition and the need to bring out new models more quickly. Android seems unstoppable with pretty much every major manufacturer considering using it as part of their offering. But the technology of the network is really at a standstill, in the UK at least. Other countries are looking beyond 3G whereas we are still struggling to provide 3G coverage.
From the network operators’ point of view, the market is reaching saturation point – everyone has a mobile. It is more important than ever to keep existing customers and tie them in for as long as possible, to get as much money from them as possible. Recently most of the main networks have been more open about caps on ‘unlimited’ packages. Simple features like tethering have been made intentionally awkward. The fear must be that the new glitzy phones will actually be used for video calls and such… but the network hasn’t been upgraded at the same rate as the handsets, and might not cope too well.
In the good old days, it worked well because there was a longer time between new models and the operators were in a better position to subsidise phones to consumers. Consumers were happy to pay for the novelty and luxury of a mobile phone. Operators could justify investment in the network to enable growth. But new phones coming out every six months and 18 or 24 month contracts don’t add up. Phones that can’t be used because of patchy coverage or stingy tariffs are bad news. The business models aren’t working to promote the technology, and if the technology can’t be realised, consumers will sooner or later realise that it’s just a telephone, and an expensive one at that.
Innocent Smoothies
I love the product and marketing but I haven’t managed to collect an ‘x’ from their range of fridge magnets. There are some great designs though…
Moving from Textpattern to WordPress – part 2
I wrote recently about moving this blog from Textpattern to WordPress. I promised more details of the Perl script I used to transfer the content. I saw I was getting more than a handful of hits about the topic so decided I would complete my writeup. Some familiarity with Perl, Textpattern and WordPress is assumed from here on. If you want to use the script yourself you’d better be willing to edit it a bit 
What follows is a description of the task and a breakdown of the Perl script itself. The two platforms have broadly similar tables for posts and comments. There are some differences in the way categories are stored. I wasn’t merging data – so I could map the post IDs from one database to another in a straightforward way. I wasn’t too bothered about recreating the old blog permalinks using the new platform (instead I chose to redirect them with a rewrite rule). Running my own server, I had shell access and was able to run Perl scripts and mysql backup commands directly on the server. I chose to use DBI to access the database from Perl. Some of my posts had custom markup in them for images, and I wanted to convert that markup to the new custom markup (to use Highslide). I used HTML::Parser for that. Anyway, here’s the code…
# tp2wp.pl
#
# convert textpattern stories and comments to wordpress
my $dbname = 'bc';
my $dbpass = 'textpattern-password-here';
my $dbuser = 'textpattern-db-user';
my $tprefix = 'tp';
use strict;
use DBI;
use CGI; # for escapeHTML
use Data::Dumper;
I defined a few variables to let me log in to the textpattern database. I used a prefix for my tables so added a variable for that (the prefix is optional but it seemed like a good idea to add one when I configured Textpattern). I listed the modules I would use.
my $thandle = DBI->connect("dbi:mysql:wordpress:localhost", 'wp-mysql-user', 'wp-mysql-pass');
# handle is global in this script
my $handle = DBI->connect("dbi:mysql:$dbname:localhost", $dbuser, $dbpass);
I created variables thandle and handle that would be used to access the target database (WordPress) and the source (Textpattern) respectively.
# txp_discuss - comments
sub comments
{
return generic_read('txp_discuss', 'discussid', qw(discussid parentid name email web ip posted message visible));
}
# txp_image - images
sub images
{
return generic_read('txp_image', 'id', qw(id name category ext w h alt caption date author thumbnail));
}
# textpattern - stories
sub stories
{
return generic_read('textpattern', 'ID', qw(ID Posted AuthorID LastMod LastModID Title Title_html Body Body_html Excerpt Excerpt_html Image Category1 Category2 Annotate AnnotateInvite comments_count Status textile_body textile_excerpt Section override_form Keywords url_title custom_1 custom_2 custom_3 custom_4 custom_5 custom_6 custom_7 custom_8 custom_9 custom_10 uid feed_time));
}
These three subs know all about the structure of the Textpattern tables for comments (txp_discuss), images (txp_image) and stories (textpattern). The actual work of reading the content from the database is done by the sub generic_read described later. In each case the table has a ‘key’ (the second argument) and the returned value is a Perl hash representing the contents of that table. This is where it starts to get interesting…
{
my ($row) = @_;
# always this author
$row->{fixed_approved} = 1;
$row->{fixed_parent} = 0;
# just skip non approved comments (visible!=1)
if ($row->{visible} != 1){
print "Skipping non visible comment '$row->{message}'\n";
}
return ($row->{visible} == 1);
}
sub story_preprocess_and_validate
{
my ($row) = @_;
# always this author
$row->{fixed_author} = 3;
$row->{post_status} = 'publish'; # always posted
$row->{post_name} = $row->{Title};
$row->{post_name} =~ s/[^a-zA-Z0-9]+/-/g;
$row->{post_name} = lc($row->{post_name});
$row->{post_type} = 'post';
$row->{post_parent} = 0;
$row->{comment_count} = 0;
$row->{guid} = 'http://myurl.net/path/?p='.$row->{ID};
# just skip draft posts (status==1)
if ($row->{Status} == 1){
print "Skipping draft article '$row->{Title}'\n";
}
# convert image code to highslide
my $p = new HTMLimgxform;
$row->{post_content} = $p->parse_string($row->{Body_html});
return ($row->{Status} != 1);
}
To convert content from one format to the other I decided to first read in the whole of the source material, then loop through each entry in turn. For each one I would call a sub to ‘preprocess and validate’ the source material. If that returned true then I would call a second sub to map the source material (as modified by the pre-processing) into the new format as needed by the WordPress database. So above are the preprocessing and validation subs for comments and stories. For some fields in the target database there was no equivalent in the source, so these values had to be calculated in the pre-processing. As an example all the Textpattern comments lacked threading (there was no parent ID) so I set the parent ID to 0. I also skipped all invisible comments – ie. unapproved spam. For stories I fixed the author to a hardcoded value (one I had set up to identify posts as automatically imported). The post status was fixed to published. The post name was a munged version of the title. Post type and parent were hardcoded. A guid was constructed. I skipped any unfinished draft articles.
At the end of the story preprocessing I use a custom HTML parser to transform the content and rewrite any image references – more on this later. It’s done by creating a parser object and feeding the old text to it. The parser returns the text with any modifications needed. With a modified parser this could cope with any arbitrarily complex transformations of content.
Having defined the code that does the transformation, we still need some Perl to put it all together and sequence things. Here is that code…
my $s = stories();
my $c = comments();
our $i = images();
for my $row (values(%$s))
{
if (story_preprocess_and_validate($row)){
# transform takes the row hash, the destination table
# and a hash mapping WP column name to TP column name
transform_row($row, 'wp_posts',
{
ID => 'ID',
post_author => 'fixed_author',
post_date => 'Posted',
post_date_gmt => 'Posted',
post_modified => 'Posted',
post_modified_gmt => 'Posted',
post_content => 'post_content',
post_title => 'Title',
post_excerpt => 'Excerpt_html',
post_status => 'post_status',
post_name => 'post_name',
post_type => 'post_type',
post_parent => 'post_parent',
guid => 'guid',
comment_count => 'comment_count',
});
categorize($row->{ID}, $row->{Category1});
categorize($row->{ID}, $row->{Category2});
}
}
for my $row (values(%$c))
{
if (comment_preprocess_and_validate($row)){
transform_row($row, 'wp_comments',
{
comment_ID => 'discuss_id',
comment_post_ID => 'parentid',
comment_author => 'name',
comment_author_email => 'email',
comment_author_url => 'web',
comment_author_IP => 'ip',
comment_date => 'posted',
comment_date_gmt => 'posted',
comment_content => 'message',
comment_approved => 'fixed_approved',
comment_parent => 'fixed_parent',
});
# update comment count
my $cqry = 'UPDATE wp_posts set comment_count = comment_count + 1 where ID = ' . $thandle->quote($row->{parentid});
my $cres = $thandle->prepare($cqry);
$cres->execute();
$cres->finish();
}
}
$handle->disconnect();
$thandle->disconnect();
# end of source specific code
First, the stories, comments and image metadata are read into Perl hashes with $s, $c and $i being handy references to them. There’s a loop for stories first where each story is preprocessed and checked. If it’s ‘OK’ the story is entered into the WordPress database using the ‘transform_row’ sub. That takes the source hash, destination table name, and a hash that maps source table column names into destination table column names. Note that I had to do category handling in a separate way, as there isn’t a one-to-one mapping. The source database had two separate category fields whereas the target had a separate system of tables for categories. This approach is minimal and relies on me setting up the categories in WordPress manually beforehand. Similarly for comment processing, the story comment count is updated after processing each comment to make sure the stories have the right comment count. Finally, I disconnect from both databases.
sub query
{
my ($dbh, $qry) = @_;
my $res = $dbh->prepare($qry);
$res->execute();
$res->finish();
}
This was my ‘generic query’ – there isn’t much to it because the DBI API is already clean.
sub generic_read
{
my ($table, $key, @fieldnames) = @_;
my $tbl;
$table = $tprefix . $table;
my $qry = "SELECT * FROM $table";
my $res = $handle->prepare($qry);
$res->execute();
# prepare a hash to receive each row of the table
my $rh;
$res->bind_columns(map {\$rh->{$_}} @fieldnames);
# fetch rows
while ($res->fetch())
{
# pack row into 'tbl' hash
$tbl->{$rh->{$key}} = {map {$_,$rh->{$_}} @fieldnames};
}
# clean up and return the table in Perl format
$res->finish();
return $tbl;
}
This is slightly more complex, and reads a whole table into a Perl hash. There is probably a neater way to achieve this but I didn’t have time to research it.
This was where I started coding – the basic idea was to enable a data driven approach to transforming one table into a different format.
sub categorize
{
my ($id, $cat) = @_;
my $val = {'Signs-of-the-Times' => 7,
'Down' => 6,
'UP' => 5,
'Nothing-honest' => 4,
'Domestic-Entropy' => 3,
}->{$cat}||return;
my $qry = 'INSERT into wp_term_relationships (object_id, term_taxonomy_id) values(' . $thandle->quote($id) . ', ' . $thandle->quote($val) .')';
query($thandle, $qry);
# need to update the usage counts in the wp_term_taxonomy table
my $cqry = 'UPDATE wp_term_taxonomy set count = count + 1 where term_id = ' . $thandle->quote($val);
query($thandle, $cqry);
}
This doesn’t look very neat, but it’s functional. I map a textually named category into a numeric one. Anyone reusing this will have to insert their categories into the WordPress database by hand, note the IDs and create a map in $val in the same way I have.
Finally, the HTML parser that transforms my TextPattern markup for images into my Highslide markup. The way I use HTML::Parser is by subclassing it. I declare a package ‘HTMLimgxform’ in the same file. Perl lets you do this although it might be neater to put it in a separate file. I have overridden the base class methods as you are supposed to, but some of them are not used (declaration and comment) so there isn’t much of interest there. The gist of the parser is to copy the input to the output, except for what it recognises as standalone images in the old format, which are rewritten in the new format; and image galleries in the old format are rewritten as image galleries in the new format. There is some error handling and that bails out with an informative message.
package HTMLimgxform;
use base "HTML::Parser";
# contain the processed HTML
my $result;
my $thumbing = 0; # state used in image processing
my $skipping_span = 0; # state used in caption processing
# entry point - process a string and return the transformed output
sub parse_string
{
my ($self, $string) = @_;
$result = '';
$self->parse($string);
$self->eof();
return $result;
}
That was the module globals used to maintain the state of the parser, and the interface to the module.
# plain text, copy it to the output, unless
# we are skipping a span or processing thumbnails
sub text
{
my($self, $text) = @_;
$result .= $text
unless ($skipping_span or $thumbing);
}
# don't expect any of these
sub declaration
{
my($self, $decl) = @_;
print "decl: ",Dumper($decl);
exit 0;
}
# don't expect any of these
sub comment
{
my($self, $comment) = @_;
print "comment: ",Dumper($comment);
exit 0;
}
Simple behaviour for the text and nothing at all for declarations and comments.
{
my($self, $tag, $attr, $attrseq, $origtext) = @_;
# $attr is reference to a HASH, $attrseq is reference to an ARRAY
if ($tag eq 'img'){
# in the source material there are two distinct uses.
# 1. a standalone image
# 2. a series of thumbnails with javascript followed by a img
# we need to keep some context
if (exists($attr->{onmouseover})){
# thumb with JS
# start gallery if needed
if (!$thumbing)
{
$result .= qq(\n\n<div class="highslide-gallery">\n);
$thumbing = 1;
}
# get info from tag
my $caption = $attr->{alt} || die "Couldn't find alt text for caption in $origtext, aborting";
$caption = CGI::escapeHTML($caption);
my $src = $attr->{src} || die "Couldn't find img src for image in $origtext, aborting";
$src =~ s!^.*/!!;
$result .= qq(<highslide image="$src" altdesc="$caption" />\n);
}else{
# standalone img
# if we have seen thumbs then we need to 'end' the gallery that we have been writing; otherwise it's really a standalone image
if ($thumbing)
{
# end gallery
$result .= qq(</div>\n\n);
$thumbing = 0;
}else{
$result .= $origtext;
}
}
}elsif($tag eq 'txp:image'){
use Data::Dumper;
print Dumper($tag, $attr, $attrseq, $origtext);
my $imgid = $attr->{id};
my $imgrow = $main::i->{$imgid};
print "$origtext\n",Dumper($imgrow);
my $ext = $imgrow->{ext};
my $img = $imgid.$ext;
my $desc = $imgrow->{alt} || $imgrow->{caption} || $imgrow->{name} || '';
$result .= qq(<highslide image="$img" altdesc="$desc"/>);
}else{
# if it's a span with an id matching '^cap' then lose it -
# it's an old caption
if ($tag eq 'span' && exists($attr->{id}) && ($attr->{id} =~ m/^cap/)){
# skip
$skipping_span = 1;
}else{
# not interesting, just copy it
# unless it's stray tags in between thumbs...
$result .= $origtext
unless ($skipping_span or $thumbing);
}
}
}
This is called for the start of an HTML tag and it processes only ‘img’ and ‘txp:image’ tags converting them into ‘highslide’ tags.
sub end
{
my($self, $tag, $origtext) = @_;
if ($tag eq 'span' && $skipping_span){
$skipping_span = 0;
}
else
{
$result .= $origtext;
}
}
That is called for an end tag and maintains the parser state ’skipping_span’ or makes sure the end tag is copied to the output.
That’s all there is to the parser. It saved me a good amount of editing although I still had to make some more specialised changes by hand. In case it helps anyone the Perl script is available here: tp2wp.pl – reuse it however you like, but be sure to back up your databases before letting any scripts loose on them…
Unfinished business
I was searching for some Gnome 3 theme previews online after trying out, and liking Gnome Shell. I didn’t have much success but I did happen across art.gnome.org where I remembered I’d uploaded some wallpapers ages ago. I saw that the whole site had been redesigned though and wondered if I could still find my wallpapers there.
To my surprise the wallpapers were still there and weren’t doing too badly in the rankings. This was helped no doubt by having had a longish time to amass downloads. My most popular one was on page four with 40,000 plus downloads. Not bad I remembered that I was originally planning to make a whole spectrum of these wallpapers, only I never got round to finishing the set. Maybe I will some day…
Moving from Textpattern to WordPress
I’ve been running my blog under Textpattern for a while, and mostly loved it. I had a lot of fun making my own theme and tweaking the software to add in word counts, picture counts, and some elementary Javascript based image gallery feature. But times have moved on and I felt that it would be better to move to more modern and mainstream software, ideally achieving more by doing less.
So I looked at what packages I could add to my Fedora server to make a blog, and WordPress seemed to be the first choice. I installed it, had a play with the admin panel and looked at the code. It looked more readily understandable than Textpattern and there was certainly a lot of community support, plugins and themes. My thoughts turned to getting the content of my blog into the new format and that was where things started to get interesting. I’ll cover the process of transferring the blog content in detail in a future post.
I looked for automated import scripts but the ones that were available were a little old and came with no guarantees – in fact the complete opposite, there were dire warnings about just about every aspect. Also my requirements were kind of unique because the source material was my specially hand crafted content… and I wasn’t too worried about losing permalinks and the like.
I decided to replace my custom Javascript image gallery with something a bit more modern and featureful. I had looked around and there were several candidates, but Highslide stood out for a few reasons. There is a WordPress plugin; it’s simple; also it’s very widely used. In my old approach I used all the images in the text as thumbnails by scaling them down in HTML; then Javascript was used to replace a larger image with the thumbnail when the user hovered over it. The alt text was also copied from the thumbnail to a visible caption. Highslide does most of this and a lot more.
My old Javascript gallery was intended to avoid having separate thumb image files, and was search engine friendly by having captions in the alt tags of the images. Highslide was just a good for search engines but relied on thumb images. I decided that with ImageMagick’s convert the task of maintaining thumb images for my uploads wasn’t going to be a deal breaker. I made thumb images for all my uploaded pictures like this:
I installed the CodeColorer plugin to get syntax highlighting of code snippets, as above. I found a theme I liked, Journalist, and tweaked it just a little. Concerned for the quality of the HTML I was writing, I added a XHTML Validator plugin. That conflicted slightly with the CodeColorer and Highslide plugins so I had to make some minor changes to allow the validator to ignore the custom tags that those plugins used. There was already a validator_wrap_content function in the validator (wp-validator.php) so I added in a couple of regular expression replacements:
// replace any highslide tags with representative HTML for validation purposes,
// since highslide tags are not valid HTML!
$newcontent = preg_replace('/<(highslide[^>]*)>/i','<!-- $1 --><a href="#"><img src="#" alt="#"/></a>',$content);
// same for <code lang= tags, note the non-greedy regex which has the multi line modifier
$newcontent = preg_replace('/<code lang=.*?<\/co'.'de>/s','<code>some code removed for validation purposes</co'.'de>',$newcontent);
return $validate_header . $newcontent . $validate_footer;
I had to make some changes to keep the icons I attached to my post categories – I added a new function to the template:
global $post;
if (in_category('domestic-entropy', $post)) {
echo 'class="dom"';
}
elseif (in_category('down', $post)) {
echo 'class="down"';
}
elseif (in_category('nothing-honest', $post)) {
echo 'class="nothing"';
}
elseif (in_category('signs-of-the-times', $post)) {
echo 'class="sot"';
}
elseif (in_category('up', $post)) {
echo 'class="up"';
}
return;
}
Then I inserted a call to that in the php code that generated pages, for example:
Also, a change to the stylesheet:
padding-left: 20px;
background:url(images/cat-sot.png) no-repeat left;
}
h2 a.nothing {
padding-left: 20px;
background:url(images/cat-nothing.png) no-repeat left;
}
h2 a.up {
padding-left: 20px;
background:url(images/cat-up.png) no-repeat left;
}
h2 a.down {
padding-left: 20px;
background:url(images/cat-down.png) no-repeat left;
}
h2 a.dom {
padding-left: 20px;
background:url(images/cat-dom.png) no-repeat left;
}
That was the extent of the customisation I had to do to make the blog look how I wanted it to. I felt I had something that was personal to me and yet pleasantly implemented. The next step was to import all my content from the old blog…
Premature optimization
Every coder has heard of premature optimization; paying too much attention to details that might turn out to be unimportant in the scheme of things. “Don’t sweat the small stuff” to borrow a phrase from the self-help world.
If you’re working with GNU compilers though (gcc and g++) there’s one kind of optimization that it won’t hurt to try. I mean the one where the compiler does the work. Check the man page for the -O options (-O, -O1, -O2, -O3 and -Os) if you’re not aware of the differences.
By default, the compiler assumes you know what you are doing and write good code that will easily compile up. So, if you don’t ask for anything special it will just do a quick job of compiling the code. In the days when computers were slow and short of memory that was probably a good option.
These days, however, computers are faster and have more memory. That means that it makes sense to have the compiler take more care over the code that is produced. This is what the optimization options do. The compiler carefully considers what can be removed or reorganized to make the program do the same thing but faster or with fewer instructions (which usually amounts to the same thing).
Recently I took some code that wasn’t optimized at compilation time and changed it to -Os (optimize for size). The compiler managed to squeeze the code into 75% of the space it used to take up and the speed was slightly improved. But there was an unexpected benefit too.
Since the compiler looks harder at the code, it can find more mistakes. Because of this, more warnings are produced by the compilers when they are optimizing than when they run without options. To make the most of the warnings, use the flags -Wall and -Werror – they will make sure the compiler doesn’t let you ignore any warnings.
Things like bits of code that can’t be reached, or variables that aren’t always initialized before use can be spotted. Some of these bugs would be hard to find with testing and hard to debug if they happened in software once it was in a customer’s hands.
You can buy expensive static analysis tools to find similar problems, although there is a risk they produce so many warnings that the important ones are hard to pick out. I found that turning on optimization has highlighted some of the more interesting bugs that I could have found with other tools. But I did it for free with gcc. So the moral is: always compile with -Wall -Werror and -O something and it’s never too early in a project to start doing so.
Basic DVD authoring on Fedora
I’ve found my notes on DVD copying useful before, I thought to do the same for authoring DVDs based on readily available content such as downloaded or streamed video or self recorded content. Note that copyright restrictions probably apply to content available online, and insofar as these notes describe the use of downloaded or streamed content they are for educational purposes only. Respect copyright notices and don’t bite the hand that feeds
You might find that digital cameras record .mov files, or you could download them from services like BBC’s iPlayer using software like get_iplayer
I planned to use Brasero as it’s readily available in Fedora. Unfortunately, though it runs easily it really needs lots of other packages to make it useful. I tried to author a DVD and couldn’t add many types of content (eg. .mov files). Checking the docs reveals that:
“In order to use all the potential of the video project, you need to install all GStreamer’s plugins, ffmpeg, vcdimager and dvdauthor.”
I searched and found a few other helpful packages for video that weren’t in the default install. I installed the lot with yum:
yum install xine xine-lib libdvdcss
yum install ffmpeg vcdimager dvdauthor
yum install gstreamer-plugins*
Unfortunately that stopped Brasero from working completely, so I had to remove gstreamer-plugins-bad-extras. It turns out there is an easier way to get the plugins. First try and open the file you want to put on a DVD, by just clicking on it. The default video player opens. If it has trouble understanding the file it will ask if you would like to search for the appropriate codecs. Brasero doesn’t know how to do this and just complains that it can’t use the file.
When trying to view the .mov file, Movie Player installed gstreamer-ffmpeg for me. After that, Brasero was able to add the .mov files to my project.
To my disappointment, trying to use Brasero further to write a disc didn’t result in success. The burning process stalled at the point of making an image. At least I didn’t waste any discs.
Having failed with Brasero I was recommended to try k3b or similar (k9copy also does authoring and has been seen to shrink things to the right size for a disc before – very useful). I checked gnomebaker but it wasn’t yet usable for creating video DVDs. If it’s possible to use k9copy and k3b to make a video DVD from scratch… I couldn’t find the way. The first method I found that worked was DeVeDe, and creating the iso image with that was trivial. Just add tracks, add files, ‘adjust disc usage’ to set the compression rates for the videos, and sit back while the iso is prepared.
Once you have an iso pretty much any burning tool will reliably write it to a blank disc.
Bluetooth PAN from Windows Mobile to Fedora 11
If you have a mobile phone with a generous data plan, and a laptop without an internet connection, it’s tempting to put the two together to be able to use the internet on the move. While on holiday I managed to do it with a Windows Mobile phone and a laptop running Fedora 11, here’s how…
A working internet connection involves, amongst other things, being able to look up the addresses of hosts, ie. converting a name like www.google.com into a numerical IP address for a server. This is typically done with DNS. Also, the internet connection provides a router which will accept packets from the laptop and send them onwards towards their destination. This is the default gateway. Exactly how this is achieved doesn’t really matter, so long as it works and a reply from the destination can find its way back to the laptop.
When using a mobile phone to access the internet, the network operator will provide certain information that enables the phone to access the internet. Again the details are not that important so long as it works. Parts of the phone might be acting like a modem – in the past I have connected a phone to a laptop and the laptop recognises the phone as a modem. In this case the laptop has to provide the same esoteric information to be able to make the phone connect to the network.
A more modern way of connecting the phone and laptop is to use a Bluetooth PAN (Personal Area Network). This has the advantage of being wireless, as well as being completely removed from the details of the phone’s connection to the network. As we can see, the laptop only needs to connect to the PAN and doesn’t need any mobile network information.
First, some notes about the hardware and software I used. The phone was an O2 XDA Ignito (rebranded HTC touch diamond) running Windows Mobile 6.1 Professional (CE OS 5.2.20779 Build 20779.1.4.11). The laptop was a Dell Inspiron 6000 with Bluetooth, running Fedora 11. The kernel was kernel-2.6.29.6-217.2.3.fc11.i586 and the Bluetooth packages were bluez-4.42-1.fc11.i586. I was using the Gnome desktop on the laptop.
Setting up Bluetooth is relatively simple. The basic steps are to enable Bluetooth on the phone and laptop, ensuring both devices are ‘visible’. On the laptop, right-click on the Bluetooth icon on the panel and choose ‘Setup new device…’. The Bluetooth device wizard guides you through the process.
To set up the PAN, the laptop and the phone have to be paired – this is a nod to security in the Bluetooth world. On the phone start ‘Internet Sharing’ and choose PC Connection ‘Bluetooth PAN’. The Network Connection should be the default that the mobile operator has set up. Click ‘connect’ on the phone and it should say it’s waiting for the PC to connect. That’s all you need to do on the phone.
On the laptop, the bnep kernel module is needed – I noticed it was already loaded though. The software that sets up the PAN is called PAND (PAN Daemon) and it can be run manually like this:
pand —ethernet=bnep0 —connect 00:21:BA:FE:7E:C4 -r PANU -d NAP -n
This means to use the bnep0 device, to connect to a certain Bluetooth address (ie. your phone) with a role PANU (PAN User) and service NAP. The only mystery here is finding the phone address – it can be done like this:
sdptool search NAP
Which will report the Bluetooth address of nearby phones providing NAP.
One day this will all ‘just work’ with Network Manager, according to people I’ve spoken to on IRC, but I’ll believe that when I see a Gnome notification balloon pop up to tell me that everything is working. In the meantime the closest I could get was to add the file /etc/sysconfig/network-scripts/ifcfg-bnep0 which matches the eth0 (wired network) and eth1 (wireless network) files. Of course since it is a different kind of device to wired or wireless ethernet, it has different configuration options. In Fedora I feel this networking stuff has a whiff of Hitchhikers’ Guide to it – if anyone ever figures out exactly how it is supposed to work, and what it’s for, it will disappear in a yum upgrade to be replaced by something even more bizarre and inexplicable. This has definitely already happened.
I found out what to put in the ifcfg-bnep0 script by looking at the ifup-bnep script and seeing what variables it needed to call bnep with:
DEVICE=bnep0
BOOTPROTO=dhcp
ONBOOT=no
REMOTEBDADDR=00:21:BA:FE:7E:C4
ROLE=PANU
This results in several things happening when you type ifup bnep0, including this pand command being run:
/usr/bin/pand —persist —pidfile=/var/run/pand-bnep0.pid —ethernet=bnep0 —autozap —cache —connect 00:21:BA:FE:7E:C4
You can also shut it down with ifdown bnep0.
This works in the sense that the phone will go from ready to connected then disconnected in response to ifup and ifdown, but it’s not enough to get internet access that works. The interface needs bringing up and configuring. I found some things online suggesting that DHCP should work, but I wasn’t able to get any responses to dhclient – not a single packet. It wouldn’t even respond to a broadcast ping. But if I assumed that the phone would be 192.168.0.1/24, and set the laptop to 192.168.0.2, it just worked. To configure this we need ifconfig:
ifconfig bnep0 192.168.0.2 up
We also need to add a default route:
route add default gw 192.168.0.1 bnep0
I found that there is a switch for the pand command that will run a command on bringing up the connection, —devup. So I put the commands into a script and tagged it on to the end of the REMOTEBDADDR in the ifcfg-bnep0 script:
DEVICE=bnep0
BOOTPROTO=dhcp
ONBOOT=no
REMOTEBDADDR=“00:21:BA:FE:7E:C4 —devup /etc/sysconfig/network-scripts/ifup-bnep-post”
ROLE=PANU
This was my /etc/sysconfig/network-scripts/ifup-bnep-post script:
ifconfig $1 192.168.0.2 up
route add default gw 192.168.0.1 $1
cp /etc/resolv.conf.opendns /etc/resolv.conf
I figured out that the device name (bnep0) was passed as the first argument to the devup script. The Bluetooth address is passed as the second parameter but that wasn’t any use. The last line is optional, it configures OpenDNS servers. Using the default gateway (ie. the phone) to look up DNS names works, but the OpenDNS servers seem faster and more reliable, and avoid burdening the phone with the responsibility to cache and proxy the requests. This is my /etc/resolv.conf.opendns file:
nameserver 208.67.222.222
nameserver 208.67.220.220
So finally I have a setup that works reliably, with one command to bring up and take down the interface on the laptop and a few clicks on the phone. It does seem a lot simpler than the old modem strings and wired connections. If only Gnome desktop would automate things a bit it would be great…
Caramel Crater Shortbread
I made some caramel shortbread because I was bored, here is the recipe…
I made standard shortbread – in other words half as much sugar by weight as butter and twice as much plain flour by weight. I used half a block of butter, about 120g, so that calls for 60g sugar and 240g flour. I used Fairtrade dark brown sugar.
The method is to chop up the butter small, then add everything else and rub it until it looks like sand. Then spoon it into whatever you’re baking in and press it lightly. I chose muffin cases and the amount of mix fitted into 12 cases.
I wanted to put caramel into them so I used a round thing to press a crater into the shortbread. I baked them at gas mark 5 until they looked and smelled about right.
The caramel is just melted butter, sugar and a little milk. About a quarter block of butter should be plenty (60g) and as much sugar as will go into it (about 150g). A splash of milk seems to help it cook evenly without burning. It needs to boil gently with constant stirring. When ready a few drops on onto a cold plate should turn gooey – usually about 10 minutes of boiling.
When the shortbreads are cooled the caramel can be poured in carefully. Then they need to cool again before they can be eaten.
Adding a big disk to a Fedora server
I made some notes about what happened when I added a Terabyte of storage to an ageing server, how I rearranged things and what tools I used.
A home server consisting of a headless PC (ie. no screen or keyboard) is fairly simple to start with, but as it grows and evolves some care is needed to keep things well organised. In my case the server is a reused PC case, an Athlon CPU and motherboard with one ATA bus and two SATA connectors. An ancient 80GB ATA disk dated back to the Windows 95 era, it had been updated with another 80GB disk, then again with a 160GB SATA disk. Now I add a 1TB SATA and the machine is fully expanded. This is the second motherboard/CPU combination that the PC has seen, in the future I could add another SATA PCI card, or upgrade to a motherboard with more on-board SATA interfaces.
The OS is Fedora Linux – I had been running F10 for a while and took the opportunity to upgrade to F11 using preupgrade-cli, which went flawlessly. For this I attached a keyboard and screen, just in case I needed to intervene when the network connection was down – I didn’t. Normally access to the server is over ssh from a normal laptop or desktop PC.
The machine recognised the disks starting with the 160GB disk as the boot disk, the second SATA bus wasn’t connected, then the two ATA disks appeared, so I had sda, sdb and sdc. The /etc/fstab file mentioned several partitions on the sdb and sdc drives, so after adding the new disk, which became sdb, the fstab file had to be edited. On the first boot I got errors and a repair console. I just had to remount the root read-write and edit the fstab file to correct the partition names to reflect the changed device names, so
mount / -o remount,rw
In /etc/fstab:
/dev/sdb1 -> /dev/sdc1
/dev/sdb2 -> /dev/sdc2
/dev/sdb5 -> /dev/sdc5
/dev/sdc1 -> /dev/sdd1
/dev/sdv2 -> /dev/sdd2
I exited and the next boot was fine. But I had an empty disk without even a partition table. I could create a partition table but I wanted to assign the whole disk to a logical volume without any waste – this is unlike the first disk where a partition table was needed for booting. So I used the pvcreate command:
pvcreate /dev/sdb
Then I created a volume group:
vgcreate fatty /dev/sdb
You can display the physical volumes and volume groups using pvdisplay and vgdisplay. The latter is useful because it tells you how many extents are available, and that number is used in the next step. I had 238467 extents in ‘fatty’.
[root@mediator ~]# vgdisplay fatty
—- Volume group —-
[...]
Total PE 238467
Then I created a logical volume on the physical volume, using all the extents:
lvcreate fatty -l 238467
You can display the logical volume with lvdisplay, but if the command line tools aren’t user friendly enough you can use a graphical version. At first I hoped gparted would work, but unfortunately it doesn’t support LVM yet – this is a real shame. In fact my LVM on a physical disk without partition table showed up as an empty disk – which would be a bad thing if a naive user decided to overwrite it. Maybe I should have made a partition table and a single LVM partition in it after all.
The other graphical tool worth mentioning is system-config-lvm. It’s not installed by default so I had to grab it with yum:
yum install system-config-lvm
This is a good way to view and modify LVM but it doesn’t support formatting, cryptsetup or anything like that, which is a shame. I found no enterprise-ready storage management graphical tools. Maybe there are some and I just didn’t find them.
Resolving to use the command line tools, I looked at the existing layout: one boot disk with a boot partition, and a volume group split into home, root and swap; one ‘recycled’ disk with boot (unused), root (unused), swap (unused) and home (mounted as backup) partitions; one ‘upgrade’ disk split into two partitions, for backup and music.
The idea for the new order was to have the bulk of the storage on the Terabyte disk. a separate system disk for booting and home directories, and a disk for the web server / databases. This would leave a spare disk for backups or emergency. I would do all the copying I needed to do first and leave the existing disks untouched, just in case… I chose the ext4 filesystem as it seems reasonably stable now. I made an ext4 filesystem on the new volume:
mkfs.ext4 /dev/fatty/lvol0
I made a mount point and added the filesystem to the fstab file:
mkdir /opt/storage
In /etc/fstab:
UUID=7028c111-de8a-4cfb-b044-6ef9606a4faf /opt/storage ext4 defaults 1 2
I got the UUID using blkid:
blkid /dev/mapper/fatty-lvol0
/dev/mapper/fatty-lvol0: UUID=“7028c111-de8a-4cfb-b044-6ef9606a4faf” TYPE=“ext4”
Knowing that, I changed the rest of the filesystems in fstab to use the UUID instead of a changeable device name – no more booting problems after changing disks!
I set about copying the existing stuff to the new locations. There are many ways to do this but I chose rsync, because I like its progress reporting.
cd /opt/old-home
mkdir /opt/storage/old-home
rsync -a —progress * /opt/storage/old-home
Then a long wait for the old, full filesystem to be copied into its new home…