Sorting disk usage, Perl to the rescue.

My MacBook Air was close to full up after a lot of music editting (see results at http://www.stmaryssingers.com/recordings.html ). In the past I would run

du -s * | sort -rn

to identify the biggest users of disk space giving for example:
14104880        iTunes
683304  Noël Français
682592  sibs
221488  DancingDay
93664   MissaAlmePater
27480   ChristmasLullaby
17680   JesuJoy
13656   MissaBenedicamus
12256   MassInHonorOfSaintJoseph
8760    pdfs
3584    ChrissyCarols
2456    LookingAtTheStars
1352    BriggsMass
408     Bach-Jesu
264     40_The_First_Nowell.sib
64      Thou-knowest-Lord-Z-58b.pdf
24      StMS20140412
24      StMS20140322
24      StMS20140309
24      StMS20140208
24      StMS20131214
0       GarageBand

and the numbers are in 512-byte blocks. Nowadays that's a lot of digits to decipher in the listing. So I started using the 'h' ('human readable') option:

du -sh * | sort -rn

giving:
676K    BriggsMass
334M    Noël Français
333M    sibs
204K    Bach-Jesu
132K    40_The_First_Nowell.sib
108M    DancingDay
 46M    MissaAlmePater
 32K    Thou-knowest-Lord-Z-58b.pdf
 13M    ChristmasLullaby
 12K    StMS20140412
 12K    StMS20140322
 12K    StMS20140309
 12K    StMS20140208
 12K    StMS20131214
8.6M    JesuJoy
6.7M    MissaBenedicamus
6.7G    iTunes
6.0M    MassInHonorOfSaintJoseph
4.3M    pdfs
1.8M    ChrissyCarols
1.2M    LookingAtTheStars

  0B    GarageBand
Unfortunately sort doesn't know how to sort the unit suffixes. But Perl does. It's a while since I used the Schwartzian Transform but it seems perfect for the task. I copied the Wiki code into a file, dusort.pl, which I placed in a directory in my PATH variable (~/bin in this case) and modified the regex extraction to make it sort by unit suffix first and then by number giving this:

#!/usr/bin/env perl 
use 5.010;

my $size = {P => 6, T => 5, G => 4, M => 3, K => 2, B => 1};
print
  map { $_->[0] }
  sort {
  $size->{$b->[2]} <=> $size->{$a->[2]}
                  ||
        $b->[1] <=> $a->[1]
  }
  map { [$_, /^([ \.0-9]{3,4})([PTGMKB])\t/] }
  <>;

and now when I run the command(s):
du -sh * | dusort.pl
I get the result:
6.7G    iTunes
334M    Noël Français
333M    sibs
108M    DancingDay
 46M    MissaAlmePater
 13M    ChristmasLullaby
8.6M    JesuJoy
6.7M    MissaBenedicamus
6.0M    MassInHonorOfSaintJoseph
4.3M    pdfs
1.8M    ChrissyCarols
1.2M    LookingAtTheStars
676K    BriggsMass
204K    Bach-Jesu
132K    40_The_First_Nowell.sib
 32K    Thou-knowest-Lord-Z-58b.pdf
 12K    StMS20131214
 12K    StMS20140208
 12K    StMS20140309
 12K    StMS20140322
 12K    StMS20140412
  0B    GarageBand
Obviously the next thing to do is to make a shell alias:
alias dus='du -sh * | dusort.pl'
and now I even save a few keystrokes in my task to pinpoint the Biggest (L)User.

Threes! and Threesus A.I.

I decided to try the Threes! app/game on my iPhone. I got to around 21,000 just playing by myself but I went Googling to see if there were ways to improve my score. Found quite a few "tips" articles had discovered most of what I was doing. (No-one mentions how to speed up the score totalling at the end of the game. (Swipe the screen a second or third time. Saves seconds per game.))

But then I discovered Threesus, watched the video for nearly 30 minutes in total awe and decided I would download Threesus and use the Assistant to help me get my score higher.

And once again it became another exercise in yak-shaving...

Threesus is written in C#, Microsoft's attempt at Java+Objective-C+... and is apparently pretty-well compulsory if you are programming on Windows. The author states that one can install Visual Studio Express for Windows Desktop and Threesus should compile and run in commandline Assistant mode.

Well, I don't have a Win box sitting around but I do have Parallels on my MacBookAir so I downloaded and installed Express and was able to fire up ThreesusAssistant very quickly. I had to read the source code to work out the commands but they are simple enough.

What intrigued me was the mention of the 'bonus' cards and how they are indicated by a '+' sign on the card. I'd never seen the '+' sign. The bonus cards were just like the normal '3' card on my version of Threes. Walt Destler has also blogged a high-level overview of Threesus and he refers to a detailed description of the Threes! algorithm and that's when I realised my copy of Threes! was out of date. Version 1.0.3 introduced the '+' sign on bonus cards.

So after a quick update I was able to use the Assistant to get my score upto around 28,000.

But disaster hit! The Assistant failed with a null reference error before I could get a higher score. I didn't fancy attempting to debug a run-time error in a language I don't know using an operating system I don't know.

Destler also mentions Xamarin Studio as a possible alternative for Mac users. I duly installed it and spent quite a few hours attempting to get Threesus to compile and run. Lots of googling for obscure errors, settings etc. etc. The usual yak-shaving for any new software.

At last I got Threesus to start in Xamarin. I input the initial grid data and it commenced "thinking". About 5 minutes later it returned the next move. Then it took 10 minutes for the next move. No way is Xamarin/Mono usable on Mac OSX. The Assistant is already slow enough in Visual Studio on Parallels but it seems to be 100s of times slower in Xamarin on Mac.

So now I have a problem. Any sensible person would give up at this stage and find something useful to do.

But not me. I threw a few dollars into the RPerl Kickstarter project last year in the hope that they could get further than a faster bubblesort and it seems to be coming along nicely. Perhaps it's ready to take on a Threes! playing A.I.? Once again a lot of yak-shaving to get the source downloaded and installed but it appears to be working. Now to see if I can translate/recode C# code into working Perl and then possibly speeding it up with RPerl. RPerl uses the Perl Inline module which also allows raw C or C++ to be in-lined into Perl code. That might save a lot of hassle.

I also realised that this could be a nice project to try out OCRing the iPad screen of the Threes! game grid. I don't want to build a full game-playing robot but the slowest and most error-prone part of using the Assistant is inputting the changes. If I could add a screen-reading section so that the Assistant merely has to output Left, Right, Up or Down it would speed manual play greatly. And while I'm at it, why not use a text to speech converter so I don't even have to look at the Assistant's output. (Time to start lobbying the game developer to add speech input!)  I've got an old iPhone 3GS to supply video input and there are some great articles on OCRing Sudoku grids from newspapers etc. The Threes! grid is very similar. So this might turn out to be a very elaborate but maybe quite fun project. Or a total disaster and time-sink...

A GA144, a 10MHz ceramic resonator and an LED.

I've found some time to continue my dabbling in the intricacies of the GA144 chip. I spent too many hours trying to copy a demo which used a 32KHz watch crystal to provide accurate clocking for a couple of nodes in the chip. In the end I was unable to get the crystal to sustain oscillation for longer than about 30 seconds. And the nature of the component meant I was unable to monitor the oscillations with an oscilloscope. So I switched to using a 10MHz ceramic resonator and there is a similar demo in the ArrayForth sample code. The main difference is that this demo works! (Ceramic resonators are apparently easier to drive.)

I was also experimenting with the PWM demo application in the User Guide Ch. 9 and there is a small side note to the effect that maybe the constant used to set the LED's brightness ought not to be simply a linear increase but perhaps a Fibonnaci sequence because we perceive brightness increases logarithmically.

So I wondered if I could make the oscillator node send a signal every 100ms to its neighbour to cause the brightness constant to increase in a Fibonacci sequence and thus cause the LED brightness to increase slowly in an apparently "linear" fashion.

A couple of minor details: I used node 417 as the oscillator and attached a Murata CST 10.00 MHz resonator between pin 417.17 and ground. I used node 317 to drive the LED with the anode connected to the 3.3V supply from USB port A and the cathode connected to pin 317.17. (On the EVB001 eval board 317.17 and 417.17 feed out to header J21.) (Refer User Guide Ch. 9 for details of LED connections.)

The following code is very similar to that given in Block 956 in the ArrayForth sample code. However I incorporated changes as suggested by Chuck Moore in his StrangeLoop 2013 talk. And I added bigger and bigger delays between signals to node 317 (via up port). In the end the delay is a million to one so that the 10MHz oscillation is reduced to a 10Hz signal.

Block 864
init waits for a commencement signal from node 317 via the up port, loads the stack with 8 numbers and sets them to become a repeating stack array.

go runs 10 hi-low pulses to the resonator then drops into an infinite loop which calls pump and ring to keep the resonator pulsing and periodically outputs an instruction to node 317.

864 list 
417 orig tx osc 417 node 0 org
drive 00 4n drop !b !b for unext ;
pump 03 4n !b @ drop !b !b drop ;
ring 05 4n !b @,
.........drop drop drop drop ;,
,
go 07 9 for drive drive next 0c .,
...begin 0d,
......999 for 999 for,
.........pump pump ring ring,
......next next,
......17 a up a!,
.........@p 19 ; .. 1a ! a!,
...end,
,
init 1c up a! @
xx 1e drop io b! left a!,
7 dup dup or 20000 800,
6 dup dup or 30000 over,
dup dup drop drop go ;,
2c reclaim

Block 866
init sets the a and b registers and loads the stack with 1 and 1, the first numbers in the Fibonacci sequence, then sends an instruction to node 417 causing it to commence the oscillation loop.

pol tests if the up port has a signal on it. If so, elements on the stack being used by the cyc routine are dropped and the next value in the Fib sequence is calculated. If the Fib value becomes "negative", the stack is reset to 1 and 1 otherwise the Fib number is left on the stack to be used by cyc.

cyc uses the PWM method explained in the User Guide. The stack contains a value between 0 and 20000 hex representing brightness i.e. 0 (1) means no light, 20000 means full on/max. brightness.

866 list 
pwm demo 317 node 0 org br

init io b! up a! 1 dup dup dup,
.....@p . .. 06 ; 07 !,
pol 08 @b 200 uw and if,
...0b up b! @b drop drop drop,
...+ -if 1 dup dup dup,
...then over over dup dup dup or,
...io b! then 15 drop br


cyc ie- 1ffff and over . + -if cr
...19 .....20000 !b pol ; cr
...1c then 10000 !b pol ; 1f                  

Block 862
is the loader block.

862 list
loader template

host load loader load,

using default ide paths,

kill boots 0 708 hook 0 -hook,

path to 317 0 317 hook,

,

setup application,

417 +node 417 /ram 1c /p,

317 +node 317 /ram 0 /p,

panel pause 0 ship upd ?ram 




                  

A Dockerfile for FixMyStreet Part 2

More days later...

Update: many of the bugs and glitches I mention below have been fixed in Docker version 8.

I was able to access the FMS server running in the VirtualBox image on my MBA but only from within the image. Couldn't access it from my browser running on my MBA.

I tried a variety of hacks. The first one was to ignore dvm/vagrant and use VB's console to change the networking setting for Adapter 1 to bridge => en2: Display Ethernet. This allowed me to set the IP address to one within my LAN and I was able to access FMS in my browser.

But this required manual intervention and I wanted the image to be as automatic as possible. So I tried editting ~/.dvm/Vagrantfile to created a public network and to set the adapter to bridge. But this caused a conflict error message to halt the startup. I tried a variety of changes and none remove the conflict.

Eventually I realised that the default setting (192.168.42.43) address is accessible to the browser running on my MBA. Not sure why I didn't check this first. Unfortunately it's not accessible by other browsers on my LAN (192.168.1.*) but it's only for development so I'm happy.

Then I used Intercity's note about flattening the Docker image. It dropped from approx. 2GB down to 750MB. Still quite large but better.

Then I tried to push the image to Docker's public repository.  And I'm still trying... The upload keeps timing out, or the Docker DNS's disappear or something else.

Intercity's flattening note also shows how to export the image to a tarball and reimport it and I've at least got a tarball with which I can re-import the image as needed.

So lots of "learning experiences". I still think Docker might be the best devel tool since the cloud appeared but still many rough edges for people who use Macs.

A Dockerfile for FixMyStreet.

This is going to be a long and rambling report of how I go about writing a Dockerfile to build FixMyStreet (FMS) as an (almost) stand-alone web app. I'm making this up as I go so there will probably be lots of false turns.

Some preliminaries

I'm developing using Docker on my MacBook Air and deploying the image on Google Compute Engine. I installed the following on my MBA:

• VirtualBox
• Vagrant
• dvm (installs and fires up boot2docker, a minimal Linux image in VirtualBox and runs docker in daemon mode)
• Docker client for Mac OS X (connects to docker daemon running in VirtualBox, WARNING: Client is faulty, see below)
• FixMyStreet (git clone --recursive https://github.com/mysociety/fixmystreet.git)

In the fixmystreet repository there is a Debian install script: commonlib/bin/install-site.sh, which I "translated" into Dockerfile commands as much as I was able to.

Then I followed the FMS manual installation instructions but once again I translated them to the equivalent Dockerfile instruction.

Two days later...

Well, I've wasted another two days of my life in futility. It seems that, at least on my MBA, the Mac OS X client has a fatal flaw: the connection to the VirtualBox image seems to drop after about ten minutes. It took me a day to realise that it was the Mac client that was faulty. I must have run and re-run the docker build a few hundred times, trying to isolate the fault.

And then in desperation I tried dvm ssh to log in to the boot2docker image running in VirtualBox. From there I copied my Dockerfile into the login directory and ran docker build and docker run from the login shell. And it ran without dropping the connection once. D'oh!

However it wasn't plain sailing from there. The final manual instruction is to run a script which installs all the requisite Perl modules. The script uses Miyagawa's Carton module to manage all the dependencies of the 84 modules used to build FMS. I ran the script both as a Dockerfile command and by logging into the container and running the script from the commandline.

No matter what I tried, modules were failing to compile (i.e. the gcc compiler was fatally crashing). Once again I wasted many hours Googling for the error messages until I found out the unexplained gcc crashes are usually from lack of RAM. Sure enough the default RAM for the container was only 512MB. In ~/.dvm/dvm.conf on my MBA I was able to change the RAM to 1024MB (my MBA has 4GB, plenty to spare). And at last all the Perl modules compiled and installed without complaint.

Finally, there were some Postgresql initialisation commands. These can only be performed when the postgres daemon is running so I decided to echo the commandline instructions into ~/.bashrc (along with a couple of useful aliases) so that when I run the image and fire up a bash shell, the remaining initialisation is performed upon my first login. I can then manually delete the instructions and I have an initialised and running instance of FixMyStreet.

Update: .bashrc only needs to start Postgres and FMS, other initialisations are now in the Dockerfile.

Image and Dockerfile: https://index.docker.io/u/garyaj/fms/

One RN104 was a disaster. The perils of small sample sizes.

Netgear support over the days following my report of the problems I was having with the RN104 was awesome. They tried everything, kept me informed at all stages and eventually asked me to return the old one and get a replacement.

The replacement now has an uptime of 26 days and hasn't skipped a beat.

Moral: don't generalise based on a sample of one. (Of course I could quibble about how the faulty unit got past QA in the first place...)

ReadyNAS RN104 is a disaster.

You get what you pay for they say. In my upset about almost losing a year's data from my ReadyNAS NV+ I purchased an RN104, Netgear's latest version of "consumer level" NAS. I was kind of hoping that after all these years of development the system software would be more stable than that on the NV+.

After much googling and angst I was finally able to copy my data from the stalled NV+ onto my new 104 and I thought it was all plain sailing after that.

Three weeks later and I'm ready to demand my money back for the 104. It has stalled at least once a day for the past three weeks, sometimes twice, even three times. Thankfully redundancy in the disks has allowed resynching (if it doesn't stall first) so I haven't lost any data (yet) but basically I've got a very power hungry brick. Obviously its performance as a file server is very poor while it's resynching.

I don't think Netgear support has the slightest clue about why it's happening. The ReadyNAS forum  is full of complaints about the stalling. I've filed a support request but so far none of the replies has indicated they have any idea why it stalls.

It might be snapshots, it might be automated backups, it might be power supply under load, it might be the anti-virus software (not enabled in my case), it might be CPU too hot under load...

At the moment my old NV+ is sitting next to the 104 looking very smug. At least it only has unrecoverable stalls once a year or so...

As an indicator of how clueless Netgear support people are, I posted a complaint on the forum about the iTunes server in OS6 being compiled to serve only Windows boxes. It serves up FLAC files with the bytes reversed for a Mac and all one hears is a deafening squeal. "Thanks for the heads up" was the reply from Netgear. They never tested it!