Sunday, June 15, 2014

Porting the MusicBox app to GA144

Introduction

The pre-cursor to the GreenArrays  GA144 was the SeaForth S40C18. The SeaForth had 40 nodes whereas the GA144 has 144 but they have the same instruction set. Chuck Moore parted ways with Intellasys, the company which produced the SeaForth, a couple of years ago and the acrimonious lawsuit which resulted was settled last year.

One of the demo apps which was included in the VentureForth compiler kit was a musicbox app. It uses a synthesised plucked string algorithm to generate random but quite pleasant 'plucked string' music.

I've always liked the app so I decided for my own education to port it to the GA144 using ArrayForth.

The first thing I discovered was how great the divergence there has been between VentureForth, the version of Forth used on the SeaForth chip and ArrayForth, as used on the GA144. In addition ArrayForth is a closed universe. It's almost impossible to import any program files into aF. They have to be hand-typed, whereas vF is ANSI text based and I could use standard text editors such as Vim to manipulate the program code. However the need to understand each instruction meant hand-typing was a relatively small hurdle. The bigger task was to understand what the instruction meant in vF and replace it with the equivalent aF instruction. While the instruction set is one-to-one compatible, the compiler directives in vF are completely different and in some cases there is no equivalent. This caused me a few headaches.

The other handicap I faced is my lack of knowledge of Forth. In the end this wasn't a big problem because the code for MusicBox is designed for Forth chips like the SeaForth and GA144 and doesn't rely on a lot of standard Forth familiarity.

Overview of code

The code uses the best feature of the GA144, namely the ability to offload work onto adjacent nodes while continuing with another task. The "central" node, called the composer, decides which note to play next. It then relays this choice to one of six "plucked string" synthesis nodes which generate code using the Karplus-Strong algorithm. The resulting streams of note-generation code are fed to a moving average filter node which feeds the result to a 'pre-dac' node which converts the PCM music code to PWM code which is in turn fed to a node which controls one of the digital-analog converters (dac). The output of the dac is fed to headphones or a speaker.

Eighteen nodes are used. The composer node uses "random" input from an ADC to select the next note to play from a list of harmonically related notes. There are no "discordant" notes.

The output of composer is sent to a router node which keeps track of which nodes are busy synthesising notes and channels the next note to the next free node.

There are six "pluck" nodes. The KS algorithm uses a one-sample delay and so there are six "delay" nodes, one for each "pluck" node. Similarly the MA filter node requires a one-sample delay node.

The output of the MA filter node is passed to a "pre-dac" node which calculates the three parameters needed to drive the dac node. The calculations are time-expensive and are thus off-loaded to a separate node rather than attempting to run them in the 'dac' node.

Thus 1 composer, 1 router, 6 pluck, 1 filter, 7 delay, 1 predac, 1 dac = 18 nodes. Composer node has to be a node with analog input and obviously the dac node must have analog output. This constrains where in the GA144 the nodes can be. Thus I chose the following path:

717 (composer), 617 (router), 616, 615, 614, 613, 612, 611 (pluck), 610 (filter), 609 (predac), 709 (dac). Delay nodes are: 710, 711, 712, 713, 714, 715 and 716.

Composer

bc (bit count, by Michael Montvelishsky) The note to be played is chosen by counting bits in the number on top of the stack. That bitcount is used to index the table of frequencies at the beginning of the code, and that frequency is passed on to the router node, to be given to one of the voice nodes.

note Use note number to lookup frequency then send it on to the router. 400 is a constant used to determine the length of a rest, and therefore the tempo. Decrease constant to play faster.

play The actual note to play is derived by counting the bits in the number on top of the stack. If the new note, determined by counting the bits in 'new' is the same as the old then play a rest i.e. be quiet. Otherwise give the< new note number to 'note'. The note played is also left on the top of the stack to be compared with the next note.

piece Read a random number from the adc counter at 'data', "and" it with 511 to keep it reasonable (dac has only 512 levels). Store number in A register to be used as an increment to find the next note in the "piece".

compose Play 127 notes, beginning with 0 and incrementing the "note" number by the amount stored in the A register by piece. The actual note is determined by counting the bits in the number fed to play.
 
1204 list 
musicbox - plucked string synthesis
713 node 0 org
lookup table of frequency data
27400 , 27400 , 24500 , 21800 ,
19400 18300 , 16300 , 14500 ,
13700 , 12200 , 10900 , 9700 ,
9100 , 8100 , 7200 , 6800 ,
6100 , 5400 ,
bc 12 bitcount dup dup or - for
      
dup push zif drop pop - ;
      
then pop and next
note
 17 a push a! . 18 @ !b pop a!
rest
 400 ;
delay
 1b dup for
         
dup for unext dup or -
      
next
      
1f drop ;
play
 20 bc over over or if
         
drop dup push note
         drop pop ;
      
then rest drop drop ;
piece
 28 random adc
       data a! @ 1ff and a! ;
compose 2c a dup dup or 
notes 127 for
       
2f over play 30 push
        a . + pop
     
next drop drop ;
start 33 e000 !b down b! rest
      
begin piece compose end 3a

Router

ring The address following ring is a variable that holds the next address to
be executed as a coroutine in the list that follows the variable. When
used in voice and force, the effect is to cycle through the
numbers in the "tables", returning the next number each time voice or
force is executed.

+note  sends the note-on message on to the mixer chain, along with a
voice number and force number, telling the chain which node should
process this note and how loud it should be.

The main loop of the router first checks io register to see if the composer is
requesting attention. If so then a note is received from the composer
and passed on to the appropriate voice node. In either case a "play"
message is sent on to the next node in the mixer chain, to keep the
note samples going.


1216 list 
router 613 node 0 org
ring pop b! @b push ex pop !b ; 
voice ring
5 ,
r1 0 ex 1 ex 2 ex 3 ex 4 ex 5
   ex r1 ;
force ring
14 ,
f1 120 ex 100 ex 80 ex
    70 ex 60  ex 50 ex f1 ;
+note 4 voice @p ! ! ' a relay '
   
! @p ! . ' @p a! @p . '
   
' w lit ! ! @p ! @p a!
   
! force ! ;
start 2c up a! io down
   
begin
      
over b! @b 2* 2* 2* 2* -if
         
33 over a push a! @ pop
         a! +note
      
then @p ! dup . ' @p play '
           
or !
   
end 3a

Pluck

Pluck uses the Karplus-Strong algorithm, http://en.wikipedia.org/wiki/Karplus-Strong_string_synthesis
A better explanation is at music.columbia.edu - Start with a buffer full of random numbers which is equivalent to an energetic string pluck, read through the buffer using the values as sample values, average each value with the previous value and write it back to the buffer as well as forwarding the sample to the dac player. Over time the averaging is equivalent to a low-pass filter and will remove the higher frequencies until eventually the waveform will be flat i.e. the string has stopped vibrating.

1214 list 
pluck - karplus-strong string synthesis
0 org
pluck dup push . + 2/ pop a -if
         
drop 2/ 2/ ;
      
then push zif
swp
 05 over push push drop pop pop ;
      
then pop a! drop drop @p drop @p
!rnd
 dup !p ; 3ffff , rnd 0b -if
      
2* 2cd81 or @p
   
then 2* dup .. drop !rnd 1ff and dup ;
rwrw
 12 @p !b @b . ' !+ @ !b .. ' 1ff and ex
   
@p !b !b . ' @p .. ' ex
   
@p !b @b .. ' @ !b .. '
   
2/ 2/ 2/ 2/ 2/ 2/ 2/ 2/ 2/ 1ff and ex
   
8 for 2* unext
   
@p .. ' @p . + . ' !b !b ex rwrw ;
play 26 @p drop !p
mix
 27 dup @p + ;
0 ,
pop drop push -if
   
1ffff and over b! pop ex pluck ex push
then push over b! pop @p . +
w 33 1ffff ,
pop mix @p !b !b mxplay 36 ' @p play ' 37
                                              

A full listing is on GitHub: https://github.com/garyaj/musicbox


Monday, May 19, 2014

KiCad on Mac OS X Mavericks

Eventually I want to design a new circuit that I would like to implement as a real gadget. This means, eventually, I will need to design a PCB (printed circuit board) to hold all the parts and wire them together. I looked at a few different products. There's the really expensive Altium (supposedly the best :); there's a few no cost but proprietary ones e.g. ExpressPCB; there's a few no cost for initial limited sizes (pay for unlimited) e.g. Eagle and there's a few FOSS products e.g. gEDA and KiCad. I can't explain why but I decided I liked the look of KiCad and I wanted a tool without artificial limitations. And it (supposedly) runs on Mac OS X.

So here's the story of how I spent nearly a week getting KiCad EDA (Electronic Design Automation) to work on my MacBook Air running 10.9 (Mavericks).

To give me motivation and direction for my efforts I used a series of video tutorials from Contextual Electronics called 'Getting to Blinky'. The videos seem (to me at least) to be perfectly paced so I didn't get bored or (often) lost. Obviously the first step was to install KiCad on my MBA.

Native

There are at least three different ways to run KiCad on Mac OS X. One is to run it as a native app either by compiling it yourself or downloading a compiled version. It quickly became apparent that the Mac version is very poorly supported despite a lot of community effort. The KiCad downloads page offers three different ways to install a native KiCad. The "official" way is to use the shell script (poorly) maintained on GitHub. It hasn't been updated for three months and doesn't work anymore but the comments in the Issues page do point to working solutions. In summary use this. BUT make sure you have all the prerequisites installed (using Homebrew) and then use brew to uninstall 'xz' (you can re-install it after build is complete). (Damned if I know why build crashes if 'xz' is there but it does.) And don't forget to install Doxygen which isn't mentioned in the prerequisite list. And don't use the multi-CPU option on Mac. Seems 'make' has a problem on Mac.

So after a successful install of KiCad in /usr/local/bin my troubles really began. Seems that KiCad changed it's library format sometime in the past year and most of the docos refer to the old library but all the code expects the new format. Also on Mac the "home" directory is all over the place. Sometimes it really is $HOME. Other times it is $HOME/Library. Libraries can supposedly be downloaded in realtime from GitHub but it never worked for me so then you have to install them locally. There's a script on EEVblog that will do this. Eventually I got a working version of KiCad on my MBA. Working through the Blinky videos quickly showed up the limitations of using an MBA trackpad instead of a 2-button mouse with track wheel. Zooming takes a long time to adjust to. In the end I tended to use function keys. And it takes a while to work out which keys correspond to the Linux keys. You can save yourself some grief by installing MiddleClick now. But I don't want to use a mouse. So I persisted with using the trackpad for zooming and eventually I started to get the hang of it. But it will never be as smooth as a mouse wheel.

You can also install a compiled version of KiCad from here or here but both are fairly old versions of KiCad and both use the old library format.

The deal breaker for me was when I was placing tracks on the PCB and wanted to change the grid because the steps were too coarse. You can't change the grid on the Mac version. Neither of the compiled versions worked either. End of story!

Linux VM

From all the comments I read in my searches, the Linux version appears to get the best support so a second approach is to install VirtualBox, install Ubuntu on VirtualBox then install the latest and greatest KiCad using apt-get. This is a really straightforward approach with no surprises. It simply worked. BUT Ubuntu in VirtualBox is painfully slow in swapping applications. It does a really tedious animation of drawing small windows of the running apps and then one clicks on one to display it. And the disk image takes up 7.4GB whereas the native app takes only a couple hundred MB. And VirtualBox obviously uses a lot of OS resources. Firefox in VB was particularly slow. So almost usable but not quite. An added bonus is that I subscribed to the nightly updater for KiCad so bugs were being fixed quickly.

Wine

The tediously slow Ubuntu on VB plus the sheer size of the disk image made me think that perhaps I could try running a Windows image of KiCad under Wine. I use Wineskin Winery to install Win apps on my MBA. The task in this case is to find a Win installer for KiCad. There's plenty of scripts to compile it but not many actual install .exe's. I could only find the one mentioned on KiCad's download page and it's nearly a year old which for a fast changing product like KiCad is really old. However it installed simply and it works (almost). The main problem once again is the trackpad on my MBA. I'm so used to using taps instead of clicks but this version of KiCad doesn't seem to handle the trackpad well. Sometimes I have to double tap, other times tapping won't work at all and then I have to click but sometimes I have to double-click all for the same task. Very confusing. Also when drawing the PCB I had to move components out of the way to get access to components underneath. Not sure if this is simply an old version problem or a trackpad issue. Other versions pop up a window asking me to choose which component or track I want to move/edit/etc. And sometimes this version does pop up the window. But not consistently. As for disk image size, the Wine version clocks in at 790MB. But the older version might not have used up as much space. Hard to tell.

But none works well

So I can't decide which version to go with. A recent Win installer would be worth a try if I can find one. Maybe running the Linux version under a smaller distribution would be worth a try. TinyCoreLinux in VirtualBox is currently using 2.4MB(!) on my MBA. (I've been using TCL as part of the boot2docker app.) However the nightly compiles of KiCad are done under Ubuntu so there is a lot of advantage in using the "standard" environment. Maybe I can get rid of Ubuntu's painful animations. It's the first thing I do on Mac OS X. Or maybe the Mac native app will get grid change working...

In summary, none of the Mac alternatives works well. If I were going to use KiCad seriously I think a separate machine running Ubuntu would be a good idea and a mouse with track wheel seems almost compulsory.

As for KiCad itself, I really like it. I found each of it's stages easy to follow and was (eventually) able to get to a set of Gerber files for Blinky ready for manufacture. Dave Jones of EEVblog's opinion notwithstanding, KiCad is a serious contender for Open Source hardware developers. I just wish I could get a usable version for Mac OS X.

Update

I tweaked Ubuntu using the tips here (especially the gnome-session-flashback tip) and now KiCad screams along! (Cannot understand how or why anyone would release such a crippled desktop version.) Really smooth use of all sections. Even the trackpad scrolling is usable now (barely) but I still prefer F1 and F2 keys. I'm looking forward to some productive KiCad use on my MBA now.


Monday, April 14, 2014

Sorting disk usage, Perl to the rescue.

My MacBook Air was close to full up after a lot of music editting (see results at http://www.stmaryssingers.com/recordings.html ). In the past I would run

du -s * | sort -rn

to identify the biggest users of disk space giving for example:
14104880        iTunes
683304  Noël Français
682592  sibs
221488  DancingDay
93664   MissaAlmePater
27480   ChristmasLullaby
17680   JesuJoy
13656   MissaBenedicamus
12256   MassInHonorOfSaintJoseph
8760    pdfs
3584    ChrissyCarols
2456    LookingAtTheStars
1352    BriggsMass
408     Bach-Jesu
264     40_The_First_Nowell.sib
64      Thou-knowest-Lord-Z-58b.pdf
24      StMS20140412
24      StMS20140322
24      StMS20140309
24      StMS20140208
24      StMS20131214
0       GarageBand


and the numbers are in 512-byte blocks. Nowadays that's a lot of digits to decipher in the listing. So I started using the 'h' ('human readable') option:

du -sh * | sort -rn

giving:
676K    BriggsMass
334M    Noël Français
333M    sibs
204K    Bach-Jesu
132K    40_The_First_Nowell.sib
108M    DancingDay
 46M    MissaAlmePater
 32K    Thou-knowest-Lord-Z-58b.pdf
 13M    ChristmasLullaby
 12K    StMS20140412
 12K    StMS20140322
 12K    StMS20140309
 12K    StMS20140208
 12K    StMS20131214
8.6M    JesuJoy
6.7M    MissaBenedicamus
6.7G    iTunes
6.0M    MassInHonorOfSaintJoseph
4.3M    pdfs
1.8M    ChrissyCarols
1.2M    LookingAtTheStars

  0B    GarageBand
Unfortunately sort doesn't know how to sort the unit suffixes. But Perl does. It's a while since I used the Schwartzian Transform but it seems perfect for the task. I copied the Wiki code into a file, dusort.pl, which I placed in a directory in my PATH variable (~/bin in this case) and modified the regex extraction to make it sort by unit suffix first and then by number giving this:

#!/usr/bin/env perl 
use 5.010;

my $size = {P => 6, T => 5, G => 4, M => 3, K => 2, B => 1};
print
  map { $_->[0] }
  sort {
  $size->{$b->[2]} <=> $size->{$a->[2]}
                  ||
        $b->[1] <=> $a->[1]
  }
  map { [$_, /^([ \.0-9]{3,4})([PTGMKB])\t/] }
  <>;

and now when I run the command(s):
du -sh * | dusort.pl
I get the result:
6.7G    iTunes
334M    Noël Français
333M    sibs
108M    DancingDay
 46M    MissaAlmePater
 13M    ChristmasLullaby
8.6M    JesuJoy
6.7M    MissaBenedicamus
6.0M    MassInHonorOfSaintJoseph
4.3M    pdfs
1.8M    ChrissyCarols
1.2M    LookingAtTheStars
676K    BriggsMass
204K    Bach-Jesu
132K    40_The_First_Nowell.sib
 32K    Thou-knowest-Lord-Z-58b.pdf
 12K    StMS20131214
 12K    StMS20140208
 12K    StMS20140309
 12K    StMS20140322
 12K    StMS20140412
  0B    GarageBand
Obviously the next thing to do is to make a shell alias:
alias dus='du -sh * | dusort.pl'
and now I even save a few keystrokes in my task to pinpoint the Biggest (L)User.



Monday, April 7, 2014

Threes! and Threesus A.I.

I decided to try the Threes! app/game on my iPhone. I got to around 21,000 just playing by myself but I went Googling to see if there were ways to improve my score. Found quite a few "tips" articles had discovered most of what I was doing. (No-one mentions how to speed up the score totalling at the end of the game. (Swipe the screen a second or third time. Saves seconds per game.))

But then I discovered Threesus, watched the video for nearly 30 minutes in total awe and decided I would download Threesus and use the Assistant to help me get my score higher.

And once again it became another exercise in yak-shaving...

Threesus is written in C#, Microsoft's attempt at Java+Objective-C+... and is apparently pretty-well compulsory if you are programming on Windows. The author states that one can install Visual Studio Express for Windows Desktop and Threesus should compile and run in commandline Assistant mode.

Well, I don't have a Win box sitting around but I do have Parallels on my MacBookAir so I downloaded and installed Express and was able to fire up ThreesusAssistant very quickly. I had to read the source code to work out the commands but they are simple enough.

What intrigued me was the mention of the 'bonus' cards and how they are indicated by a '+' sign on the card. I'd never seen the '+' sign. The bonus cards were just like the normal '3' card on my version of Threes. Walt Destler has also blogged a high-level overview of Threesus and he refers to a detailed description of the Threes! algorithm and that's when I realised my copy of Threes! was out of date. Version 1.0.3 introduced the '+' sign on bonus cards.

So after a quick update I was able to use the Assistant to get my score upto around 28,000.

But disaster hit! The Assistant failed with a null reference error before I could get a higher score. I didn't fancy attempting to debug a run-time error in a language I don't know using an operating system I don't know.

Destler also mentions Xamarin Studio as a possible alternative for Mac users. I duly installed it and spent quite a few hours attempting to get Threesus to compile and run. Lots of googling for obscure errors, settings etc. etc. The usual yak-shaving for any new software.

At last I got Threesus to start in Xamarin. I input the initial grid data and it commenced "thinking". About 5 minutes later it returned the next move. Then it took 10 minutes for the next move. No way is Xamarin/Mono usable on Mac OSX. The Assistant is already slow enough in Visual Studio on Parallels but it seems to be 100s of times slower in Xamarin on Mac.

So now I have a problem. Any sensible person would give up at this stage and find something useful to do.

But not me. I threw a few dollars into the RPerl Kickstarter project last year in the hope that they could get further than a faster bubblesort and it seems to be coming along nicely. Perhaps it's ready to take on a Threes! playing A.I.? Once again a lot of yak-shaving to get the source downloaded and installed but it appears to be working. Now to see if I can translate/recode C# code into working Perl and then possibly speeding it up with RPerl. RPerl uses the Perl Inline module which also allows raw C or C++ to be in-lined into Perl code. That might save a lot of hassle.

I also realised that this could be a nice project to try out OCRing the iPad screen of the Threes! game grid. I don't want to build a full game-playing robot but the slowest and most error-prone part of using the Assistant is inputting the changes. If I could add a screen-reading section so that the Assistant merely has to output Left, Right, Up or Down it would speed manual play greatly. And while I'm at it, why not use a text to speech converter so I don't even have to look at the Assistant's output. (Time to start lobbying the game developer to add speech input!)  I've got an old iPhone 3GS to supply video input and there are some great articles on OCRing Sudoku grids from newspapers etc. The Threes! grid is very similar. So this might turn out to be a very elaborate but maybe quite fun project. Or a total disaster and time-sink...