Thursday, November 21, 2013

ReadyNAS RN104 is a disaster.

You get what you pay for they say. In my upset about almost losing a year's data from my ReadyNAS NV+ I purchased an RN104, Netgear's latest version of "consumer level" NAS. I was kind of hoping that after all these years of development the system software would be more stable than that on the NV+.

After much googling and angst I was finally able to copy my data from the stalled NV+ onto my new 104 and I thought it was all plain sailing after that.

Three weeks later and I'm ready to demand my money back for the 104. It has stalled at least once a day for the past three weeks, sometimes twice, even three times. Thankfully redundancy in the disks has allowed resynching (if it doesn't stall first) so I haven't lost any data (yet) but basically I've got a very power hungry brick. Obviously its performance as a file server is very poor while it's resynching.

I don't think Netgear support has the slightest clue about why it's happening. The ReadyNAS forum  is full of complaints about the stalling. I've filed a support request but so far none of the replies has indicated they have any idea why it stalls.

It might be snapshots, it might be automated backups, it might be power supply under load, it might be the anti-virus software (not enabled in my case), it might be CPU too hot under load...

At the moment my old NV+ is sitting next to the 104 looking very smug. At least it only has unrecoverable stalls once a year or so...

As an indicator of how clueless Netgear support people are, I posted a complaint on the forum about the iTunes server in OS6 being compiled to serve only Windows boxes. It serves up FLAC files with the bytes reversed for a Mac and all one hears is a deafening squeal. "Thanks for the heads up" was the reply from Netgear. They never tested it!

Tuesday, October 29, 2013

Mounting Sparc-based ReadyNAS Drives on a Raspberry Pi

dbott at http://home.bott.ca/webserver/?p=306 gives a simple process for mounting ReadyNAS disks on a Linux box. Unfortunately, that Linux box is presumed to be a Debian Linux PC with at least a few spare disk trays.

In my case, my ReadyNAS NV+ stalled and wouldn't reboot. I installed the latest firmware (4.1.12) a couple of days ago and I am thus suspicious about the cause of the stalling...

However it was obvious from the way the boot process happened that the disks were OK. The NV+ simply refused to read them. I was fed up. A similar event happened just over a year ago and I am tired of this potentially complete loss of data. In the light of my experience last year I at least had a full backup on my old Drobo 1 but the backup itself took so long I was only running it monthly.

I googled for info about reading ReadyNAS Sparc disks and found the above link. I don't have a spare Linux box sitting around. I do, however, have a Raspberry Pi which was running my backups to the Drobo and was thus doing nothing.

Courtesy of dbott's notes, I realised I had to install fuse-ext2 and lvm2 on the RPi in order to read the disks. I used apt-get to install lvm2 but there didn't seem to be a .deb package for fuse-ext2.

I googled once more and found this:
http://loveraspberrypi.blogspot.com.au/2013/07/modify-raspberian-image-file-on-mac-osx.html which is actually the reverse of what I want. I want to read an ext2 disk on an RPi. This article tells how to read an RPi SD card on a Mac. However it contains the critical link to the fuse-ext2 source:
http://alperakcan.net/?open=projects&project=fuse-ext2. And then I had to install the GNU devel toolchain (apt-get install autoconf automake libtool libfuse-dev ext2fs-dev).

Finally the configure/make/install process succeeded and I thought I had plain sailing from there. I put disk 1 from the NV+ into a USB docking station and connected it to the RPi via a powered USB hub connected to one of the USB ports. Then I followed dbott's instructions. Unfortunately vgscan told me there were two disks missing from the set. Which was correct of course. I needed at least three of the four disks from the NV+ to make the virtual volume.

After a hasty visit to my local computer parts shop, I installed the remaining two disks using SATA to USB cables connected to the hub and was delighted to run vgscan and see the three disks.

Finally I ran
fuse-ext2 -o ro -o sync_read /dev/c/c /mnt/media
and was delighted to see the "unreadable" volume fully accessible with all my files listed.

In my disgust :) I went out and bought a ReadyNAS RN104. I'm quite happy with the Netgear hardware. I just hate their crappy (firm|soft)ware. So I'm hopeful that the latest firmware (OS6) will be a bit more stable than that on the NV+. I'm currently letting the RN104 format some disks and then I hope to be able to transfer all the files from the USB disks to the RN104. I'll add an update if/when it completes.

Update: Initially I tried a simple 'cp -r' from the llvm filesystem on the RPi to the RN104 filesystem. It stalled. So, realising that there was quite a high probability of the copy stalling again, I'm using rsync because it can pick up where it stopped if the llvm volume disappears, which for some reason, it seems to do occasionally. (Actually I suspect the cause of the stalling is the WD Green disks spinning down to "save power" and then not recovering quick enough when they are needed. This might also have caused the NV+ problems. Not sure how to verify this. The WD Greens are definitely not on the recommended list for the RN104.) I'm about 1/3rd through the data transfer and am listening to some of my music from the new server so it looks like I'll be able to pick up without data loss. Phew! what a relief!

Update2: I finally discovered that one of the power packs for the SATA to USB cables was faulty. Actually, the pack was OK. But two of the connecting cables were faulty. What can I expect for $20? Anyway, I swapped the mains cable and fixed the moving pin in the 12v/5v connector. And the uploads have been much more stable since. But the RPi still occasionally stalls. And I have to pull the power plug to restart the RPi and twice the SD card was corrupted. I had to re-copy the image to the SD card using dd. Restore still not completed but the most important files have copied.

Wednesday, June 19, 2013

Direct Digital Synthesis (DDS) on the GA144 (Part 2)

I found out why I couldn't get two nodes to run with my start script. I was too impatient. I'd forgot that I had the serial connection to the Eval board running at only 115kbps. The command to install the code in the various nodes visits all the nodes and it was simply taking forever and I assumed the IDE had stalled.

Anyway, I took the delay word out of node 617 and put a simple counting loop in node 616 which asserts the 'right' port (which is common to the nodes) when the loop finishes. Then node 617 waits till its 'right' port is asserted, then continues with code to load the sine value into the DAC port.

This makes the sine wave much more stable. With the value of 500 in the go loop the sine wave is precisely 799.9Hz and stays rock solid on that value. I doubt if a crystal would make it much more stable but I'll try a crystal next.

With the following code, make sure 866 and 868 are loaded in block 200 to ensure they are compiled. Then '870 load' will start the app.

866 list 
sine wave generator,
617 node 0 org,
0 , 40 , 80 , 120 , 170 ,,
220 , 280 , 370 , 511 ,,
hart 3300; -.00433 .07943
            -.64589 1.57079,
cos tri 2* 2* . triangle
    dup *. 2 poly
  
 -281 , 5203 , -42329 , 37407 ,
  
 push drop pop *. + ;
scaled
 2/ 8000 . +
       8191 12 interp ;
dac! io b! 155 or !b ;
start
 1f 128 phaseinc
  
 dup dup or
  
 begin
  
 dup cos scaled
  
 synch right b! @b drop
  
 dac!
  
 over . +
  
 end


868 list 
timer count cycles,

616 node 0 org,
hold for . . unext ;,
ms for 27063
     for . . . unext
  next;

start 08 right b! 1 ms,
go 500 hold !b go ;

870 list 
sine wave loader

host load loader load
using default ide paths
kill boots
0 708 hook 0 -hook
setup application
616 +node 616 /ram 8 /p
617 +node 617 /ram 1f /p
visit sine path

panel pause 2 ship

Tuesday, June 18, 2013

Direct Digital Synthesis (DDS) on the GA144

I wanted a simple demo of the GA144 producing sound. I found a sample DDS app for a SeaForth chip (predecessor to the GA144) but the code syntax is sufficiently different that I couldn't simply copy it.

The concept is simple enough. Calculate the values of a sine wave over a fixed time interval and load those values into a digital-to-analog converter (DAC). Connect the DAC to a loudspeaker and listen to the pretty tone.

Rather than look up tables of (co-)sine values, which can take up lots of memory, this app chooses to approximate the values at each interval using relatively fast calculations. The code takes much less room and easily fits into the limitations of the F18 nodes in the GA144.

But it took me a couple of days to understand the code. It all resolved to two words: cos and scaled. cos in turn uses three other words: triangle, poly and *. while scaled uses interp. Now I knew that each of those words is part of the GA144 ROM library. There was also an intriguing comment in cos: 'Hart, 3300'.

So I googled for 'Hart cosine' and one of the first hits was a page by Chuck Moore about sines and cosines in colorForth. This page also explained the 'Hart, 3300' comment, viz., Computer Approximations by John F Hart, Wiley 1968; function 3300. I googled for 'Computer Approximations Hart' and found that the only copy I could buy was hardcover from Amazon. However I also found a link to an electronic copy which allowed me to read the theory behind the magic numbers that turn up in the code.

And then I (quite by accident) discovered the shadow page for the poly word on the GA144 which contains the exact same code as the cos method but translated into colorForth.

cos f-f'
hart 3300
-0.0043 0.0794 -0.6459 0.5708
  
 2* 2* . triangle dup *. 2 poly
  
 -281 , 5203 , -42329 , 37407 ,
  
 push drop pop *. + ;                       

So now I needed to find out how to use the interp word to scale the cosine values into values between 0 and 511. Lots of test code and simulations later...

The final code was this:
866 list 
sine wave generator,
617 node 0 org,
0 , 40 , 80 , 120 , 170 ,,
220 , 280 , 370 , 511 ,,
hart 3300; -.00433 .07943
          -.64589 1.57079,
cos tri 2* 2* . triangle
   dup *. 2 poly
  
 -281 , 5203 , -42329 , 37407 ,
  
 push drop pop *. + ;
scaled
 2/ 8000 . + 8191 12 
       interp ;
dac! io b! 155 or !b ;
delay
 700 1 for . . unext ;
start
 22 128 phaseinc
  
 dup dup or
  
 begin
  
 dup cos scaled
  
 delay dac!
  
 over . +
  
 end
A couple of explanations. The delay word in the original was another node running a single timing loop and raising a signal on a comms port at the end. This in turn allows the main node to synchronise with a (relatively) accurate clock signal. However I ran into problems starting multiple nodes on the chip so I brought the synch delay into the node, preferring to sacrifice some accuracy for simplicity. The constant, 700, controls the amount of delay between DAC updates and, hence, the frequency. 500 gives around 550Hz, 700 gives around 447Hz.

interp expects an interpolation table at address 0. In this case it's not quite a linear interpolation. (Refer to DB001 F18A Technology Reference for details.) scaled halves the cos value, adds 0.5 (to scale into positive numbers only) and calls interp with s and m calculated for L=16 bits and n=3 (a 9-entry table i.e. 2**3+1) to interpolate the values between 0 and 511. dac! sets the DAC value on the node.

The output of the DAC needs to connect to a resistive load. The EVB001 kit includes some 47ohm resistors so I soldered one between the DAC pin and earth and then I was able to see a (relatively) clean sine wave output on my 'scope. It also sounded like a sine wave when I connected a speaker.

Make sure block 200 contains a load of block 866 so 'compile' will include it. The block to start the app on the GA144 (copied from 9.4 of the Users Guide) is:

872 list 
demo ide boot empty compile serial load
customize -canon 0 fh orgn !
a-com sport ! a-bps bps ! !nam
run talk 0 617 hook 0 64 617 boot
  
 upd ?ram panel 22 call ;

To load and run type '872 load' and 'run' in the IDE.

My next task is to get a second node working as a timer to increase the accuracy of the sine wave (possibly using a crystal?). Then I want to port another VentureForth app, Musicbox, to the GA144.

Tuesday, June 11, 2013

Combsort for Softsim

Chuck Moore has put a bubblesort script on one of his colorForth pages:

bubble nt a! for @+ @ - + - -if
@ a -1 + a! @ push !+ pop ! then drop next ;


Simple and compact. I was interested to see if such a simple sort could be speeded up without using too many more instructions. I found the Wikipedia entry for Combsort which calculates a decreasing gap between the two comparison values with the aim of getting the smallest values moved into place as quickly as possible.

So I started transliterating the Wiki algorithm into colorForth. The gap is shrunk by a factor of 1.3 which is apparently a result of some extensive testing with varying shrink factors. Initially I wrote a subroutine to multiply by 10 then divide by 13 (plus lots of shuffling registers around):
shrinkgap a push dup dup or over
   10 * -13 --u/mod pop a! push
   drop drop drop pop ;

And then it hit me that 1.3 is only an approximation. There's nothing absolute about it. It occurred to me that if would be much more efficient to use factors of 2 where possible. So my shrink calculator is simply:
2/ dup 2/ +
In other words take (1/2+1/4) (i.e. 0.75) of the gap. 1/1.3 = 0.77 which is amazingly close to 0.75.

In the end my Combsort still took lots of code and only saved a few hundred cycles compared to Chuck's bubblesort.

comb sort,
504 node 0 org,
4 , 19 , 7 , 1 , 12 , 13 , 0 ,,
11 , 3 , 9 ,,17 , 16 , 13 , 6 ,,
2 , 1ffff , 5 ,,
shrink 11 2/ dup 2/ + if ;
   then - 2* - ;
comb
 tn push dup dup,
...
begin,
......
16 pop dup push a!,
......
dup or swpd push,
......
shrink,
......
over over - . +,
......
for dup a + b!,
......
@ - 1 + @b + -if,
......
gt 20 @ @b ! !b pop pop,
......
dup or - push push,
......
le then drop @+ drop next,
...
26 dup dup dup or - +,
......
if dup or - then gap1?,
......
pop swpd?,
......
over - and or -,
...
until pop ;,
2e 17 0 comb 32 r---


Friday, June 7, 2013

Quicksort and GreenArrays' arrayForth simulator from a beginner's perspective.

I purchased an EVB001 eval board for the GA144 chip from GreenArrays a year ago. I wasn't sure what I wanted to do with it at the time but eventually I realised the GA144 might be more appropriate for low power digital signal processing apps than the FPGA apps I'd been developing.

A couple of months ago I started trying to program it. Conceptually the GA144 seems simple: 144 asynchronous cores, CPU instruction set is a variant of Forth called colorForth, multiple ADCs, DACs and I/O ports. What could go wrong? :)

Well, for starters I hadn't written in Forth before. One of my first jobs (about 35 years ago) was writing assembly language code for 8080 and Z80 chips and I remembered many of the problems with low-level programming. Memory management was paramount. Speed of operation was next most important.

I later became a C programmer. Once again 90% of my time was spent managing memory. When I discovered a language called Perl in 1989 which had automatic memory management I was in Heaven.

A recent blog reminded me of the sheer elegance Perl is capable of. Here is a minimal Quicksort script that had me simply gazing in admiration:

sub quicksort {
    @_ and return (
        quicksort (grep { $_ < $_[0] } @_),
        grep ({ $_ == $_[0] } @_),
        quicksort (grep { $_ > $_[0] } @_)
    );
    ();
}

$, = ','; say quicksort(10,2,5,3,1,9,7,4,2,3,4,8,6);


The algorithm takes full advantage of Perl's automated memory management. Copies of lists created by the greps and the recursive calls to quicksort probably  eat up huge chunks of RAM. But the "simplicity" and "obviousness" of the algorithm are outstanding.

So could I implement Quicksort in colorForth in the arrayForth IDE simulator?

A quick scan of the online literature brought up this. The problem for me is that it is written in "standard" Forth. I hadn't looked at Forth until about a year ago and to complicate matters colorForth and Forth have almost nothing in common syntactically. Even when the instructions are the same (e.g. 'or') they do different things ('or' is "inclusive OR" in Forth, "exclusive OR" in colorForth). So unless I was prepared to thoroughly learn Forth in addition to colorForth there was no way I could use Will Baden's algorithm.

So I decided to implement the algorithm from scratch using the GreenArrays manuals and user guides.

I quickly discovered I was back in memory management hell.

Programming the GA144 is mostly about memory management. Each core (GreenArrays call it a 'node') has only 64 18-bit words of RAM plus 64 18-bit words of ROM library. The 32 colorForth instructions are sufficient to implement a viable version of Forth. There is a 10-word data stack and a 9-word return stack, a  general purpose register, A, and a write-only B register used mostly to talk to the I/O ports but which can address all of RAM and ROM.

There are 144 nodes in the GA144 chip running asynchronously but able to synchronise with the other nodes via the I/O port communication mechanism.

So the first decision I had to make was whether to use one node or multiple? The Perl example used only 13 elements in the list and since Quicksort uses O(log N) stack elements it seemed possible to fit the code and the data in the same node. (The example below runs out of stack space with 18 or more elements in the data array. It also runs out of instruction space with more than 19 elements.)

The second decision was to follow the wikipedia example using in-place element exchanges rather than creating multiple sub-lists for later merging as used in the Perl example.

The following code is mostly a line-for-line transliteration of the wikipedia example. I did steal one idea from Will Baden's example. I used his midpoint pivot selection code to bypass Quicksort's weakness when sorting already sorted lists.

The arrayForth IDE is a PITA. It's a steep learning curve for a total beginner like me. And it can't import code from outside sources. Every character has to be hand-typed. So "literate" documentation is virtually impossible and largely a waste of time. But eventually I found that the ability to quickly change an instruction and see its effect in the Softsim simulator provided me with all I needed to learn the colorForth instruction set. Admittedly the IDE won't teach me the idioms Forth programmers have used for decades. Nor will it teach me the idioms colorForth programmers have discovered. But, again, the IDE contains the full source code for colorForth, arrayForth, polyForth and a heap of tests displaying how one uses colorForth in "real life".

I also found useful idioms in Chuck Moore's notes about colorForth's instructions.

And GreenArrays' email support has been superb. Every question of mine, no matter how dumb, has been answered quickly and comprehensively.

The program code below is in coloured text (red for method definitions, green for instructions and yellow for data) and I've interspersed my comments in black text.

The code and data fill 56 of the available 64 words of RAM. Not much room left for experimentation. The starting line (at the end of the listing) loads the data stack with the beginning and ending address of the data array and calls qsort which calls part to divide the data array into partitions of smaller and larger elements around a pivot value and then recursively calls itself on each of these subarrays.

part partitions the array into three sections. The "left" section contains all array elements less than or equal to the pivot value. The "pivot" section contains the pivot value. And the "right" section contains all array elements greater than the pivot value. At completion the data stack contains (left, right, pivot) addresses. (Pivot address is in T).

But most of my time was spent trying to keep the use of the data stack to a minimum because of the recursive calls to qsort. So often I had to store data in A or R rather than leaving it on the data stack for fear that a recursive call would push previous start and end addresses off the end of the 10-element stack. In fact this does happen when there's more than 18 elements in the data array.

503 node 0 org
I picked node (CPU) 503 randomly. Nothing special about it.
19 , 4 , 7 , 1 , 12 , 13 , 0 , 11 , 3 , 9 ,
17 , 16 , 10 , 6 , 2 , 13 , 5 ,
I put the data array at the start because I always know its start address (0).

setab over over b! a! ;
Duplicate S and T then move T into B and S into A.

exch setab @ @b ! !b ;
Array addresses are in S and T. Call setab to load A and B with array element addresses, then swap array elements pointed to by A and B.

1+ 1 . + ; 1- -1 . + ;
Increment/decrement T.

part exch drop
Exchange the pivot element (address in T) with the end element (address in S). The pivot address in T can now be dropped from the stack as it is no longer needed.
   over over - . + 1+ - push
S and T contain start and end addresses of array to sort. Calculate length of array less 1 for loop counter. Push loop cntr into R register.
   setab over
setab puts array start address (S) into A and end address (T) into B. The end array element contains the pivot value. And at start of loop S is also the address of where values less than or equal to the pivot value will be swapped into (referred to as sx, switchindex, hereafter) so put a copy of sx into T (over).
   begin
      push a
Save sx onto return stack (nowhere else to safely store it) and load A into data stack
      @ @b - . +
Load contents pointed to by A onto data stack, load contents pointed to by B into stack, subtract.
      -if drop pop exch 1+ dup push
If T is negative *A (array element pointed to by A) is less than or equal to *SX (pivot value, pointed to by SX) so pop SX into T, exchange *A and *SX, increment SX and push it back into R. The dup creates a dummy value for the next line to drop
      then drop
Drops either result of subtract or dummy value.
      1+ a! dup b! pop
Drop leaves A in T. Increment T and store it in A so A points to next element in array. T now holds the value of B. Make a copy and pop it into B because the element exchange destroys the previous contents of B. Then pop SX into T.
   next exch ;
When loop finishes T holds pointer to pivot value and S holds SX pointer. So can swap pivot value into its correct place prior to return. After exchange stack holds (left, right, sx).

qsort over - over . +
Duplicate the start and end addresses and subtract the start address from the end address.
   -if drop ;
If the array addresses are equal there is nothing more to sort so return with stack effectively unchanged by dropping the result of the subtraction.
   then 2/ push over pop . +
If the array addresses are unequal there is something to sort. After the subtraction T contains the length of the array so halve it with a left shift (2/) and add it to the start address giving the address of the midpoint of the array.
   part
part is called with the data stack containing (left, right, mid) addresses.
   a! push a 1- a 1+ pop
part returns (left, right, sx) so pop SX into A, push right index into R, retrieve A, decrement it, retrieve another copy of A, increment it and then pop R into T. This leaves (left, pivot-1, pivot+1, right) in stack.
   qsort drop drop qsort ;
First call to qsort will sort right partition. Returns when right partition is (recursively) sorted. So upon return drop pivot+1 and right. Second call to qsort will (recursively) sort left partition. Upon return whole array is sorted.

0 16 qsort r---
Call qsort with start and end addresses of data array.

Code without comments:
503 node 0 org
19 , 4 , 7 , 1 , 12 , 13 , 0 , 11 , 3 , 9 ,
17 , 16 , 10 , 6 , 2 , 13 , 5 ,
setab over over b! a! ;
exch setab @ @b ! !b ;
1+ 1 . + ; 1- -1 . + ;
part exch drop
   over over - . + 1+ - push
   setab over
   begin
      push a
      @ @b - . +
      -if drop pop exch 1+ dup push
      then drop
      1+ a! dup b! pop
   next exch ;
qsort over - over . +
   -if drop ;
   then 2/ push over pop . +
   part
   a! push a 1- a 1+ pop
   qsort drop drop qsort ;
0 16 qsort r---

I wonder what a multi-core version of this algorithm would look like? And it's also pretty obvious that a recursive algorithm is not feasible with such a limited data stack. Might try a comb sort next.

Saturday, April 13, 2013

Vim assistance for linking index entries

I thought the macros I'd written to assist indexing the epubs I'm creating were sufficient. But the latest book I'm converting has 2883 index entries. It took me an hour to index 100 and I'm not prepared to waste another 27 hours for the rest.

So I need more help from Vim. I purchased Steve Losh's Learn Vimscript the Hard Way (because he's been very generous in writing up what he's learned about Vim and VimScript) to learn a bit more about VimScript. I suspect what I need to write is more a case of RTFM than needing any "hidden", "secret" tips but reading the epub has inspired me to proceed with some experimentation.

The macro/keymapping has to do the following:
  • Starting with two panes
    • kwiclist open in 1st
    • MD file open in 2nd
  • Keymap to select line number under cursor in kwiclist and jump to corresponding line in MD file, select keyword and wrap it with scan tag then return to kwiclist pane.
    • must make wrap macro only wrap currently selected text
      • i.e. don't mess with any other wraps on line
  • Fix current tag wrap keymap to only wrap current selection
Being laid low with the 'flu, I was forced to spend a lot of time flat on my back in bed. MacBook Air to the rescue. In many ways writing a script for Vim is the ideal programming environment. I was able to add features keystroke by keystroke and test them. It allowed me to debug and clarify functions and operations of VimScript which I either didn't know or couldn't work out from the documentation. The immediately available help function is also immensely useful.

Anyway, I cheated a lot in terms of good practice. The suggested procedure is to define a Vim function then call its various methods via keymappings. Too long; couldn't wait. In the end I set up function keys F5, F6 and F8 as a chord that allows me to skip through the links quite quickly.

Vim is opened with the two panes:

5021| 156|  development social|all its forms
4886|4922|153|Device(s)
6213|6249|191|Device(s)
6284|6319|193|Device(s)
6456| 197|           Device(s)|State which l
4447|4484|140|Dialectic
4593|4629|144|Dialectic
============================================
produce a reflective activity, while the pra
automatically, as a result of established ha
<!-- 117 -->

If, now, we replace the <span id="ix6">abstr
or society of agents, this society must be c
<span id="ix540">differentiation of the Agent
They have their being, as agents, in the con

The key mappings are:

noremap <F5> :<c-u>execute "normal!
 \"ayw/\\a/\r:nohlsearch\rve\"by:wincmd
 j\r:let @\"=@a\r:@\"\r:let @/=subst
 itute(@b, '.*','\\L\\0', '')\r-nve"
 <cr>

vnoremap <F6> :s/\%V.*\%V./\='<span
 id="ix'.(@y+setreg('y',@y+1)).'">'.
 submatch(0)."<\/span>"/<cr>

noremap <F8> <c-w>k0+

For future reference, let's decode some of that cryptic VimScript.

Lines in the upper pane of the screen are KWIC (Key Word In Context) references. The lower pane is the Markdown file of the epub I am indexing.

If the KWIC indexer found a keyword in the context given it outputs a line like this:
5021| 156|  development social|all its forms
The 1st field is the linenumber in the Markdown file open in the lower pane. The 2nd field is the 'pagenumber'. Pagenumbers allow the KWIC indexer to define a startline and stop line for context but are no longer relevant for an epub so I enclose them in HTML comment tags. The 3rd field are the keywords that were searched for and the 4th field is the context in which the first keyword was found.
If the KWIC indexer could not find a keyword in the context it outputs a line like this:
4593|4629|144|Dialectic
The fields are: start-linenumber, end-linenumber, pagenumber, keyword. I haven't built any parsing of English into the KWIC indexer so it will not find 'dialectical' even though 'dialectic' occurs in the page.

So the procedure is first of all, set the 'y' register in Vim to the next ID it should insert:
:let @y=123
Then switch to the upper pane and place the cursor at the beginning of the next line to index. If the context of the keyword(s) looks correct i.e. the keyswords appear in the line of text shown, press F5.

F5 does the following;
:<c-u>execute "normal!
Clear the line markers and execute the following as though I were typing the keys
 \"ayw/\\a/\r
Yank the word under the cursor into register 'a'; search forward in the line to the next alpha character (i.e. skip over the numerics, whitespace and vertical bars)
:nohlsearch\r
Turn off the search highlighting (in this case practically everything in the file)
ve\"by:wincmdj\r
Visually highlight the word and yank it into register 'b'; change focus to the lower pane
:let @\"=@a\r
Copy register 'a' into the 'anonymous' register
:@\"\r
Run anonymous register as a command. In this case it means 'goto linenumber'
:let @/=substitute(@b, '.*','\\L\\0', '')\r
Load 'search' register with lower-cased text in register 'b'
-nve"
Skip back one line, search forward for next occurrence of search text and visually highlight the word.

At this point all the other occurrences of the primary keyword will be 'search highlighted' in the pane and I can check if 'visually highlighted' one is correct. If correct, I press F6, if not I skip to the correct occurrence and visually highlight it (i.e. 've') and then I press F6.

F6 does the following:
:s/\%V.*\%V./
Substitute the highlighted text
\='<span id="ix'.
for a 'span' tag with an id of 'ix' followed by
(@y+setreg('y',@y+1)).
the number in register 'y'; increment reg 'y' for next time
'">'.
Close the opening 'span' tag
submatch(0).
Insert the highlighted text
"<\/span>"/
Insert the closing 'span' tag

Finally I press F8 which moves the cursor back to the upper pane and skips down one line, ready for the next F5.

So with my MacBook Air set so that function keys don't need the 'Fn' key as well (default is to require the 'Fn' key to be held), I can leave my fingers hovering over F5, F6 and F8 like a piano chord and press them accordingly. Quite often I have to also hit 'j' to skip over a 2nd or 3rd occurrence of a keyword in a page. Also often the 1st occurrence of the keyword is not the correct one and I have to press 'j' to skip down to the correct occurrence. F5 adjusts accordingly and highlights the later occurrence.

Edit: (Too close to the trees to see the forest) With only two panes, F8 does the exact equivalent of the 'j' key so in actual usage the function keys really are like playing a piano: F5, F6, F8, F8, F8, F5, F6, F8 etc. However, even with all this assistance index linking is still very time-consuming because of all the reading and thinking.

Indexing is at least twice as fast now. I did 400 last night in about an hour and that was after discovering a couple of typos in my index which required correction and re-running the KWIC indexer.

Sunday, March 31, 2013

Using Raspberry Pi for backups

Very annoyed with myself for shutting down my RPi without synching to "disk" i.e. SD card. So it corrupted the disk image and I have to do yet another install from scratch. (Yes, yes, I know: "backup the image when it's stable". I'll learn eventually.)

So once again I followed Naich's great tutorial for setting up a headless RPi. The main improvement I made is to configure DHCP on my router to give the RPi the same IP address every time it requests one.

And the latest Raspian wheezy image seems to enable sshd at boot time, a very useful setting which saves my having to connect a keyboard and monitor to configure it.

In a previous blog I listed the arduous tasks involved in setting up an automatic backup of my ReadyNAS to a largely obsoleted Drobo 1 networked via an Apple Airport Extreme Base Station.

But it turned out that the load on the ReadyNAS was too big. The ReadyNAS is already handling my BitTorrent downloads and music and video streamings. Duplicity broke the camel's back.

The Rpi was sitting around doing nothing so why not make it the backup machine. Because RPi is using the latest Debian it seemed likely that installs of all the apps would be more straightforward.

I decided that duply was a much simpler way to invoke Duplicity so installed both. Also now need to NFS mount ReadyNAS filesystem on RPi as well as mounting Drobo using afpfs-ng.

So using apt-get:

apt-get install duplicity
apt-get install duply
apt-get install afpfs-ng


All installed with no problems. The trick with afpfs is that the userfilesystem device (/dev/fuse) has group 'fuse' and I needed to change my primary group to 'fuse' so I don't get permission errors when I try to mount the Drobo filesystem. Thus:

sudo su
usermod -g fuse myusername


Then I had to reboot the RPi to get this change to stick:

shutdown -r now

Can now mount the Drobo as an AFP filesystem:
mount_afp afp://user:password@192.168.1.2/Drobo /mnt/drobo

Also followed Raspian FAQ to mount NFS filesystem:
sudo apt-get install nfs-common portmap
sudo service rpcbind start
sudo update-rc.d rpcbind enable


ReadyNAS is already set up to NFS export /media and /temp so I added the following to /etc/fstab:

nas:/media /mnt/media nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
nas:/temp /mnt/temp nfs rsize=8192,wsize=8192,timeo=14,intr 0 0


Then mounted the filesystems with:
sudo mount -a

I then followed the duply man page to setup the duply config file in ~/.duply/nasbak:

duply nasbak create

Edit the ~/.duply/nasbak/conf file to set config options. The most important are:
GPG_KEY='disabled'
TARGET='file:///mnt/drobo/backup/nas'
SOURCE='/mnt/media'
MAX_FULLBKP_AGE=1M
VOLSIZE=100
TEMP_DIR=/mnt/temp
ARCH_DIR=/mnt/temp/duply-cache
DUPL_PARAMS="$DUPL_PARAMS --exclude-globbing-filelist $HOME/.duply/nasbak/exclude "


/mnt/temp is an NFS-mounted temp directory on the ReadyNAS. I don't have enough storage on the RPi SD card and Drobo is way too slow for temp files.

exclude file contains:

**.AppleD*
**.DS_Store
**Trash*
**Transmission*
**Temporary*
**TV*
**Movies*
**temp*
**tmp1*


I don't want to backup Movies, TV shows or anything currently being downloaded by Transmission.

Check syntax of config file:
duply nasbak backup --preview

If syntax is OK run the backup (first time will be a full backup):
duply nasbak backup

Monday, February 11, 2013

Frequency counter using Avnet S3A EVK


My love affair with digital electronics continues. I found Mike Field's wiki entry for a frequency counter using an FPGA. I was struck by how inexpensive the DealExtreme 8-digit display Mike used was. $7 including shipping! So of course I had to order one.

Well it arrived last week. Took me a day to find out the connector cable they included was faulty. The first project I tried simply followed Ricardo Batista's Arduino code for the TM1638 chip on the dxdisplay. I used a Freetronics LeoStick I purchased from Jaycar a year ago. Turns out the dxdisplay uses a lot of current, more than my MacBook Air can supply from its USB ports. Luckily I had also purchased Freetronics' PRV28 power regulator at the same time and it was able to supply enough current.

Then I remembered I had an old Avnet Spartan3A Eval Kit I had purchased in 2008 for $39. At the time I had no idea what to do with it so it languished in the junk pile. I fished it out and I was amazed! The components on the board are still (almost) state of the art. FPGA, switching regulators, SPI and BPI flash, capacitive touch buttons etc. etc. Five years later and it's still very modern. Only problem is Avnet and Xilinx have pretty well given up on Spartan 3A so a lot of the support material on their sites was hard to find. And to complicate matters the Cypress PSoC used to program the FPGA was loaded with outdated firmware no longer compatible with more recent software. So the first task was to update the firmware. That took days of Googling and experimentation. Then I was able to use a more recent version of Avnet's AvProg loader software. The exercise of getting Xilinx's ISE Design Suite installed on my laptop was Herculean but now it runs nicely on Parallels Desktop+Win 7.

So it occurred to me that connecting a $39 FPGA kit to a $7 8-digit display to make a 0-99MHz frequency counter might be an interesting exercise. As usual there has been a lot of yak shaving. First of all the clock on the EVK was only 16MHz whereas Mike's code expected 32MHz. He then used a standard DCM clocking module from Xilinx to synthesise a 244MHz clock frequency. Luckily the DCM multiplies upto 16 times so I was able to select 14 to get the same frequency (although I suspect jitter probably invalidates the clock accuracy). Mike also intends his code to use an external GPS module to supply a 1 pulse per second signal used to restart each count. But he pointed out that for less accuracy, simply connecting an internally generated 1pps signal to the 1pps input can suffice for a while.

Code available here.