Pages

Wednesday, 22 June 2016

Three easy ways to avoid off-by-one errors in your software

Here's how to avoid one of the commonest software bugs

If you're in the software business you've probably encountered the dreaded 'off-by-one' error.

An off-by-one error often occurs when an iterative loop iterates one time too many or too few. Someone might use "is less than or equal to" where "is less than" should have been used.

It's also easy to forget that a sequence starts at zero rather than one. To put it another way, we sometimes fail to distinguish between a count (how many are there?) and an offset (how far is that from here?)

Ten gaps in eleven posts
These problems aren't unique to programming.

At the time of the last millennium, there was a lot of discussion as to whether it should have been celebrated on 1st January 2000 or 2001.

In debates about the millennium off-by-one errors may not be too serious. In software they can be fatal.

How can you avoid them?

Here are three techniques which help. I'll save the best till last.

Do manual tests


If you've got a bit of code that is looping or indexing, do some manual tests to check it's doing the right thing. If you're using a language with a REPL, like Python, Ruby, Clojure or APL, you can do the test in no time at all.

Test the boundary conditions. What happens if the loop shouldn't be traversed at all, or if the array you're iterating through has no elements?

 

Do automated tests


Even better, use an automated testing framework like junit, pytest or its equivalent for your language.

Make sure you write tests that check normal operation and operation at the boundary.

 

Prevention is better than cure


If you can, use a language construct or language that eliminates the need to loop.

In Java, Python or Ruby you can use a for-loop to operate on each element of an array. In Python or Ruby you can use generators to lazily collect the results of operating on the elements.

Many other languages now have similar features. And of course, in APL it's very rare to need explicit loops.

Want to add together two APL arrays?

Just type a←b+c

and there's not a loop in sight!

 

Loop-free code is safer 


If the idea of writing loop-free code appeals, take a look at my introduction to APL. It's incomplete, but free for the next few days, and if you 'buy' it you'll get free updates as further sections of the book appear.

Get it here.


Sunday, 19 June 2016

Learn APL on the $5 Raspberry Pi

APL is one of the most productive programming languages ever invented, and it's available free on all models of the Raspberry Pi.

I've just launched an early access version of an Introductory eBook, and it's free for the next seven days.

You should read this book if you want to
• find out what programming in APL is like
• learn how to use the language effectively
• decide if APL is appropriate for your project
• take part in the Dyalog annual APL problem-solving competition

The fast-paced introductory text will teach you the core of the language in a few short, fun sessions.

Once you've finished the book you'll get links to free resources you can use to master the rest of APL's amazing capabilities.

The book is only 30% complete at present, but if you 'buy' the free version now you'll pay nothing for the book, and you'll get free updates as soon as they are published.

I'll start to increase the price next Sunday, and it will continue to go up as the book nears completion.

So don't wait - click here to get it now!

A new/old approach to Parallel processing

Morten Kromberg of Dyalog APL has just published a video of his keynote from the PLDI 2016 ARRAY Workshop.

It's titled Notation for Parallel Thoughts and it describes some exciting innovations in the field of programming for parallel processing.



Thursday, 16 June 2016

More reasons to enter the Dyalog Problem-solving competition

A few weeks ago I mentioned the Dyalog APL annual problem-solving competition.

I've been researching previous contests, and wanted to share my findings.

It's worth entering, even if you don't know APL.

Many of the previous winners learned APL just for the competition. They spent a few days learning the language, and a few more working on the contest problems. Some won top prizes (worth $2000 this year).

It's worth re-entering, even if you entered last year.

Many of the winners had applied before. Some even won prizes in successive years.

It's worth entering wherever you live.

Winners have come from all over the world.

It's worth entering even if you don't win.

I've taken this list of reasons from my forthcoming introduction to APL. (The book should be available in time to help you with your competition entries!)

5 good reasons to learn this powerful language
  1. APL is concise and expressive, so you can try out new ideas very quickly.
  2. APL is focused on arrays, so it will change the way you think about programming.
  3. APL is challenging and fun, so it will stretch your mind.
  4. The APL community is full of bright and helpful people, so you will expand your network of contacts.
  5. The demand for APL developers exceeds the supply, so knowing APL can help you find a job.

 

Sign up today!

You can find out more about the competition on the Dyalog website.

Sign up today.

Monday, 13 June 2016

Help me with a book title and win a Raspberry Pi model 3!

I need a snappy title for a book.

The book is an introduction to the Dyalog implementation of theAPL programming language.

The book is aimed primarily at people learning it on the Raspberry Pi.

APL runs on all models of the Pi, including the £4/$5 Pi zero shown on the right.

You can download a copy of Dyalog APL for the Pi here.

If you submit a title as a comment, if you are the first to submit it, and if I use it, I will send you a Raspberry Pi model 3 complete with a power supply and an SD card.

No royalties, though, and you will need to find a monitor, keyboard and mouse.

If you have a Pi already I will send one to the beneficiary of your choice.


Pi3B - Image (c) the Raspberry Pi Foundation



The book is not yet complete but it should be available in early access format on Leanpub in a few days time.

Have a go - post your title below.

Friday, 3 June 2016

ANNSER - A Neural Network Simulator for Education and Research

I've just launched ANNSER on GitHub.

ANNSER stands for A Neural Network Simulator for Education and Research.

It is licensed under the MIT license, so it can be used for both Open Source and Commercial projects.

ANNSER is just a GitHub skeleton at the moment. I have some unreleased code which I will be committing over the next few days.

I'm hoping that ANNSER will eventually offer the same set of features as established ANN libraries like TensorFlow, Caffe and Torch, and I would like to see a GUI interface to the ANNSER DSL.

ANNSER will be implemented in Dyalog APL. The GUI will probably be implemented in JavaScript and run in a browser. All the code will run on the Raspberry Pi family, though you will be able to use other platforms if you wish.

There's a huge amount of work to complete the project but we should have a useful Iteration 1 within a few weeks.

Why APL?

I have several reasons for choosing APL as the main implementation language.
  1. It's my favourite language. I love Python, and I've used it since the last millennium, but I find APL more expressive, performant and productive.
  2. With APL you can run serious networks on the $5 Raspberry Pi zero. This makes it very attractive for educational users.
  3. APL was created as a language for exposition.
  4. APL is unrivalled in its handling of arrays, and ANN algorithms are naturally expressed as operations on arrays.
Consider the following code fragments. Each creates a random input vector, creates a matrix of random weights and then calculates the output of a sigmoid neuron.

Python version

import random
from math import exp


def random_vector(cols):
    return list([random.random() for i in range(cols)])

def random_vov(rows, cols):
    return list([random_vector(cols) for j in range(rows)])

def dot_product(v1, v2):
    return sum((a*b) for (a,b) in zip(v1, v2))

def inner_product(vov, v2):
    return list([dot_product(v1, v2) for v1 in vov])

def sigmoid(x):
    return 1.0/(1.0+exp(-x))

def sigmoid_neuron(vov, v2):
    return list([sigmoid(x) for x in inner_product(vov, v2)])


mat = random_vov(3, 4)
vec = random_vector(4)
print sigmoid_neuron(mat, vec)

numpy version

from numpy.ma import exp
from numpy.random import random
from numpy import array, inner


def random_vector(cols):
    return array([random() for i in range(cols)])


def random_mat(rows, cols):
    return array([random_vector(cols) for j in range(rows)])


def sigmoid(m, v):
    return 1,0+1.0/(1.0+exp(-inner(m,v)))

mat = random_mat(300, 400)
vec = random_vector(400)
s = sigmoid(mat, vec)

APL

mat ← 0.01×?3 4⍴100
vec ← 0.01×?4⍴100  
sn  ← {÷1+*-⍺+.×⍵}
mat sn vec


I know which I prefer :)

Contributing

If you think you might be interested in joining the ANNSER project, comment below or ping me at romilly dot cocking at gmail dot com

Thursday, 2 June 2016

Mapping Kent Beck's Mind :)

If you don't work in Software you may never have heard of Kent Beck but he's had a huge influence on the way we test and write code.

Yesterday Kent posted a fascinating list on Facebook. He shared some of the key ideas that guide his thinking.

The post is interesting, and stimulating, but it's a wall of text. I love reading, but I also like to think visually, so I started to mind map what he wrote.

It's slowly growing.


The map source (made with Freeplane) and images are now on GitHub.

Kent suggested that this might be the basis of a workshop:

Seems like this could turn into a workshop pretty easily. Spend three days mapping your current ideas, figuring out the holes you want to fill, what you want to eliminate.

Three days sounds a lot, but maybe we could do a shorter version via a google hangout. Anyone interested? If so, please comment.

Tuesday, 31 May 2016

Neural networks on the Raspberry Pi: Sigmoid, tanh and RL neurons

A brief introduction to ANNs - part 3


In the previous post about ANNs we looked at the linear neuron and the perceptron.

Perceptrons have been used in neural networks for decades, but they are not the only type of neuron in use today. When they were first invented, they seemed capable of learning almost anything.

However, in 1969, Minsky and Papert published their book 'Perceptrons' which showed that a single perceptron could never be trained to perform the XOR function. You'll see in the next post why this is so (and why it's not a huge problem), but for now, let's look at three other common neuron models.

Like the linear neuron and perceptron, these start by calculating the weighted sum of their inputs. Recall that you can implement the linear neuron like this:

      ln←{⍺+.×⍵}

sigmoid neuron calculates the same weighted sum of inputs, but then it applies the sigmoid function to the result. The sigmoid function is defined in wikipedia like this:

Here's how you define that function in APL:

      sigmoid←{÷1+*-⍵}

That definition says 'take the reciprocal of 1 plus e to the power minus  ⍵', where ⍵ is the argument to the function.

You can implement a sigmoid neuron by combining the sigmoid function with a linear neuron.

      sn ← sigmoid ln

You can test it like this: 

      inputs←0.2 0.3 0.1
      weights ← 1 2 0.5
      inputs sn weights
0.7005671425


As the name suggests, the sigmoid function is S-shaped. Here is a graph of the function, plotted using Dyalog APL's SharpPlot library:


As you can see, the sigmoid functions value is close to zero for large negative arguments; it has the value 0.5 when its input is zero; and it rises towards one as its input grows larger.

Another commonly used function, with a similar shape, is the tanh function.

APL has implementations of all common trigonometry-related functions. Sin is 1○⍵, and Cos is 2○⍵.  You can find a complete list here.

The definition you need is just

      tanh←{7○⍵}

Here is its graph:


As you can see, tanh ranges from -1 for large negative arguments to +1 for large positive arguments. Its value at zero is zero.

The last neuron we'll consider in this post is the Rectified Linear Neuron or RLN.

The transfer function for this neuron is zero for inputs that are negative or zero, and is equal to the input for inputs that are positive.

Here is the APL definition:

      rln←{0⌈⍵}

The symbol (max) returns the maximum of its arguments. Here's a plot of the RLN function:

I mentioned earlier that the perceptron has some limitations, but why are these other functions popular? A future post will cover back-propagation - one of the most widely used techniques for training an network - and the functions you've been looking at work well for that purpose.

Before then, you'll take another look at the perceptron, you'll see how to train it, and review its limitations and ways of avoiding them.

 



Friday, 27 May 2016

Student? Expert Problem Solver? Win $2000 and a free trip to Glasgow

If you like coding and solving problems, and are a full-time student, you could win up to $2000 and an expenses-paid trip to a conference in Glasgow later this year.

All you need is a computer and some free software. The computer could be a Raspberry Pi (any model) or a laptop running Windows, OS/X or Linux. I'll tell you where to get the APL software further down this post.
 
First, though, a warning. If you enter this competition it could change your life!

I’m serious. Just under fifty years ago I had a chance to learn APL.


I did, and it shaped my whole career. I'm still using APL to research neural networks.

Now, if you want, it’s your turn.


The Dyalog APL 2016 problem solving competition 


Dyalog have just announced their annual APL problems solving competition. They want to introduce more people to this extraordinary, powerful language.

If you are a full time student you could win a big cash prize (up to $2000) and an expenses-paid trip to Glasgow later this year.

If you’re not a student, you can still enter, stretch your mental muscles, and have fun.

In a minute I’ll explain how you can enter, and how you can start getting familiar with the language. Before that I’d like to show you a little of what APL can do and why it’s so powerful.


Meet APL: the most productive programming language I know

 

Suppose you’re a scientist, or an engineer, or an entrepreneur and you need to crunch some numbers. Perhaps you’ve just done an experiment or got some sales figures in. Whatever the background, you have two sets of data:

    expected ← 10 15 13 27 30
  actual ← 9 12 15 25 28

How much do the actual figures differ from what you expected?

     difference←expected-actual
  difference
1 3 ¯2 2 2

What’s the total difference?


   +/difference
6

If you’re into statistics, you might ask for the average difference. You could start by writing a program to calculate averages:

    average ← {(+/⍵)÷⍴⍵} ⍝ divide sum by number of elements
    average difference
1.2

As you can see, APL is powerful and concise. If you want to find out more, and maybe win a cash prize, you should enter the competition today.


5 Steps to enter

 

  • Register for the competition here and click the purple ‘Start the competition’ button.You should get a confirmation email within 10 minutes. If you don’t, check your spam folder. If the email is not there, notify support@sqore.com
  • Install Dyalog APL on your own machine. You can use a Windows, Linux or OS/X laptop or a Raspberry Pi. 
If you're using a Raspberry Pi you can find out how to install the free software here.

If you're using a laptop you will need to get a license from Dyalog; students can get a free educational licence, and anyone else can get a personal licence for a minimum charge. 
  • Work on the problems for phase 1 and phase 2. There's more information here.
  • When you’re ready, submit your entry.
Then wait to hear the results. Make sure you keep October 9-13, 2016 clear in case you win a trip to the Glasgow conference!

 

Free tips by email

 

I’ll be sending out a few tips about getting started (but no solutions!) over the next few days. If you want them, sign up below. I won’t spam you, and you can unsubscribe at any time.

I’m also working on an extra bonus which I hope to offer in one of the emails later this month.

Sign up for the email tips below.




Get Dyalog Competition Tips




Thursday, 19 May 2016

A new Raspberry Pi robot joins the family

Yesterday saw the arrival of a Raspberry Pi robot kit from The Pi Hut, and I'm finding it hard not to drop everything and have a play.

The Pi Hut has close links with CamJam. CamJam is, I think, the first Raspberry Jam, based in the Cambridge area.

Working with The Pi Hut they have created three excellent EduKits: inexpensive, fun kits which introduce Raspberry Pi owners of all owners to the fun of physical computing.

The earlier kits came with excellent instructions and the Robot kit does too. I'm sure I will succumb to temptation and start exploring the kit in the next day or two. Expect a progress report soon.

My immediate priority is more urgent. I'm talking at the BAA meeting tomorrow, and I need to make sure I'm properly prepared.

Dyalog Visit

I nearly blew it earlier this week. I went along to visit my friends at Dyalog to talk about my neural network research and show them APL running on the new Pi zero.

I thought I had taken everything I needed, but I forgot to take a USB hub. I won't repeat that mistake tomorrow, as I expect there will be a lot of interest in the newest member of the Pi family.

BAA meeting tomorrow - 19th May

If you're an APLer, current or lapsed, and can get to central London tomorrow, do come along to the meeting. I think there are a few places left. Go here to book.

Monday, 16 May 2016

The new Raspberry Pi zero is here - and it's snappy!

Spot the difference!
The new Raspberry Pi zero is out and it has a camera connector.

The picture on the right compares the new zero with its predecessor. They are very, very similar but the clever folks at Pi towers have re-routed the board to make room for a camera connector while keeping the size of the board unchanged.

I've had a chance to play with the new Pi for a few days now and I love it. You can read my plans below but the main thing is that the new feature has been added without sacrificing the zero's already awesome capabilities.

As you'd expect, existing software runs just as it did before.

The new zero is currently in stock at several dealers in the UK and the USA. Details are on the Raspberry Pi website. Dealer info is at the bottom of their post.

A camera has been one of the most-requested features for the zero. It opens up a huge range of new, exciting projects. There will be a huge demand for the new zero. Let's hope the stocks hold out for a while!

Share this post so your friends can place their orders quickly.
Tweet: The new Pi zero is here!

The new Pi zero as a mobile eye

If you want to give your mobile robot vision the new zero is a great solution. I can see it being used in wheeled robots, submarines and drones. Drones will need some fail-safe method of operator control for legal reasons but wheeled robots and subs can be completely independent if their software is smart enough.

Computer vision and neural networks

The zero has enough memory and processing power to run openCV. I'm working on experiments to add visual input my neural network software. I'll post about the project as it progresses.

If you're interested in neural networks I'm writing a tutorial series for beginners. Start here.

Stay informed

If you want more news about the Pi and the project, follow the blog on twitter.


Friday, 13 May 2016

Neural networks on the Raspberry Pi: More Neurons

A brief introduction to ANNs - part 2

The previous example of a neuron was a bit far-fetched. Its activation function doubled the weighted sum of its inputs. A simpler (and more useful) variant just returns the sum of its weighted inputs. This is known as a linear neuron.

The linear neuron

In APL, you could implement the linear neuron like this:
      ln←{+/⍺×⍵}   
and use it like this:
      1 0.5 1 ln 0.1 1 0.3
0.9

Inner product

However, there's a neater and more idiomatic way to define it in APL. A mathematician would call the ln function the dot product or inner product of ⍺ and ⍵ and in APL you can write it as
     ln←{⍺+.×⍵}

There are several reasons to use the inner product notation. It's concise, it's fast to execute, and (as we'll see later) it allows us to handle more than one neuron without having to write looping code.

Linear neurons are sometimes used in real-world applications; another type of neuron you're likely to come across is the perceptron.

The Perceptron

The output of a linear neuron is a real number. The output of a perceptron is binary: a 0 or a 1. This is useful in classification networks, where the output of a perceptron might be used to indicate the presence or absence of a particular feature, An output of zero would mean the feature was absent, while an output of 1 would mean that the feature was present.
Let's look at a concrete application, which we'll come back to later in this series.

An example - handwritten number recognition

Let's imagine that you want to construct a neural network to recognise handwritten digits. The input to your network might be a 28 x 28 matrix of pixels. A matrix of real numbers might represent how bright each pixel is.
You might have ten perceptrons corresponding to the digits 0 to 9. When the image of a handwritten digit is input, the relevant perceptron should fire. In other words, its output should be 1.

The perceptron firing rule

A perceptron calculates its output by looking at the value of the weighted sum of its inputs, just like a linear neuron. However, a perceptron outputs 0 if the sum is zero or negative, and it outputs a 1 if the sum is positive number.

Implementing the Perceptron in APL

You could define the perceptron like this:
p←{0<⍺+.×⍵}
and test it like this:
     0.1 1 0.3  p 1 0.5 1
1
     0.1 1 0.3  p 1 0.5 ¯1
1
    0.1 1 0.3 p 1 ¯0.5 ¯1
0

Notice that you represent negative numbers in APL using' ¯' rather than '-', which means 'do a subtraction'.

The bias input to the perceptron

If you look at a typical article on the perceptron, you will see that the algorithm it gives often includes an extra term called the bias b. Here's the definition from the Wikipedia article:
The bias is just like the other inputs except that its contribution is not weighted. To keep your code simple, you can do what many neural networks do. You can treat the bias term as the first element in the input vector, and prefix the weights by the constant 1.
Now when you calculate the  inner product of the extended inputs and weights the bias is added to the weighted inputs.
Here's the code:
     p2←{0<⍺+.×1,⍵}
In APL the comma (called catenate) is the symbol you use to concatenate two arrays together. In the definition of p2, the 1 is added to the beginning of the vector of weights ⍵. The bias b is the first element of the input vector ⍺.
You can test p2 like this:
        0.1 1 0.3  p2 0.5 1
1
      0.1 1 0.3  p2 ¯0.5 ¯1
0
Here the bias is 0.1 and the other inputs are 1 and 0.3. The weights of the two inputs are 0.5 and 1
Implementing multiple perceptrons
A single perceptron can't do much on it's own. A useful application is likely to have lots of neurons.These are often grouped into layers, and in many cases the layer is fully-connected.
A fully-connected layer of neurons is a collection of neurons in which
  1. each neuron is of the same type
  2. each neuron has its own set of weights
  3. each neuron receives all of the inputs to the layer
The layer has a vector of inputs. The inputs of every neuron are the elements of that vector.
The layer has a vector of outputs. Each element in the output vector is the output from a single neuron.
Using matrices in APL
Since there are multiple neurons, each of which has its own weights, you can represent the weights as a matrix. Column i of the matrix should contain the weights for neuron i.
In APL you can create a matrix by using the reshape function which is represented by the Greek letter rho.
Create a matrix like this:
    mat←2 4⍴20
    mat
20 20 20 20
20 20 20 20

You've created a matrix with two rows and four columns. Each entry in the matrix is the number 20.

Creating random test data

You can create test data with a bit more variety like this:

     mat←?2 4⍴20
     mat
14 11  7  4
10  3 13 15

The ? function (called roll) rolls a 20-sided die for each element in its argument. Your expression created an array of 2 by 4 random numbers.

They are not really random, of course, but they will be different each time you evaluate that expression. So don't be surprised when the values in your version of mat are different from mine!

Even that set of values can be improved on. Try this:

       mat←0.1ׯ10+?2 4⍴20
     mat
0   ¯0.9  0.4  0.6
¯0.4  0.1 ¯0.6 ¯0.9

Remember how APL parses expressions? You can read that first line as 'multiply by 0.1 the result of adding minus ten to a 2 by 4 matrix of numbers in the range 1 to 20'.

The APL is a bit more concise :)

Testing time

Time to try out your layer. Each neuron takes the same vector of inputs. In your case, that vector should have three elements: one for the bias and two for the weights. Create a vector of inputs like this:

    inputs ← 0.2 0.3 0.1

See what happens if you try

    inputs p2 mat
LENGTH ERROR
p2[0] p2←{0<⍺+.×1,⍵}
        
Oops! Something has gone wrong. APL's error message has told you the sort of problem it encountered, and what it was executing when the problem occurred.

To understand the source of the problem, try evaluating
      1,mat
1  0   ¯0.9  0.4  0.6
1 ¯0.4  0.1 ¯0.6 ¯0.9

Aha! APL has concatenated the 1 at the start of the matrix. That was right when you had a vector argument, but not for a matrix. Try this:

      1⍪mat
1    1    1    1
0   ¯0.9  0.4  0.6
¯0.4  0.1 ¯0.6 ¯0.9
That strange comma-with-a-bar means catenate in the last dimension, which will work for vectors and matrices. Change the definition of p2 to be

p2←{0<⍺+.×1⍪⍵}

and run inputs p2 mat again.

It works! With a minor change, you now have code that works equally well for one or many neurons.

That's enough for now. In the next article, we'll look at what the perceptron can and cannot do. We'll also look at other types types of neuron.