Making Predictions with Ikaros
Introduction
This text is a short tutorial on how to use Ikaros for prediction. It describes how delays and learning modules can be used to build predictive systems.
The Structure of a Predicton Module
A typical system for learning an input-output relation, such as a backpropagation network, will have single input vector, an output vector and an input which gives the desired output (or target) for the current input. During normal operation, the input is used by the system to produce the output and the target input is ignored. When the system is in learning mode, it also looks at the target input which is used to drive the learning to minimize the difference between the actual output and the desired target output. Systems of this kind thus have to operate either in input-output mode or in learning mode and it is common to distinguish between the training phase and the validation or testing phase.
The modules in Ikaros that can be used for prediction have a slightly different design. As above, there is an input and an output, where the output will depend on the vector received at the input in the same way as above. However, learning operates in a different way. There is an additional input T-Input with receives the training input and whis is combined with the input T-Output which sets the desired output for the input at T-Input. Learning can thus be concurrent with the normal operation of the module. Although this looks as a very small change from the standard set-up, it has several important advantages as wil be explained below.
Simultaneous Training and Validation
Since the input-output calculations are independent of training, it is possible to train on one data-set while simultaneously testing on another set. This differs from the standard approach where one data set is first used for training and then a second set is used for validation.
Prediction
Since the traing input are independent, it is possible to train on a prediction by simply time-shifting the input in relation to the output. For example, by training on an output that is delayed by one time step, the module will automatically learn a prediction. This is illustrated in the figure below.
Since the training input T-OUTPUT is delayed by one tick in relation to T-Input, the module will learn to predict T-OUTPUT at the time step after the current one.
A module suitable for prediction may also have an additional output which indictaes the error for the last trained T-INPUT-T-OUPUT pair. This output is typically called ERROR.
Some learning modules also have an input called LEARNING which is use to set the learning rate. If this input is 0, no learning will occur. When it is 1, learning will proceed with the speed set by a learning rate parameter. If LEARNING is inbetween 0 and 1, the learning rate will typically be multipled with the LEARNING input. If this input is not connected, it is equivalemt to an input of 1.
Delay Lines
There are obiously limits to how well a system can predict based on a single input vector. The simplest way to overcome this limitation is to use a tapped delay line to generate input to the prediction module. There is a simple way to do this in Ikaros by taking advantage of the fact that it is possible to set arbitrary delays on connections in combination with the ability to connect several outputs to a single input. If we wanted to create a tapped delay line that with three taps delayed by 1, 2, and 3 ticks we could make the following connections to a prediction module:
<connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="T-INPUT" delay="1" /> <connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="T-INPUT" delay="2" /> <connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="T-INPUT" delay="3" />
If the output from module S consists of an array with 5 components, this would result in an input array to PredictionModule with 15 components representing differently delayed signals from S. For the module to function properly, you would also have to connect the same inputs to the INPUT of PredictionModule:
<connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="INPUT" delay="1" /> <connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="INPUT" delay="2" /> <connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="INPUT" delay="3" />
These connections result in the network shown here:
Although it may seem unnecessary to use two identical delay lines, these are automatically handled internally by Ikaros as a single delay line.
Starting with version 1.1 of Ikaros, it is possible to use a compressed format when a tapped delay line is specified. By setting the delay to a range rather than a fixed delay, the IKC file is greatly simplified. This is specified using a color between the minimum and maximum delay. The six connections above can be defined in the following way:
<connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="T-INPUT" delay="1:3" /> <connection sourcemodule="S" source="OUTPUT" targetmodule="PredictionModule" target="INPUT" delay="1:3" />
Delays can also be specified by listing the desired delays. In this case, any combination of delays can be used. For example, delay="1,3,5". It is also possible to combine lists and ranges, for example, delay="1:5, 7:9".
Time-Series Prediction
Now let's build a simple time-series predictor using the ideas above. Assume that the module M measures some external property that changes over time and we want to predict the value of this variable one time step into the future.
This is auto-regression problem and the same signal with different delays is used both as input and training output for the prediction module. The figure below illustrates this situation. The module M produces a time series and the task for the prediction module is to predict this output ahead of time by observing the past signal from M.
Here, the delay of the current output to the training input T-OUTPUT is 0, which is one tick before the first input to T-INPUT. If we wanted the module to predict five steps ahead instead, we could delay the inputs to T-INPUT by 5, 6, and 7 instead. The inputs to INPUT would still be the same though, since we want the prediction to be based on the most current input. (Setting the first delay to 0 would make the prediction even faster, but it is not curently possible to mix zero-delays with longer delays in this way in Ikaros).
The author would like to thank Birger Johansson and Stefan Winberg for many creative discussions on these topics. This work was funded by the EU project MindRaces, FP6-511931.
blog comments powered by Disqus