Hot questions for Using Neural networks in php

Question:

I downloaded a neural network program written in PHP. It is for the XOR gate. I want to modify it into a program working for other data sets. Is it possible by just changing the training data set.

I changed this part. I used the data set ( 1,2, 1)- (3), (1,4,1)- (5), (2,2,1)-(4), (2,4,1)-6 instead of the training data set for the XOR gate.

The code is previously working but by just changing the training data set, it didn't work for me. Output generated was 1 for every data set. Would you please help me ?

        <?php
require_once ("class_neuralnetwork.php");

// Create a new neural network with 3 input neurons,
// 4 hidden neurons, and 1 output neuron
$n = new NeuralNetwork(3, 4, 1);
$n->setVerbose(false);

// Add test-data to the network. In this case,
// we want the network to learn the 'XOR'-function
$n->addTestData(array (-1, -1, 1), array (-1));
$n->addTestData(array (-1,  1, 1), array ( 1));
$n->addTestData(array ( 1, -1, 1), array ( 1));
$n->addTestData(array ( 1,  1, 1), array (-1));

// we try training the network for at most $max times
$max = 3;

echo "<h1>Learning the XOR function</h1>";
// train the network in max 1000 epochs, with a max squared error of 0.01
while (!($success = $n->train(1000, 0.01)) && ++$i<$max) {
    echo "Round $i: No success...<br />";
}

// print a message if the network was succesfully trained
if ($success) {
    $epochs = $n->getEpoch();
    echo "Success in $epochs training rounds!<br />";
}

echo "<h2>Result</h2>";
echo "<div class='result'>";
// in any case, we print the output of the neural network
for ($i = 0; $i < count($n->trainInputs); $i ++) {
    $output = $n->calculate($n->trainInputs[$i]);
    echo "<div>Testset $i; ";
    echo "expected output = (".implode(", ", $n->trainOutput[$i]).") ";
    echo "output from neural network = (".implode(", ", $output).")\n</div>";
}
echo "</div>";
//echo "<h2>Internal network state</h2>";
//$n->showWeights($force=true);

// Now, play around with some of the network's parameters a bit, to see how it 
// influences the result
$learningRates = array(0.1, 0.25, 0.5, 0.75, 1);
$momentum = array(0.2, 0.4, 0.6, 0.8, 1);
$rounds = array(100, 500, 1000, 2000);
$errors = array(0.1, 0.05, 0.01, 0.001);

echo "<h1>Playing around...</h1>";
echo "<p>The following is to show how changing the momentum & learning rate, 
in combination with the number of rounds and the maximum allowable error, can 
lead to wildly differing results. To obtain the best results for your 
situation, play around with these numbers until you find the one that works
best for you.</p>";
echo "<p>The values displayed here are chosen randomly, so you can reload 
the page to see another set of values...</p>";

for ($j=0; $j<10; $j++) {
    // no time-outs
    set_time_limit(0);

    $lr = $learningRates[array_rand($learningRates)];
    $m = $momentum[array_rand($momentum)];
    $r = $rounds[array_rand($rounds)];
    $e = $errors[array_rand($errors)];
    echo "<h2>Learning rate $lr, momentum $m @ ($r rounds, max sq. error $e)</h2>";
    $n->clear();
    $n->setLearningRate($lr);
    $n->setMomentum($m);
    $i = 0;
    while (!($success = $n->train($r, $e)) && ++$i<$max) {
        echo "Round $i: No success...<br />";
        flush();
    }

    // print a message if the network was succesfully trained
    if ($success) {
        $epochs = $n->getEpoch();
        echo "Success in $epochs training rounds!<br />";

        echo "<div class='result'>";
        for ($i = 0; $i < count($n->trainInputs); $i ++) {
            $output = $n->calculate($n->trainInputs[$i]);
            echo "<div>Testset $i; ";
            echo "expected output = (".implode(", ", $n->trainOutput[$i]).") ";
            echo "output from neural network = (".implode(", ", $output).")\n</div>";
        }
        echo "</div>";
    }
}
?>
        </div>
    </body>
</html>

Answer:

It is hard to give a definitive answer without seeing the code for the neural network implementation. But it looks like the implementation is probably using a tanh activation function, which would constrain neuron outputs to the range (-1, 1). Also, it appears that the implementation does not use an implicit bias input, which is why the example for the XOR function uses a third input that is explicitly set to 1 for all cases.

So the basic problem is that all of your target outputs are outside the range of the activation function. You need to restructure your network but how you do that will depend on what you are trying to accomplish. From your question, it isn't clear whether you are trying to train a classifier or interpolate a function.

If your four different outputs (3, 5, 4, and 6) represent classes, then you should define a network with four outputs and define the desired output values as follows:

$n = new NeuralNetwork(3, 4, 4);

$n->addTestData(array (1, 2, 1), array ( 1, -1, -1, -1));
$n->addTestData(array (1, 4, 1), array (-1,  1, -1, -1));
$n->addTestData(array (2, 2, 1), array (-1, -1,  1, -1));
$n->addTestData(array (2, 4, 1), array (-1, -1, -1,  1));

Note that you may want more than 4 hidden nodes for your example.

If you are attempting to do function interpolation, then you would keep just the single output neuron but need to scale your target output values to be in the range of the tanh function.

Question:

I'd like to learn about neural nets starting with the very basic perceptron algorithm. So I've implemented one in PHP and I'm getting weird results after training it. All the 4 possible input combinations return either wrong or correct results (more often the wrong ones).

1) Is there something wrong with my implementation or the results I'm getting are normal?

2) Can this kind of implementation work with more than 2 inputs?

3) What would be the next (easiest) step in learning neural nets after this? Maybe adding more neurons, changing the activation function, or ...?

P.S. I'm pretty bad at math and don't necessarily understand the math behind perceptron 100%, at least not the training part.

Perceptron Class
<?php

namespace Perceptron;

class Perceptron
{
    // Number of inputs
    protected $n;

    protected $weights = [];

    protected $bias;

    public function __construct(int $n)
    {
        $this->n = $n;

        // Generate random weights for each input
        for ($i = 0; $i < $n; $i++) {
            $w = mt_rand(-100, 100) / 100;

            array_push($this->weights, $w);
        }

        // Generate a random bias
        $this->bias = mt_rand(-100, 100) / 100;
    }

    public function sum(array $inputs)
    {
        $sum = 0;

        for ($i = 0; $i < $this->n; $i++) {
            $sum += ($inputs[$i] * $this->weights[$i]);
        }

        return $sum + $this->bias;
    }

    public function activationFunction(float $sum)
    {
        return $sum < 0.0 ? 0 : 1;
    }

    public function predict(array $inputs)
    {
        $sum = $this->sum($inputs);

        return $this->activationFunction($sum);
    }

    public function train(array $trainingSet, float $learningRate)
    {
        foreach ($trainingSet as $row) {
            $inputs = array_slice($row, 0, $this->n);
            $correctOutput = $row[$this->n];

            $output = $this->predict($inputs);
            $error = $correctOutput - $output;

            // Adjusting the weights
            $this->weights[0] = $this->weights[0] + ($learningRate * $error);
            for ($i = 0; $i < $this->n - 1; $i++) {
                $this->weights[$i + 1] =
                    $this->weights[$i] + ($learningRate * $inputs[$i] * $error);
            }
        }

        // Adjusting the bias
        $this->bias += ($learningRate * $error);
    }
}
Main File
<?php

require_once 'vendor/autoload.php';

use Perceptron\Perceptron;

// Create a new perceptron with 2 inputs
$perceptron = new Perceptron(2);

// Test the perceptron
echo "Before training:\n";

$output = $perceptron->predict([0, 0]);
echo "{$output} - " . ($output == 0 ? 'correct' : 'nope') . "\n";

$output = $perceptron->predict([0, 1]);
echo "{$output} - " . ($output == 0 ? 'correct' : 'nope') . "\n";

$output = $perceptron->predict([1, 0]);
echo "{$output} - " . ($output == 0 ? 'correct' : 'nope') . "\n";

$output = $perceptron->predict([1, 1]);
echo "{$output} - " . ($output == 1 ? 'correct' : 'nope') . "\n";

// Train the perceptron
$trainingSet = [
    // The 3rd column is the correct output
    [0, 0, 0],
    [0, 1, 0],
    [1, 0, 0],
    [1, 1, 1],
];

for ($i = 0; $i < 1000; $i++) {
    $perceptron->train($trainingSet, 0.1);
}

// Test the perceptron again - now the results should be correct
echo "\nAfter training:\n";

$output = $perceptron->predict([0, 0]);
echo "{$output} - " . ($output == 0 ? 'correct' : 'nope') . "\n";

$output = $perceptron->predict([0, 1]);
echo "{$output} - " . ($output == 0 ? 'correct' : 'nope') . "\n";

$output = $perceptron->predict([1, 0]);
echo "{$output} - " . ($output == 0 ? 'correct' : 'nope') . "\n";

$output = $perceptron->predict([1, 1]);
echo "{$output} - " . ($output == 1 ? 'correct' : 'nope') . "\n";

Answer:

I must thank you for posting this question, I have wanted a chance to dive a little deeper into neural networks. Anyway, down to business. After tinkering around and verbose logging what all is happening, it ended up only requiring 1 character change to work as intended:

public function sum(array $inputs)
{
    ...
    //instead of multiplying the input by the weight, we should be adding the weight
    $sum += ($inputs[$i] + $this->weights[$i]);
    ...
}

With that change, 1000 iterations of training ends up being overkill. One bit of the code was confusing, different setting of weights:

public function train(array $trainingSet, float $learningRate)
{
    foreach ($trainingSet as $row) {
        ...
        $this->weights[0] = $this->weights[0] + ($learningRate * $error);
        for ($i = 0; $i < $this->n - 1; $i++) {
            $this->weights[$i + 1] =
                $this->weights[$i] + ($learningRate * $inputs[$i] * $error);
        }
}

I don't necessarily understand why you chose to do it this way. My unexperienced eye would think that the following would work as well.

for ($i = 0; $i < $this->n; $i++) { 
    $this->weight[$i] += $learningRate * $error;
}

Question:

I get a text string from a scanned receipt. Here are couple of examples:

George's Restaurant 300 72th Street Miami Beach fl 33141 305-864-5586 Server: Ronald 01/19/2013 Table 20/1 10:53 PM Guests: 1 10062 Reprint #: 1 Ferrari Carano Insalate Cesare Caprese with prosciutto FISH SPEC Spinach Ricotta Ravioli Seafood Pasta Ossobucco 47.00 7.50 9.50 25.95 15.95 19.95 29.95 Sub Total Tax 155.80 14.02 Total 169.82 169.82 Balance Due GRATUITY NOT INCLUDED!!! Thank you for your business

How do I identify what the total amount is in each case (169.82 and 52.88)?

I was thinking I can remove all non-numeric characters, split remaining into array and look for the largest. But it can get confusing with address and phone numbers. I suppose I need to make sure the word TOTAL, SUB-TOTAL, or AMOUNT DUE is close by.

Any suggestions? Thanks.

Another example:

933 ece tur New OrlerS LA 70116 504.:25.1602 wwwfranksresta.ratnewor leans.com 219 KATHY U che 1750 Feb03'1 (7:-2PM Tbl 6/1 Gst 4 1 GARLICBREAD 2 Diet 2 Iced Tea 2 TASTE OF NO 1 Whole Muff 1 Alfredo 3,95 6.00 6.00 33.90 14.95 14.95 Food Tax TOTAL DUE 79.75 7.78 87.53

image here


UPDATE:

It appears I need to look into neural networks to solve this.


Answer:

Try this:

<?php

function checktotal($rcpt) {
    if (preg_match_all('/(\d+\.\d{2})(?:\D|$)/', $rcpt, $match))
        echo 'Total is $' . max($match[1]) . "\n";
    else echo "No numbers!\n";
}

$rcpts = [
    "George's Restaurant 300 72th Street Miami Beach fl 33141 305-864-5586 Server: Ronald 01/19/2013 Table 20/1 10:53 PM Guests: 1 10062 Reprint #: 1 Ferrari Carano Insalate Cesare Caprese with prosciutto FISH SPEC Spinach Ricotta Ravioli Seafood Pasta Ossobucco 47.00 7.50 9.50 25.95 15.95 19.95 29.95 Sub Total Tax 155.80 14.02 Total 169.82 169.82 Balance Due GRATUITY NOT INCLUDED!!! Thank you for your business",
    "SUSHI HARA 8701 W PARMER LANE STE 2128 AUSTIN, TX 78729 123835218 ORDER: A9 Dine-in 25-Jan-2018 6 10 53 1 다tASHU DON SHRIMP TEMPURA (3PCS HARU COMBO SALMON ROLL $11.95 $8.95 $20.00 $7.95 to go Subtotal $48.85 $4.03 S52.88 Tax Total Order 05852ZSBGOW4M Thank you for dining at Sushi Hara",
    "933 ece tur New OrlerS LA 70116 504.:25.1602 wwwfranksresta.ratnewor leans.com 219 KATHY U che 1750 Feb03'1 (7:-2PM Tbl 6/1 Gst 4 1 GARLICBREAD 2 Diet 2 Iced Tea 2 TASTE OF NO 1 Whole Muff 1 Alfredo 3,95 6.00 6.00 33.90 14.95 14.95 Food Tax TOTAL DUE 79.75 7.78 87.53"
    ];
foreach ($rcpts as $rcpt) checktotal($rcpt);

The output for your test group is:

Total is $169.82
Total is $52.88
Total is $87.53

Question:

I am a new comer to ANN and i was hoping someone would help me understand the concept of normalization with an example as the following. preferable in php, just a small clear example will do.

Lets say >>

    Rooms               Area              Type 
     1                   350               Apartment
     1                   800               Apartment
     3                   300               Apartment
     8                   1500              House
     7                   900               house
     1                   800               flat

how can i normalize this data? most of the books that read regrading ANN states that the out put has to come as 0 or 1, where do this values come from?

I want the system to be able to tell the difference between a flat, house and apartments. please give me an example in Php code. thanks a lot in advance.


Answer:

Since this is a classification problem (as it seems from the data), you don't need to normalize the output. So, you need to normalize the input value only. If you are using numpy in Python, you can normalize it in the following two ways:

  1. Gaussian Normalization (0 mean and 1 std): This normalizes the data to have zero mean and st deviation equal to 1.

    xNor = (x-x.mean())/x.std()

  2. (0, 1) Range Normalization: This normalizes the data to the range (0, 1).

    xNor = (x-x.min())/(x.max()-x.min())

To vectorize the output, you can use one-vs-all classification or binary-encoding classification. I would personally recommend one-vs-all classification. You can refer to this here https://en.wikipedia.org/wiki/Multiclass_classification

Question:

I'm trying to translate this algorithm from C language to PHP (for study) This is an example of a perceptron. I copied the example written in c and I'm trying to translate it into PHP. Currently I wrote this code, what am I wrong? As output I only know 101 iterations with result always 1.

This is the C program:

    #include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <time.h>

#define LEARNING_RATE    0.1
#define MAX_ITERATION    100

float randomFloat()
{
    return (float)rand() / (float)RAND_MAX;
}

int calculateOutput(float weights[], float x, float y)
{
    float sum = x * weights[0] + y * weights[1] + weights[2];
    return (sum >= 0) ? 1 : -1;
}

int main(int argc, char *argv[])
{
    srand(time(NULL));

    float x[208], y[208], weights[3], localError, globalError;
    int outputs[208], patternCount, i, p, iteration, output;

    FILE *fp;
    if ((fp = fopen("test1.txt", "r")) == NULL) {
        printf("Cannot open file.\n");
        exit(1);
    }

    i = 0;
    while (fscanf(fp, "%f %f %d", &x[i], &y[i], &outputs[i]) != EOF) {
        if (outputs[i] == 0) {
            outputs[i] = -1;
        }
        i++;
    }
    patternCount = i;

    weights[0] = randomFloat();
    weights[1] = randomFloat();
    weights[2] = randomFloat();

    iteration = 0;
    do {
        iteration++;
        globalError = 0;
        for (p = 0; p < patternCount; p++) {
            output = calculateOutput(weights, x[p], y[p]);

            localError = outputs[p] - output;
            weights[0] += LEARNING_RATE * localError * x[p];
            weights[1] += LEARNING_RATE * localError * y[p];
            weights[2] += LEARNING_RATE * localError;

            globalError += (localError*localError);
        }

        /* Root Mean Squared Error */
        printf("Iteration %d : RMSE = %.4f\n", iteration,
               sqrt(globalError/patternCount));
    } while (globalError != 0 && iteration<=MAX_ITERATION);

    printf("\nDecision boundary (line) equation: %.2f*x + %.2f*y + %.2f = 0\n",
           weights[0], weights[1], weights[2]);

    return 0;
}

and this is the code that I wrote

   <?php
    define("LEARNING_RATE", 0.1);
    define("MAX_ITERATION", 100);
    function randomFloat(){ return (float) mt_rand() / mt_getrandmax(); }
    function calculateOutput($weights, $x, $y){
        $sum = (float) $x * $weights[0] + $y * $weights[1] + $weights[2];
        return ($sum >= 0) ? 1 : -1;
    }
    srand(time());
    $i = 0;
    $ars = explode("\n",file_get_contents('https://raw.githubusercontent.com/RichardKnop/ansi-c-perceptron/master/test1.txt'));
    foreach($ars as $ar){
        $temp = explode("\t", $ar);
        $x[$i] = (float) $temp[0];
        $y[$i] = (float) $temp[1];
        $output[$i] = (int) $temp[2];
        if($output[$i] == 0)
            $output[$i] = -1;
        $i++;
   }
   $patternCount = $i;
   $weights[0] = randomFloat();
   $weights[1] = randomFloat();
   $weights[2] = randomFloat();
   $iteration = 0;
   do{
       $iteration++;
       $globalError = 0;
       for ($p = 0; $p < $patternCount; $p++) {
           $output = calculateOutput($weights, $x[$p], $y[$p]);
           $localError = $outputs[$p] - $output;
           $weights[0] += LEARNING_RATE * $localError * $x[$p];
           $weights[1] += LEARNING_RATE * $localError * $y[$p];
           $weights[2] += LEARNING_RATE * $localError;
           $globalError += ($localError*$localError);
       }
       $r .= "Iteration $iteration : RMSE = " . 
sqrt($globalError/$patternCount)."<br>";
}while($globalError != 0 && $iteration<=MAX_ITERATION);
echo $r;
echo "<br><hr><br>";
echo "Decision boundary (line) equation: ".$weights[0]."*x + ".$weights[1]."*y + ".$weights[2]." = 0<br>";

it's practically identical, but why does it not work?


Answer:

    $ars = explode("\n",file_get_contents('…'));

Since the file ends with a \n, this yields an empty string as the last array value, which disrupts the foreach($ars as $ar) loop. To read the file into an array, simply use:

    $ars = file('…');

In the foreach($ars as $ar) loop, you used the wrong name $output[$i] instead of $outputs[$i].


       $r .= "Iteration $iteration : RMSE = " . 
sqrt($globalError/$patternCount)."<br>";
}while($globalError != 0 && $iteration<=MAX_ITERATION);
echo $r;

You didn't initialize $r. Instead of the above, you can use:

       echo "Iteration $iteration : RMSE = " . 
            sqrt($globalError/$patternCount)."<br>";
    } while ($globalError != 0 && $iteration<=MAX_ITERATION);