Artificial Intelligence is quickly becoming part of everyday life. Whether you are using your PC or your mobile phone, you are making use of some variation of AI. The Web has given Artificial Intelligence wings and it just keeps on improving.
I have written about AI and .NET before. To understand the underlying concepts of Deep Neural Networks, I recommend that you read the following two articles first. These articles explain how a neural network is put together, as well as setting up a genetic algorithm.
What is Deep Learning?
Deep Learning (or Hierarchical Learning or Deep Structured Learning) is a type of machine learning method that is based on learning data representations instead of task-specific algorithms. Deep Learning can be unsupervised, semi-supervised, or supervised.
Some Deep Learning architectures, such as deep neural networks, deep belief networks, and recurrent neural networks have been applied to the following fields:
- Computer Vision
- Speech Recognition
- Natural Language Processing
- Audio Recognition
- Social Network Filtering
- Machine Translation
- Bioinformatics
What Are Deep Neural Networks?
A Deep Neural Network (DNN) is an artificial neural network that has multiple hidden layers between the input and output layers. Deep Neural Networks models complex non-linear relationships. Deep Neural Networks are usually feedforward networks in which data flows from the input layer to the output layer without looping back.
Read: Machine Learning versus Deep Learning
How to Implement Deep Learning Logic in .NET
The small program that you will build today is a small example of how to implement Deep Learning logic into your application. You can use either C# or VB.NET. Let’s jump right in, shall we?
Create a new Console App and add a class to hold all the Deep Learning Logic. Name this class DeepNeuralNetwork or DNN.
Add the following fields into the class you have just created with the following C# code:
private int intInput;
private int intHiddenA;
private int intHiddenB;
private int intOutput;
private double[] dblInputs;
private double[][] dbliaWeights;
private double[][] dblabWeights;
private double[][] dblboWeights;
private double[] dblaBiases;
private double[] dblbBiases;
private double[] dbloBiases;
private double[] dblaOutputs;
private double[] dblbOutputs;
private double[] dbloutputs;
private static Random rndRand;
You can achieve the same thing using VB.NET:
Private intInput As Integer Private intHiddenA As Integer Private intHiddenB As Integer Private intOutput As Integer Private dblInputs As Double() Private dbliaWeights As Double()() Private dblabWeights As Double()() Private dblboWeights As Double()() Private dblaBiases As Double() Private dblbBiases As Double() Private dbloBiases As Double() Private dblaOutputs As Double() Private dblbOutputs As Double() Private dbloutputs As Double() Private Shared rndRand As Random
Adding Constructors in C#
These fields will hold the input Weights and their corresponding biases. Add the Constructor in C#:
public DNN(int iInput, int iHiddenA, int iHiddenB,
int iOutput)
{
intInput = iInput;
intHiddenA = iHiddenA;
intHiddenB = iHiddenB;
intOutput = iOutput;
dblInputs = new double[iInput];
dbliaWeights = CreateMatrix(iInput, iHiddenA);
dblabWeights = CreateMatrix(iHiddenA, iHiddenB);
dblboWeights = CreateMatrix(iHiddenB, iOutput);
dblaBiases = new double[iHiddenA];
dblbBiases = new double[iHiddenB];
dbloBiases = new double[iOutput];
dblaOutputs = new double[iHiddenA];
dblbOutputs = new double[iHiddenB];
dbloutputs = new double[iOutput];
rndRand = new Random(0);
InitWeight();
}
Adding the Constructor in VB.NET
Public Sub New(ByVal iInput As Integer, ByVal iHiddenA _
As Integer, ByVal iHiddenB As Integer, _
ByVal iOutput As Integer)
intInput = iInput
intHiddenA = iHiddenA
intHiddenB = iHiddenB
intOutput = iOutput
dblInputs = New Double(iInput - 1) {}
dbliaWeights = CreateMatrix(iInput, iHiddenA)
dblabWeights = CreateMatrix(iHiddenA, iHiddenB)
dblboWeights = CreateMatrix(iHiddenB, iOutput)
dblaBiases = New Double(iHiddenA - 1) {}
dblbBiases = New Double(iHiddenB - 1) {}
dbloBiases = New Double(iOutput - 1) {}
dblaOutputs = New Double(iHiddenA - 1) {}
dblbOutputs = New Double(iHiddenB - 1) {}
dbloutputs = New Double(iOutput - 1) {}
rndRand = New Random(0)
InitWeight()
End Sub
Read: Machine Learning in .NET
Add the CreateMatrix Function in C#
private static double[][] CreateMatrix(int iRows, int iCols)
{
double[][] dblRes = new double[iRows][];
for (int r = 0; r < dblRes.Length; ++r)
{
dblRes[r] = new double[iCols];
}
return dblRes;
}
Add the CreateMatrix Function in VB.NET
Private Shared Function CreateMatrix(ByVal iRows As Integer, _
ByVal iCols As Integer) As Double()()
Dim dblRes As Double()() = New Double(iRows - 1)() {}
Dim r As Integer
While r < dblRes.Length
dblRes(r) = New Double(iCols - 1) {}
Threading.Interlocked.Increment(r)
End While
Return dblRes
End Function
This creates a matrix for each of our weights. Add the Weight functions to set random values for each weight in C#:
private void InitWeight()
{
int intWeights = (intInput * intHiddenA) + intHiddenA +
(intHiddenA * intHiddenB) + intHiddenB + (intHiddenB *
intOutput) + intOutput;
double[] dblWeights = new double[intWeights];
double dblLO = -0.01;
double dblHI = 0.01;
for (int i = 0; i < dblWeights.Length; ++i)
{
dblWeights[i] = (dblHI - dblLO) * rndRand.NextDouble()
+ dblLO;
}
SetWeight(dblWeights);
}
public void SetWeight(double[] dblWeights)
{
int numWeights = (intInput * intHiddenA) + intHiddenA +
(intHiddenA * intHiddenB) + intHiddenB + (intHiddenB *
intOutput) + intOutput;
if (dblWeights.Length != numWeights)
{
throw new Exception("Weight Length Error");
}
int j = 0;
for (int i = 0; i < intInput; ++i)
{
for (int j = 0; j < intHiddenA; ++j)
{
dbliaWeights[i][j] = dblWeights[j++];
}
}
for (int i = 0; i < intHiddenA; ++i)
{
dblaBiases[i] = dblWeights[j++];
}
for (int i = 0; i < intHiddenA; ++i)
{
for (int j = 0; j < intHiddenB; ++j)
{
dblabWeights[i][j] = dblWeights[j++];
}
}
for (int i = 0; i < intHiddenB; ++i)
{
dblbBiases[i] = dblWeights[j++];
}
for (int i = 0; i < intHiddenB; ++i)
{
for (int j = 0; j < intOutput; ++j)
{
dblboWeights[i][j] = dblWeights[j++];
}
}
for (int i = 0; i < intOutput; ++i)
{
dbloBiases[i] = dblWeights[j++];
}
}
VB.NET
Private Sub InitWeight()
Dim intWeights As Integer = (intInput * intHiddenA) + _
intHiddenA + (intHiddenA * intHiddenB) + intHiddenB + _
(intHiddenB * intOutput) + intOutput
Dim dblWeights As Double() = New Double(intWeights - 1) {}
Dim dblLO As Double = -0.01
Dim dblHI As Double = 0.01
Dim i As Integer
While i < dblWeights.Length
dblWeights(i) = (dblHI - dblLO) * rndRand.NextDouble() + _
dblLO
Threading.Interlocked.Increment(i)
End While
SetWeight(dblWeights)
End Sub
Public Sub SetWeight(ByVal dblWeights As Double())
Dim numWeights As Integer = (intInput * intHiddenA) + _
intHiddenA + (intHiddenA * intHiddenB) + intHiddenB + _
(intHiddenB * intOutput) + intOutput
If dblWeights.Length <> numWeights
Throw New Exception("Weight Length Error")
End If
Dim j As Integer = 0
Dim i As Integer = 0
While i < intInput
While j < intHiddenA
dbliaWeights(i)(j) = dblWeights(Math.Min(Threading. _
Interlocked.Increment(j), j - 1))
Threading.Interlocked.Increment(j)
End While
Threading.Interlocked.Increment(i)
End While
While i < intHiddenA
dblaBiases(i) = dblWeights(Math.Min(Threading. _
Interlocked.Increment(j), j - 1))
Threading.Interlocked.Increment(i)
End While
While i < intHiddenA
While j < intHiddenB
dblabWeights(i)(j) = dblWeights(Math.Min(Threading. _
Interlocked.Increment(j), j - 1))
Threading.Interlocked.Increment(j)
End While
Threading.Interlocked.Increment(i)
End While
While i < intHiddenB
dblbBiases(i) = dblWeights(Math.Min(Threading. _
Interlocked.Increment(j), j - 1))
Threading.Interlocked.Increment(i)
End While
While i < intHiddenB
While j < intOutput
dblboWeights(i)(j) = dblWeights(Math.Min(Threading. _
Interlocked.Increment(j), j - 1))
Threading.Interlocked.Increment(j)
End While
Threading.Interlocked.Increment(i)
End While
While i < intOutput
dbloBiases(i) = dblWeights(Math.Min(Threading. _
Interlocked.Increment(j), j - 1))
Threading.Interlocked.Increment(i)
End While
End Sub
Read: Azure Machine Learning Tutorial
Complete the Class in C#
Add the following C# code to complete the class:
public double[] Compute(double[] dblValues)
{
double[] dblaSums = new double[intHiddenA];
double[] dblbSums = new double[intHiddenB];
double[] dbloSums = new double[intOutput];
for (int i = 0; i < dblValues.Length; ++i)
{
dblInputs[i] = dblValues[i];
}
for (int j = 0; j < intHiddenA; ++j) {
for (int i = 0; i < intInput; ++i)
{
dblaSums[j] += dblInputs[i] * dbliaWeights[i][j];
}
}
for (int i = 0; i < intHiddenA; ++i)
{
dblaSums[i] += dblaBiases[i];
}
Console.WriteLine("nInternal aSums:");
Program.ShowGrid(dblaSums, dblaSums.Length, 4, true);
for (int i = 0; i < intHiddenA; ++i)
{
dblaOutputs[i] = HyperbolicTangent(dblaSums[i]);
}
Console.WriteLine("nInternal aOutputs:");
Program.ShowGrid(dblaOutputs, dblaOutputs.Length, 4,
true);
for (int j = 0; j < intHiddenB; ++j)
{
for (int i = 0; i < intHiddenA; ++i)
{
dblbSums[j] += dblaOutputs[i] * .dblabWeights[i][j];
}
}
for (int i = 0; i < intHiddenB; ++i)
{
dblbSums[i] += dblbBiases[i];
}
Console.WriteLine("nInternal bSums:");
Program.ShowGrid(dblbSums, dblbSums.Length, 4, true);
for (int i = 0; i < intHiddenB; ++i)
{
dblbOutputs[i] = HyperbolicTangent(dblbSums[i]);
}
Console.WriteLine("nInternal bOutputs:");
Program.ShowGrid(dblbOutputs, dblbOutputs.Length, 4,
true);
for (int j = 0; j < intOutput; ++j)
{
for (int i = 0; i < intHiddenB; ++i)
{
dbloSums[j] += dblbOutputs[i] * dblboWeights[i][j];
}
}
for (int i = 0; i < intOutput; ++i)
{
dbloSums[i] += dbloBiases[i];
}
Console.WriteLine("nInternal oSums:");
Program.ShowGrid(dbloSums, dbloSums.Length, 4, true);
double[] dblAllOut = MaxOutput(dbloSums);
Array.Copy(dblAllOut, dbloutputs, dblAllOut.Length);
double[] dblResult = new double[intOutput];
Array.Copy(this.dbloutputs, dblResult, dblResult.Length);
return dblResult;
}
private static double HyperbolicTangent(double val)
{
if (val < -20.0)
{
return -1.0;
}
else if (val > 20.0)
{
return 1.0;
}
else
{
return Math.Tanh(val);
}
}
private static double[] MaxOutput(double[] oSums)
{
double dblMax = oSums[0];
for (int i = 0; i < oSums.Length; ++i)
{
if (oSums[i] > dblMax)
{
dblMax = oSums[i];
}
}
double dblScale = 0.0;
for (int i = 0; i < oSums.Length; ++i)
{
dblScale += Math.Exp(oSums[i] - dblMax);
}
double[] dblRes = new double[oSums.Length];
for (int i = 0; i < oSums.Length; ++i)
{
dblRes[i] = Math.Exp(oSums[i] - dblMax) / dblScale;
}
return dblRes;
}
Complete the Class in VB.NET
Add the following code to complete the class in VB.NET:
Public Function Compute(ByVal dblValues As Double()) As Double()
Dim dblaSums As Double() = New Double(intHiddenA - 1) {}
Dim dblbSums As Double() = New Double(intHiddenB - 1) {}
Dim dbloSums As Double() = New Double(intOutput - 1) {}
Dim i As Integer
Dim j As Integer
While i < dblValues.Length
dblInputs(i) = dblValues(i)
Threading.Interlocked.Increment(i)
End While
While j < intHiddenA
While i < intInput
dblaSums(j) += dblInputs(i) * dbliaWeights(i)(j)
Threading.Interlocked.Increment(i)
End While
Threading.Interlocked.Increment(j)
End While
While i < intHiddenA
dblaSums(i) += dblaBiases(i)
Threading.Interlocked.Increment(i)
End While
Console.WriteLine(vbLf & "Internal aSums:")
ShowGrid(dblaSums, dblaSums.Length, 4, True)
While i < intHiddenA
dblaOutputs(i) = HyperbolicTangent(dblaSums(i))
Threading.Interlocked.Increment(i)
End While
Console.WriteLine(vbLf & "Internal aOutputs:")
ShowGrid(dblaOutputs, dblaOutputs.Length, 4, True)
While j < intHiddenB
While i < intHiddenA
dblbSums(j) += dblaOutputs(i) * dblabWeights(i)(j)
Threading.Interlocked.Increment(i)
End While
Threading.Interlocked.Increment(j)
End While
While i < intHiddenB
dblbSums(i) += dblbBiases(i)
Threading.Interlocked.Increment(i)
End While
Console.WriteLine(vbLf & "Internal bSums:")
ShowGrid(dblbSums, dblbSums.Length, 4, True)
While i < intHiddenB
dblbOutputs(i) = HyperbolicTangent(dblbSums(i))
Threading.Interlocked.Increment(i)
End While
Console.WriteLine(vbLf & "Internal bOutputs:")
ShowGrid(dblbOutputs, dblbOutputs.Length, 4, True)
While j < intOutput
While i < intHiddenB
dbloSums(j) += dblbOutputs(i) * dblboWeights(i)(j)
Threading.Interlocked.Increment(i)
End While
Threading.Interlocked.Increment(j)
End While
While i < intOutput
dbloSums(i) += dbloBiases(i)
Threading.Interlocked.Increment(i)
End While
Console.WriteLine(vbLf & "Internal oSums:")
ShowGrid(dbloSums, dbloSums.Length, 4, True)
Dim dblAllOut As Double() = MaxOutput(dbloSums
Array.Copy(dblAllOut, dbloutputs, dblAllOut.Length)
Dim dblResult As Double() = New Double(intOutput - 1) {}
Array.Copy(Me.dbloutputs, dblResult, dblResult.Length)
Return dblResult
End Function
Private Shared Function HyperbolicTangent(ByVal val As Double) _
As Double
If val < -20 Then
Return -1
ElseIf val > 20 Then
Return 1
Else
Return Math.Tanh(val)
End If
End Function
Private Shared Function MaxOutput(ByVal oSums As Double()) _
As Double()
Dim dblMax As Double = oSums(0)
Dim i As Integer
While i < oSums.Length
If oSums(i) > dblMax Then
dblMax = oSums(i)
End If
Threading.Interlocked.Increment(i)
End While
Dim dblScale As Double = 0
While i < oSums.Length
dblScale += Math.Exp(oSums(i) - dblMax)
Threading.Interlocked.Increment(i)
End While
Dim dblRes As Double() = New Double(oSums.Length - 1) {}
While i < oSums.Length
dblRes(i) = Math.Exp(oSums(i) - dblMax) / dblScale
Threading.Interlocked.Increment(i)
End While
Return dblRes
End Function
The calculated inputs, along with the biases, get computed and outputted. Add the code for the Program class or the Module to make use of the DeepNeuralNetwork Class.
C#
static void Main(string[] args)
{
Console.WriteLine("nStart Creating a 3-4-5-2 neural
network");
int intInput = 3;
int intHiddenA = 4;
int intHiddenB = 5;
int intOutput = 2;
DNN dnn = new DNN(intInput, intHiddenA, intHiddenB,
intOutput);
double[] dblWeights = new double[] {
0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09,
0.10,
0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19,
0.20,
0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29,
0.30,
0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39,
0.40,
0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49,
0.50,
};
dnn.SetWeight(dblWeights);
double[] dblX = new double[] { 1.0, 2.0, 3.0 };
Console.WriteLine("nDummy Weights + Bias Values");
ShowGrid(dblWeights, 10, 2, true);
Console.WriteLine("nDummy Inputs");
ShowGrid(dblX, 3, 1, true);
double[] dblY = dnn.Compute(dblX);
Console.WriteLine("nComputed Outputs");
ShowGrid(dblY, 2, 4, true);
Console.WriteLine("nEnd");
Console.ReadLine();
}
static public void ShowGrid(double[] dblVector, int intCols,
int intDec, bool blnNewLine)
{
for (int i = 0; i < dblVector.Length; ++i)
{
if (i % intCols == 0)
{
Console.WriteLine("");
}
Console.Write(dblVector[i].ToString("F" +
intDec).PadLeft(intDec + 4) + " ");
}
if (blnNewLine == true)
{
Console.WriteLine("");
}
}
VB.NET
Sub Main(ByVal args As String())
Console.WriteLine(vbLf & "Start Creating a 3-4-5-2 _
neural network")
Dim intInput As Integer = 3
Dim intHiddenA As Integer = 4
Dim intHiddenB As Integer = 5
Dim intOutput As Integer = 2
Dim dnn As DNN = New DNN(intInput, intHiddenA, intHiddenB, _
intOutput)
Dim dblWeights As Double() = New Double() {
0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1,
0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2,
0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3,
0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4,
0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5
}
dnn.SetWeight(dblWeights)
Dim dblX As Double() = New Double() {1, 2, 3}
Console.WriteLine(vbLf & "Dummy Weights + Bias Values")
ShowGrid(dblWeights, 10, 2, True)
Console.WriteLine(vbLf & "Dummy Inputs")
ShowGrid(dblX, 3, 1, True)
Dim dblY As Double() = dnn.Compute(dblX)
Console.WriteLine(vbLf & "Computed Outputs")
ShowGrid(dblY, 2, 4, True)
Console.WriteLine(vbLf & "End")
Console.ReadLine()
End Sub
Public Sub ShowGrid(ByVal dblVector As Double(), _
ByVal intCols As Integer, ByVal intDec As Integer, _
ByVal blnNewLine As Boolean)
Dim i As Integer
While i < dblVector.Length
If i Mod intCols = 0 Then
Console.WriteLine("")
End If
Console.Write(dblVector(i).ToString("F" & intDec) _
.PadLeft(intDec + 4) & " ")
Threading.Interlocked.Increment(i)
End While
If blnNewLine = True Then
Console.WriteLine("")
End If
End Sub
Here you supply values for the weights. The ShowGrid procedure displays the results.
Conclusion to Deep Neural Networks Programming
Deep Neural Networks are not too complicated. They are just the beginning. Hopefully, in a future article, I can delve a bit deeper. Until then, goodbye.
