# Implementing the Neural Network Structure in C++ Now that we've decided on the structure of our neural network, let's translate this into code. We'll be using matrices and vectors to represent the layers and weights of our network. This implementation will form the foundation upon which we'll build the forward and backward propagation steps. ## Step-by-Step Implementation ### 1. Include Necessary Headers First, we need to include the necessary headers for matrix operations. For this implementation, we'll use the Eigen library, which provides excellent support for linear algebra operations in C++. ```cpp #include #include #include ``` ### 2. Define the Neural Network Structure We'll define our neural network structure using Eigen's matrix and vector classes. The network will consist of: - An input layer with 2 neurons - A hidden layer with 4 neurons - An output layer with 1 neuron ```cpp // Define the input, hidden, and output sizes const int input_size = 2; const int hidden_size = 4; const int output_size = 1; // Define the weight matrices and bias vectors Eigen::MatrixXd weights_input_hidden(input_size, hidden_size); Eigen::MatrixXd weights_hidden_output(hidden_size, output_size); Eigen::VectorXd bias_hidden(hidden_size); Eigen::VectorXd bias_output(output_size); ``` ### 3. Initialize Weights and Biases Proper initialization of weights and biases is crucial for successful training of the neural network. We'll initialize the weights using a small random value scaled by the square root of the input size to help with convergence during training. ```cpp // Initialize weights randomly std::random_device rd; std::mt19937 gen(rd()); std::normal_distribution dis(0.0, std::sqrt(2.0/input_size)); // Initialize input to hidden weights for (int i = 0; i < input_size; ++i) { for (int j = 0; j < hidden_size; ++j) { weights_input_hidden(i, j) = dis(gen); } } // Initialize hidden to output weights for (int i = 0; i < hidden_size; ++i) { for (int j = 0; j < output_size; ++j) { weights_hidden_output(i, j) = dis(gen); } } // Initialize biases to zero bias_hidden.setZero(); bias_output.setZero(); ``` ### 4. Define Activation Functions We'll define the sigmoid activation function and its derivative, which we'll use in both the hidden and output layers. ```cpp // Sigmoid activation function double sigmoid(double x) { return 1.0 / (1.0 + std::exp(-x)); } // Derivative of the sigmoid function double sigmoid_derivative(double x) { return x * (1.0 - x); } ``` ### 5. Putting It All Together Here's the complete code that defines the neural network structure: ```cpp #include #include #include // Sigmoid activation function double sigmoid(double x) { return 1.0 / (1.0 + std::exp(-x)); } // Derivative of the sigmoid function double sigmoid_derivative(double x) { return x * (1.0 - x); } int main() { // Define the input, hidden, and output sizes const int input_size = 2; const int hidden_size = 4; const int output_size = 1; // Define the weight matrices and bias vectors Eigen::MatrixXd weights_input_hidden(input_size, hidden_size); Eigen::MatrixXd weights_hidden_output(hidden_size, output_size); Eigen::VectorXd bias_hidden(hidden_size); Eigen::VectorXd bias_output(output_size); // Initialize weights randomly std::random_device rd; std::mt19937 gen(rd()); std::normal_distribution dis(0.0, std::sqrt(2.0/input_size)); // Initialize input to hidden weights for (int i = 0; i < input_size; ++i) { for (int j = 0; j < hidden_size; ++j) { weights_input_hidden(i, j) = dis(gen); } } // Initialize hidden to output weights for (int i = 0; i < hidden_size; ++i) { for (int j = 0; j < output_size; ++j) { weights_hidden_output(i, j) = dis(gen); } } // Initialize biases to zero bias_hidden.setZero(); bias_output.setZero(); // Example input vector (optional for demonstration) Eigen::VectorXd input(2); input << 0.5, 0.3; // Output the initialized weights and biases std::cout << "Weights (Input to Hidden):\n" << weights_input_hidden << std::endl; std::cout << "\nBias (Hidden Layer):\n" << bias_hidden << std::endl; std::cout << "\nWeights (Hidden to Output):\n" << weights_hidden_output << std::endl; std::cout << "\nBias (Output Layer):\n" << bias_output << std::endl; return 0; } ``` ### 6. Explanation of the Code - **Matrix and Vector Definitions**: We use Eigen's `MatrixXd` and `VectorXd` to represent our weight matrices and bias vectors. These data structures provide efficient matrix operations which are essential for neural network computations. - **Weight Initialization**: We initialize our weights using a normal distribution with a mean of 0 and a standard deviation scaled by the square root of the input size. This helps in maintaining consistent variance across different layers during backpropagation. - **Bias Initialization**: Biases are initialized to zero as they typically don't require scaling during initialization. - **Activation Functions**: We define the sigmoid function and its derivative, which will be used in the forward and backward propagation steps respectively. ### 7. Next Steps Now that we've defined the structure of our neural network, the next step is to implement the forward propagation. In the next task, we'll compute the output of the network for a given input by applying the weighted sums and activation functions. ### 8. Reading Material - [Eigen Library Documentation](https://eigen.tuxfamily.org/dox/) - [Neural Network Weight Initialization](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-he-ba42f5b1e0e7) - [Activation Functions in Neural Networks](https://towardsdatascience.com/activation-functions-in-neural-networks-9d2079601331) This implementation sets up the basic structure of our neural network, which we'll build upon in subsequent tasks.