Hot questions for Using Neural networks in unity3d

Question:

I am making a flight simulation with autopilot so i need to make a DQN (Deep Q-Network) to control the autopilot but i don't know the optimal number of states.

the simulation is done in unity and all the environment and physics are done too, the DQN will only need to output (W,A,S,D) to control the plane, i found a code that controls CARTPOLE which in theory should be able to train and control the plane just fine, the only problem is that i don't exactly know if the states i chose are the right ones or not.

this is the code:

    
    import os
    import random
    import gym
    import numpy as np
    from collections import deque
    from keras.models import Sequential
    from keras.layers import Dense
    from keras.optimizers import Adam



    class DQNAGENT:

        def __init__(self,state_size,_action_size):

            self.state_size = state_sizes
            self.action_size = actions_sizes
            self.memory = deque(maxlen=2000)
            self.gamma = 0.95
            self.epsilon = 1.00
            self.epsilon_decay_rate = 0.995
            self.epsilon_min = 0.01
            self.learning_rate = 0.001
            self.model = self.build_model()

        def buildmodel(self):

            model = Sequential()
            model.add(Dense(24, input_dim=self.state_size, activation='relu'))
            model.add(Dense(24, activation='relu'))
            model.add(Dense(self.action_size, activation='linear'))
            model.compile(loss='mse',optimizer=Adam(lr=self.learning_rate))
            return model

        def remember(self, state, action, reward, next_state, done):

            self.memory.append((state, action, reward, next_state, done))

        def act(self, state):

            if np.random.rand()  self.epsilon_min:
                self.epsilon *= self.epsilon_decay_rate

        def load(self, name):

            self.model.load_weights(name)

        def save(self, name):

            self.model.save_weights(name)


    def main():
        #environemnet variables
        state_sizes=0
        actions_sizes=4
        #training Variables
        batch_size=32
        n_episodeds=100
        output_directory= 'model_output/autopilot'
        if not os.path.exists(output_directory):
            os.makedirs(output_directory)

        agent = DQNAGENT(state_sizes,actions_sizes)
        done = False

        for e in range(n_episodeds):
            state = #states of the game
            for time in range(5000):
                action = agent.act(state)
                #next_state, reward, done, _ = ##env.step(action)
                #put the next state from unity
                reward = reward if not done else -10
                agent.remember(state, action, reward, next_state, done)
                state = next_state
                if len(agent.memory) > batch_size:
                    agent.replay(batch_size)
    

where the agent class is the agent that will be trained those functions are okay but in the Main the state sizes are set to zero because i don't know yet the number also these three lines i can't convert to be able to run on my project

state = #states of the game
action = agent.act(state)
next_state, reward, done, _ = ##env.step(action)

the original code had these lines as :

env = gym.make('CartPole-v1')
state_size = env.observation_space.shape[0]
state = env.reset()
next_state, reward, done, _ = env.step(action)

because it gets these variables from the Gym package but i need to enter these manually, my environment will consist of airspeed,plane position, airport position,etc that's what i think will be write so if someone can help me figure out if this is correct or even better tell me what will be the optimal states will be really appreciated.

the excepted result is something like this.

statesizes = 4
states= "how to write those states in this variable"

Answer:

The states are simply the information your agent has in the current "frame/step". It's what the agent NEEDS in order to choose an action agent.act(state).

In the CartPole example, the state is a box of 4 values:

  • Cart Position
  • Cart Velocity
  • Pole Angle
  • Pole Velocity At Tip

In your flight autopilot, the state would be the information you need your agent to have in order to make its decision, e.g:

  • Current lattitude
  • Current speed
  • ...

Question:

I am making a game in Unity that involves creatures whose animations are determined by physics. For example, to move a limb of a creature, I can apply forces to the rigidbody it's associated with. I know how to apply forces programmatically through a script to create movements, but I'd like to create more complex and organic movements and thought that I might be able to use a neural network to do this.

I'd like each of the creatures to have a distinct way of moving in the world. I'd like to first puppeteer the creatures manually using my hand (with a Leap Motion controller), and have a neural network generate new movements based on the training I did with my hand.

More concretely, my manual puppeteering setup will apply forces to the rigidbodies of the creature as I move my hand. So if I lift my finger up, the system would apply a series of upward forces to the limb that is mapped to my finger. As I am puppeteering the creature, the NN receives Vector3 forces for each of the rigidbodies. In a way this is the same task as generating a new text based on a corpus of texts, but in this case my input is forces rather than strings.

Based on that training set, is it possible for the NN to generate movements for the characters (forces to be applied to the limbs) to mimic the movements I did with my hand?

I don't have that much experience with neural networks, but am eager to learn, specifically for this project. It would be great to know about similar projects that were done in Unity, or relevant libraries I could use that would simplify the implementation. Also, please let me know if there is anything I can clarify!


Answer:

Not really an answer but would not fit for comments

I'm not sure the strategy you want to apply to train your model is the right one.

I would go for reinforcement learning methods (you can check this question for more infos about it) using, for example, the distance traveled by the center of mass of the creature on the x-axis as a fitness. If this leads to weird behaviours (like this well known robot) you could, for example, think of strategies like penalizing your individuals given the distance traveled on y and z axis (still by the CoM) to try having guys that keep there CoM on the same plane.

Without knowing exactly what you want to achieve this is hard to give you more advices. Although, if you are not looking only for neural network based techniques, there is this really great paper you might want to have a look at (here is the video of their results).

Question:

I have a (perhaps) simple question about reading a text file in Unity (C#) containing a matrix of floats, and converting it to a matrix (or multiple arrays) of floats. Before even getting there, I seem unable to succesfully convert the 'string'-typed text to an array using the .Split method.

To summarize, I need the weights of an artificial neural network, which are trained in MATLAB, in my Unity script to use for calculating outputs and performing certain actions.

I will show you carefully what I tried; First of all, the "WeightsIH.txt" file I intend to read looks like this:

-0.493654767886117  1.96685485682002
-0.493654767886117  1.96685485682002
-1.12167737625244   -0.835107290621868
-0.168882176970878  -0.0508678270221517
0.00848314746590752 0.645890421329432
-0.445148017062813  -0.647593931130814
0.0719861441229913  0.00251854104000363
-0.0809087627607675 -0.00253116474618752
0.0677112130287254  0.00229085080614581
0.0754386783608645  0.00239167974789985
0.0809669296303145  0.00253343860819182
-3.54887781611766   -0.884835613897628
-0.459886351851150  -0.848445431434901
-0.0670274486060166 -1.39397672397051
-3.82077442331056   -2.40290337409384
0.0783082838340459  0.00245132734091342
-0.239255503684809  -0.0118048053697862
-0.0798562101865881 -0.00249224979249840
-0.0639184016149756 -0.00224822783089479
-0.778070988555323  -0.872732391008980
0.0297507291420014  -1.74952458318966
0.0963966379119354  0.00416637970011285
0.875794416782941   0.513267873404847
0.0788049456605797  0.00246400296768071
-0.301882135742981  1.29009004735214
-0.427112778202037  -0.602081710823938
-0.0287160010856207 0.876736618413824
0.174484840994587   -0.914135602108887
-1.13771093704277   -1.80803211709460
-0.842897124110693  -0.491320433352398
-0.883862027266628  0.577867664170583
-0.00732465337711950    -0.0608133682721268
0.0808154577205481  0.00252756604265255
-0.623653482256716  0.802021714172196
0.354715179032082   -1.40951283673210
0.107130434979695   0.00718795362864938
-3.25429700675793   1.15179026184665
0.00323132998826740 0.725967105940923
-0.445271160376852  -0.634848835782688
0.0353267566953751  -0.761005808087746
0.343818927585420   0.181552084533698
1.52333372694938    -1.95500299702055
1.28692979700544    2.03963829780562
0.665570336193150   -0.410729910590931
-0.0861536175683719 -0.00286332759826070
0.126619076047933   0.0171533754647685
-0.0822566525595005 -0.00259193055392493
-1.28712374245194   1.12380947288866
-1.29253445353219   -2.05175940341935
0.416493102590467   -0.617909454448863
0.0969179981825367  0.182721981022912
-0.0808788394332979 -0.00252999992128388
0.925818615324062   -1.91710736361040
-0.438361270919922  0.0119396635243124
1.05250770533487    -0.965588245752274
-0.0480257526132498 -0.00154845733604858
-0.0586872685404968 -0.00184430469780941
-0.471992797137374  -0.672492593551730
0.439729645530884   1.55878656878788
1.68156076364744    1.32277838623733
-0.455916176089339  -0.632974493608950
-2.76038741468600   1.87628535124240
0.993963121269136   0.412382925821147
0.0813294944564905  0.00254834378164385
1.05785147371486    -0.713165079912121
0.542621317508334   0.263699015691544
0.0859471661820603  0.00284559667679037
0.0752254002940845  0.00264837949098051
-0.0821494531270125 -0.00258646752960803
-0.135286303232541  -0.0230503985166856
-1.04824146276167   0.379479302836832
-1.00411135626130   0.643815076295448
-1.06427192746705   -1.71485567599474
0.0306923117644997  -0.326399702175058
-0.269230352435044  1.15492815472443
-1.09071040094221   0.974587786723127
-0.0811137162377503 -0.00253932111442298
0.843300471645878   -0.443547135621555
2.62543922875650    -1.43287453338882
-0.0879222032042109 -0.00305697486042674
1.08943207983567    -0.751402936369758
-0.0807147111768526 -0.00252376120454932
0.0920673615786699  0.00345537914425264
-3.32572250550751   2.23334579177979
0.567347532732561   -0.849919751538726
-0.981291377531284  -1.65375737600341
0.717607651399507   -0.501535458162733

Now my C# Unity code looks like this:

using UnityEngine;
using System.Collections;
using System;
using System.IO;

public class WeightsMDF : MonoBehaviour
{

    private string text;
    void Start ()
    {
        FileInfo theSourceFile = new FileInfo 
        ("C:/Users/Ajdin/Downloads/UnitySpace/minorProject/Nerd/Nerds/Assets/Scenes/WeightsIH.txt");
        StreamReader reader = theSourceFile.OpenText();

        do
        {
            text = reader.ReadLine();
            string[] floats = text.Split(" "[0]);
            //Console.WriteLine(text);
            print (text); 
            //print(floats[0]);
        } while (text != null);   

    }
}

Now this code gives me the entire textfile in the console, but with a nullreferenceexeption at last: NullReferenceException: Object reference not set to an instance of an object WeightsMDF.Start () (at Assets/Scripts/Nerds/WeightsMDF.cs:19).

This exception refers to line 19:

string[] floats = text.Split(" "[0]);

Now if I comment this line, I get the same output except the last console output line says

Null

And If I change line 19 to:

string[] floats = text.Split(" ");//[0]);

I get the following compiling error:

Assets/Scripts/Nerds/WeightsMDF.cs(19,36): error CS1502: The best overloaded method match for `string.Split(params char[])' has some invalid arguments    
Assets/Scripts/Nerds/WeightsMDF.cs(19,36): error CS1503: Argument `#1' cannot convert `string' expression to type `char[]'

Does anyone know what I'm doing wrongly? Thanks in advance!


Answer:

Try to check text before use split. ReadLine returns null at the end of file, so you need check it before split.

text = reader.ReadLine();
if (text == null) 
     break;

string[] floats = text.Split(" "[0]);

Also, you can use Peek. It returns next symbol and doesn't use it and -1 if no symbols left.

        using (StreamReader sr = new StreamReader(path)) 
        {
            while (sr.Peek() >= 0) 
            {
                text = sr.ReadLine();
                var splits = text.Split (new string[] {" "}, StringSplitOptions.RemoveEmptyEntries);
            }
        }

To split string by whitespace you can use

string[] splits = text.Split(null);
string[] splits = text.Split(new char[0]);