Finally!

A couple of weeks ago I made a list of things I wanted to work on to start prototyping things for the Kings and Castles AI. So far, I made it to item number one. The first thing I wanted to work on was the neural net system. I was never really happy with it and, up until today, I was thinking about pulling it out of Kings and Castles. So, I decided I wanted to go into the core of the neural net system and see what might be up.

Up until two weeks ago I had been treating the core neural net system as a black box. I had assumed that the system was working as intended. In hindsight, that may not have been the best idea. Now, I know patch 1.2 says that I improved the neural nets, and I did, sort of. I worked on a layer above the neural nets. I added new inputs and messed with some other stuff, but I never messed with the core neural net code. I also retrained the neural networks by having them play against each other over and over while having the AI platoons choose actions at random and feeding the results back through the neural network.

Before I go much further I should take the time to explain what a neural network is for any of you that may have no idea what it is. A neural network is basically our attempt at mimicking a cluster of neurons in code. The neurons in a brain receive signals from several other neurons and if those signals are above a threshold, the neuron fires off. A neuron in a neural network is much the same. The neurons in the neural networks that Supreme Commander 2 uses are set up as a feed forward network.

A feed forward network has layers of neurons; an input layer, one or more hidden layers, and an output layer. The input layer takes the inputs, feeds them to the hidden layer which then feeds to the next hidden layer until they finally feed to the output layer. When a neuron receives input from a neuron in a higher layer that input is weighted. These weights are how the neural networks learn. The networks learn which weights are more important for which outputs. They accomplish this via back propagation, which is where I found my first issue with the neural networks in Supreme Commander 2.

Back propagation basically goes back through the neural network and adjusts the input weights so that next time a more correct answer will be chosen. In order to do that you have to take the inputs from when the decision was made using the neural network and put them back into the neural network. You then feed the network forward so that the outputs are the same as when the decision was reached last time. Then the neural network goes back through, from the output layer to the input layer, and adjusts the weights as needed. Unfortunately, the function that was being called forgot the feed forward step, which completely invalidates any learning that was trying to be done.

In order to cope with this, there was code placed that would force the platoon to take an offensive action regardless of what the neural nets said if certain parameters were met. This is why the AI's platoons were seen to go back and forth after patch 1.2, I had forgotten to uncomment those sections of code. Even with that fixed, the neural networks still were not giving me the output I expected, until this afternoon.

I finally found the last piece of the puzzle. When a platoon encounters resistance it gathers up data about it's platoon and any enemies in the area. It then hands this info off to the neural network and gets an answer about what to do. When the platoon dies, kills all the enemies in the area, or hits its re-evaluation timeout, it sends the results to the neural network so that it can learn. It gets the results by doing the same thing it did when making the decision. It gathers the information about the platoon and the enemies in the area. The problem is, by the time it goes to report back to the neural network there could be a lot more enemies in the area. This skews the results.

So, I decided to try something different. When the platoon encounters resistance it gathers information about the platoon and enemies in the area. It also stores all of the enemies in the area. When it goes to report back to the neural network it only reports about the enemies that were in the area when the decision was made. We don't care about the platoon of gunships that just arrived after we killed that mass extractor, we only care that the decision to kill the mass extractor was a good one because it is now dead. We will deal with the gunships when we re-evaluate again. This change has made me a believer in neural nets again. They are performing exactly as I would expect and the hard-coded situations that force the platoons to attack are a thing of the past.

Before I left for the evening I had one more flash of insight. Among the hard-coded situations was one that made the AI's platoons attack an ACU if it saw one. Since I don't want those hard-coded situations anymore I had to come up with a way to still have that behavior, but not make it so two tanks think they can take on an ACU. Now, you would think the neural nets would learn that taking out the ACU is a good thing, and they probably would if it weren't for that fact that the ACU goes nuclear when it dies and kills almost the whole platoon that took it down. So, I made a change where if a platoon was told to attack the ACU during training and killed it, it will be bumped up as a good decision. This will make it so that those two tanks that encounter an ACU will run as expected, but if a Monkeylord finds your ACU, it is toast.

As an aside, I am being interviewed live over the internet by Alex Champandard of AIGameDev.com Sunday October 10th at 1PM PST. It will be available for viewing by Plus and Premium members of the site. So, if you happen to subscribe (Plus membership is like $7 a month), you should drop in.

11 comments:

Brian said... / September 30, 2010 at 10:04 PM  

You really should have finished off the neural net stuff with a Dr. Brackman quote, would have been epic, oh yes.

Good stuff, looking forward to seeing it in a future patch.

dawumyster said... / September 30, 2010 at 10:06 PM  

Very Insightful! So the big game decisions are decided via the neural net but the small stuff such as microing when are still done at the platoon level?

Also, I think the question just about everyone is asking is if these new updates will make it to SC2 or if it will only be used for K&C.

olivier said... / September 30, 2010 at 11:42 PM  

Thanks a lot Sorian. I think a White Paper with nice pictures of Neural Net System applied to STR game would be really helpfull to better understand how this works. But Yes this requires some time ! Thx anyway !!!

xenoc said... / October 1, 2010 at 12:37 AM  

Nice work Sorian.

Olivier - a quick google reveals several papers and degree-level theses on neural networks in RTS gaming.
e.g.
http://innovexpo.itee.uq.edu.au/2001/projects/s369674/thesis.pdf
http://www.gamesitb.com/nnpathgraham.pdf

Sorian said... / October 1, 2010 at 9:06 AM  

@dawumyster: The big game decisions are handled by other code. The platoon's fight or flight mechanics are handled by neural nets.

It is possible you may see these changes in SC2. No guarantees.

olivier said... / October 1, 2010 at 10:39 AM  

Thx for the links xenoc.

Sorian, what's happening if AI is facing human opponents having very different play styles ? Is NN able to quickly re-learn what to do when the opponent strategy is very different from a previous game session ?

Sorian said... / October 1, 2010 at 10:54 AM  

@olivier: The neural nets do not learn in the wild at the moment. It is an option for the future, though.

gralhas said... / October 1, 2010 at 1:46 PM  

Great article. Sorian really dominates when it comes to AI.

Also, isn't SupCom2 a good place to beta test many concepts that GPG wants to throw in KnC? Like AI and other features.

Brian said... / October 1, 2010 at 2:08 PM  

Well remember, GPG can test all kinds of new features for KnC using SC2, but it isn't up to GPG if they can release those in a patch, thats Square Enix. Square Enix doesn't care one bit about KnC so it isn't a given they will want to put stuff in a patch.

Luke Degner said... / October 1, 2010 at 5:41 PM  

That was a very nice read. Rock on, Sorian!

mocafrost said... / October 1, 2010 at 9:01 PM  

Yes is was a great read. SE has done a great job allowing GPG to support the game so far, so we could get lucky.

Post a Comment