Finally!

A couple of weeks ago I made a list of things I wanted to work on to start prototyping things for the Kings and Castles AI. So far, I made it to item number one. The first thing I wanted to work on was the neural net system. I was never really happy with it and, up until today, I was thinking about pulling it out of Kings and Castles. So, I decided I wanted to go into the core of the neural net system and see what might be up.

Up until two weeks ago I had been treating the core neural net system as a black box. I had assumed that the system was working as intended. In hindsight, that may not have been the best idea. Now, I know patch 1.2 says that I improved the neural nets, and I did, sort of. I worked on a layer above the neural nets. I added new inputs and messed with some other stuff, but I never messed with the core neural net code. I also retrained the neural networks by having them play against each other over and over while having the AI platoons choose actions at random and feeding the results back through the neural network.

Before I go much further I should take the time to explain what a neural network is for any of you that may have no idea what it is. A neural network is basically our attempt at mimicking a cluster of neurons in code. The neurons in a brain receive signals from several other neurons and if those signals are above a threshold, the neuron fires off. A neuron in a neural network is much the same. The neurons in the neural networks that Supreme Commander 2 uses are set up as a feed forward network.

A feed forward network has layers of neurons; an input layer, one or more hidden layers, and an output layer. The input layer takes the inputs, feeds them to the hidden layer which then feeds to the next hidden layer until they finally feed to the output layer. When a neuron receives input from a neuron in a higher layer that input is weighted. These weights are how the neural networks learn. The networks learn which weights are more important for which outputs. They accomplish this via back propagation, which is where I found my first issue with the neural networks in Supreme Commander 2.

Back propagation basically goes back through the neural network and adjusts the input weights so that next time a more correct answer will be chosen. In order to do that you have to take the inputs from when the decision was made using the neural network and put them back into the neural network. You then feed the network forward so that the outputs are the same as when the decision was reached last time. Then the neural network goes back through, from the output layer to the input layer, and adjusts the weights as needed. Unfortunately, the function that was being called forgot the feed forward step, which completely invalidates any learning that was trying to be done.

In order to cope with this, there was code placed that would force the platoon to take an offensive action regardless of what the neural nets said if certain parameters were met. This is why the AI's platoons were seen to go back and forth after patch 1.2, I had forgotten to uncomment those sections of code. Even with that fixed, the neural networks still were not giving me the output I expected, until this afternoon.

I finally found the last piece of the puzzle. When a platoon encounters resistance it gathers up data about it's platoon and any enemies in the area. It then hands this info off to the neural network and gets an answer about what to do. When the platoon dies, kills all the enemies in the area, or hits its re-evaluation timeout, it sends the results to the neural network so that it can learn. It gets the results by doing the same thing it did when making the decision. It gathers the information about the platoon and the enemies in the area. The problem is, by the time it goes to report back to the neural network there could be a lot more enemies in the area. This skews the results.

So, I decided to try something different. When the platoon encounters resistance it gathers information about the platoon and enemies in the area. It also stores all of the enemies in the area. When it goes to report back to the neural network it only reports about the enemies that were in the area when the decision was made. We don't care about the platoon of gunships that just arrived after we killed that mass extractor, we only care that the decision to kill the mass extractor was a good one because it is now dead. We will deal with the gunships when we re-evaluate again. This change has made me a believer in neural nets again. They are performing exactly as I would expect and the hard-coded situations that force the platoons to attack are a thing of the past.

Before I left for the evening I had one more flash of insight. Among the hard-coded situations was one that made the AI's platoons attack an ACU if it saw one. Since I don't want those hard-coded situations anymore I had to come up with a way to still have that behavior, but not make it so two tanks think they can take on an ACU. Now, you would think the neural nets would learn that taking out the ACU is a good thing, and they probably would if it weren't for that fact that the ACU goes nuclear when it dies and kills almost the whole platoon that took it down. So, I made a change where if a platoon was told to attack the ACU during training and killed it, it will be bumped up as a good decision. This will make it so that those two tanks that encounter an ACU will run as expected, but if a Monkeylord finds your ACU, it is toast.

As an aside, I am being interviewed live over the internet by Alex Champandard of AIGameDev.com Sunday October 10th at 1PM PST. It will be available for viewing by Plus and Premium members of the site. So, if you happen to subscribe (Plus membership is like $7 a month), you should drop in.

One year down

August 31st was my 1 year anniversary date with Gas Powered Games. My how the time has flown. This time last year I was working mostly in Lua wiring up the two new characters for Demigod. A couple of weeks after that I was learning C++. Now, a year later, I am spending most of my time in C++. This whole situation still has a dream-like quality to it. I still can't really believe I am here.

Patch 1.20 went out last week, finally. If you haven't seen the patch notes, you can find them here: http://www.supremecommander2.com/na/blog/2010/08/30/supreme-commander-2-patch-notes-v1-20/. A lot of changes went into this patch. I think my favorite change has to be the new AI options menu. Being able to custom tailor an AI like that is awesome and should be a standard feature for any game that features a skirmish mode.

Work still has not stopped on Supreme Commander 2. In fact, there was a video released yesterday that showed off a few things that will be included with the upcoming DLC pack. You can find the video on GameSpot's site here: http://www.gamespot.com/pc/strategy/supremecommander2/video/6275172/supreme-commander-2--strategies-from-team-gpg. In addition we are working on fixing the bugs that have been brought up after the latest patch as well as work on other super secret stuff that I am sure you will all enjoy.

More videos to come, more cool patches on the way, and the epic return of the Monkeylord is drawing nearer. It is a good time to be a SupCom fan.