4. Neural Network

For our Neural Network model, we will apply advanced concepts. The idea of the model is to extract the structure and relationship between the Pokémon stats, types and their movesets, in order to predict who will be the winner in a battle, against another Pokémon. We do not include information about types strengths and weaknesses, neither the Pokémon generation or group classification (like Legendary, Mythical, Mega Evolution).

Our entire dataset contains 18,484 battles entries, between two Pokémon, from generations 1 to 8. Each Pokémon is represented as a table of 167 rows (each row is a learnable move by that Pokémon) and 110 columns (all features of stats, types, and moves details like power, pp, accuracy, status ailment, target, etc). It is important to note that we removed Mew from the dataset, since it is the only Pokémon capable of learning and using all movesets, being an outlier to the data.

Trained on 10,000 entries with Pokémon from generations 1 to 6, the model is able to capture the entire structure by its own and achieves an accuracy of 86% on new generation (7 and 8), unseen Pokémon in battle (8,000 entries).

Below we have a preview of the data used.
X is a tensor of 1216 x 167 x 110 and it works as a dictionary of moves per Pokémon:

• Dimension 0: 1216 Pokémon, including variants and mega-evolution status;
• Dimension 1: 167 moves, each Pokémon has all possible learnable moves and the rest is filled with 0;
• Dimension 2: 110 features, each combinaton of Pokémon and move has 110 features, which expresses its stats, Pokémon type, move type, power, pp, accuracy, status ailment, chance of activation, target, other move informations.

X.shape

TensorShape([1216, 167, 110])

Battles is a dataframe containing 18,515 entries of battles, all between two different Pokémon, indicating also who was the winner. This dataset does not contain mirrored entries, all of them are unique battles.
We removed from this dataset all battles involving Mew.

battles

	N_Pokemon_1	N_Pokemon_2	Winner
0	1	617	1
1	2	168	1
2	3	267	0
3	3	15	0
4	4	824	0
...	...	...	...
18510	445	248	0
18511	252	114	1
18512	253	212	1
18513	403	310	1
18514	887	376	1

18515 rows × 3 columns

For our training set we used battles involving Pokémon from generation 1 to 6 and for test set we used battles involving Pokémon from generation 7 and 8, mixed with all previous generations.
We also included data from generation 9 Pokémon and attacks, but they were not used.

In order to prepare the data, we had to create two inputs for training data and two inputs for test data, each one corresponding to a Pokémon battling. Each Pokémon is represented by a tensor of 167 rows (moves) x 110 columns (features).

train_set

	N_Pokemon_1	N_Pokemon_2	Winner
0	1	617	1
1	2	168	1
2	3	267	0
3	3	15	0
5	5	412	0
...	...	...	...
18509	396	16	0
18510	445	248	0
18511	252	114	1
18512	253	212	1
18513	403	310	1

12060 rows × 3 columns

test_set

	N_Pokemon_1	N_Pokemon_2	Winner
4	4	824	0
7	6	794	0
10	8	737	1
13	10	742	1
14	11	825	1
...	...	...	...
18501	889	60	0
18502	890	61	0
18503	890	62	0
18506	25	877	1
18514	887	376	1

6424 rows × 3 columns

4.1. Embedding Layer

Since we want to capture the structure of the battles, with 110 features for each Pokémon/move set, we need to embed this information. Word embedding is a well known concept, when dealing with NLP problems and recurrent neural networks (RNN). However, here instead of embedding words, we embed each feature importance. We do not use RNN since it is not applicable to this problem.

Therefore, during the training step, the model is capable of extracting the proximity and vector representation of each Pokémon feature and create a similar concept of Pokémon vector.

We use the same Embedding layer for two inputs, representing the two Pokémon battling, so both Pokémon have the same vector representation transformation applied.

4.2. N-grams to Movesets

One problem with our battles dataset is that in the game each Pokémon is capable of learning and using only four moves at the same time. However they are capable of learning around 100-150 different moves. Since we do not have the information of what move was used during each battle, we took a different approach.

Each Pokémon is represented by all moves it can learn, and to capture this information, we applied an N-gram concept. In NLP problems, we usually have 2-gram or 3-gram to extract the information of 2 or 3 words together.
In this problem each move represents a word and we capture the overall Pokémon information, based on the moves it can learn. Therefore we apply here a 167-gram for each Pokémon.

4.3. Neural Network Structure

We built our network using the Model API class of tensorflow, since it uses shared layer and its Sequential API does not support such structure.

Our model has 1 dense layer and contains around 4.7 million total parameters.
The model and its structure can be seen below.

model.summary()

Model: "Battle_Model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 Input_Pokemon_1 (InputLayer)   [(None, 167, 110)]   0           []                               
                                                                                                  
 Input_Pokemon_2 (InputLayer)   [(None, 167, 110)]   0           []                               
                                                                                                  
 Embedding_Model (Functional)   (None, 293920)       4096        ['Input_Pokemon_1[0][0]',        
                                                                  'Input_Pokemon_2[0][0]']        
                                                                                                  
 subtract (Subtract)            (None, 293920)       0           ['Embedding_Model[0][0]',        
                                                                  'Embedding_Model[1][0]']        
                                                                                                  
 Dense (Dense)                  (None, 16)           4702736     ['subtract[0][0]']               
                                                                                                  
 Output (Dense)                 (None, 1)            17          ['Dense[0][0]']                  
                                                                                                  
==================================================================================================
Total params: 4,706,849
Trainable params: 4,706,849
Non-trainable params: 0
__________________________________________________________________________________________________

tf.keras.utils.plot_model(model, show_shapes=True, show_layer_names=True)

4.4. Training and testing

For the training process we searched for a series of hyperparameters (not shown here). The final model is trained with a validation set of 20%.

Below we can see the training and testing sets performance, with an accuracy of 86% on the test set.

model.fit([in_train_1, in_train_2], train_labels, epochs=10, validation_split=0.2,
          callbacks=[early_stopping_monitor])

Epoch 1/10

/usr/local/lib/python3.8/dist-packages/tensorflow/python/data/ops/structured_function.py:264: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

302/302 [==============================] - 43s 141ms/step - loss: 0.3154 - accuracy: 0.8551 - val_loss: 0.3407 - val_accuracy: 0.8425
Epoch 2/10
302/302 [==============================] - 44s 146ms/step - loss: 0.2182 - accuracy: 0.9071 - val_loss: 0.3416 - val_accuracy: 0.8507
Epoch 3/10
302/302 [==============================] - 43s 142ms/step - loss: 0.1825 - accuracy: 0.9223 - val_loss: 0.3578 - val_accuracy: 0.8541
Epoch 4/10
302/302 [==============================] - 42s 140ms/step - loss: 0.1640 - accuracy: 0.9283 - val_loss: 0.3745 - val_accuracy: 0.8524

<keras.callbacks.History at 0x7f492eb3f880>

model.evaluate([in_test_1, in_test_2], test_labels)[1]

  3/201 [..............................] - ETA: 7s - loss: 0.4776 - accuracy: 0.8542 

/usr/local/lib/python3.8/dist-packages/tensorflow/python/data/ops/structured_function.py:264: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`.
  warnings.warn(

201/201 [==============================] - 7s 36ms/step - loss: 0.3658 - accuracy: 0.8636

0.8636363744735718

We can also check the confusion matrix generated by the testing predictions.

predictions = model.predict([in_1, in_2])

201/201 [==============================] - 7s 36ms/step

conf_matrix = tf.math.confusion_matrix(labels, np.round(predictions))
conf_matrix

<tf.Tensor: shape=(2, 2), dtype=int32, numpy=
array([[3819,  549],
       [ 327, 1729]], dtype=int32)>

mosaic(conf_matrix.numpy())
plt.show()

In this picture, the lower left quadrant represents cases where the first Pokémon was the winner, and it was predicted correctly (3,819 occurrences).
The upper right quadrant represents cases where the Second Pokémon was the winner, and it was predicted correctly (1,729 occurrences).

5. Conclusions

This problem presented to be quite challenging. During its development, we had to analyze and adapt neural networks concepts like Embedding layers and N-grams representation, and apply them to a different point of view and applications.

We also had to work around memory usage, since our entire battles dataset contained 2 x 18,484 x 167 x 110 values (although it was sparse), representing almost 680 billion data points.

The solution presented here can also be applied to a serious of real word problems, like sports competition predictors and market, sales, retail and customers stratification / clustering analysis.