A Policy-Gradient algorithm that solves Contextual Bandit problems.

ContextualPolicy.ipynb

A below line occurs an error. Tensorflow version on my local is 0.11.

output = slim.fully_connected(state_in_OH,a_size,\
            biases_initializer=None,activation_fn=tf.nn.sigmoid,weights_initializer=tf.ones)

Traceback (most recent call last):
  File "c_bendits.py", line 50, in <module>
    myAgent = agent(lr=0.001,s_size=cBandit.num_bandits,a_size=cBandit.num_actions) #Load the agent.
  File "c_bendits.py", line 34, in __init__
    biases_initializer=None,activation_fn=tf.nn.sigmoid,weights_initializer=tf.ones)
...
  File "/Library/Python/2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 666, in <lambda>
    shape.as_list(), dtype=dtype, partition_info=partition_info)
TypeError: ones() got an unexpected keyword argument 'partition_info'

So the line should be below codes.

output = slim.fully_connected(state_in_OH,a_size,\
            biases_initializer=None,activation_fn=tf.nn.sigmoid,weights_initializer=tf.ones_initializer)

These tutorials of yours are quite awesome and i am really loving them. in addition to the work you have done so far, can i suggest that since this tutorial material is designed to be read by beginners as part of a tutorial, would it not be more accommodating to use full variable names that represent the data it contains? for example, the first thing i struggled with was figuring out what lr meant in the parameters to the agent class's init, and the conciseness of the shortened name made it even more difficult to find where it is used in the code. I guessed that s_size and a_size were state size and action size, but i think it was an unnecessary barrier to understanding the actual content, as well as state_in_OH (i.e. the OH).

What do you think about this suggestion? i am hoping it can help others learn and understand the content better

also, i didn't see anything on one hot encoding in the post, is it perhaps in one of your other posts?

This is a great demo. Can you also suggest how to I store the model as .h5 file, (like in Keras), and re-use it ?

@alphashuro agreed. I struggle the same prob what is OH mean?

Hi, I'm learning RL with your articles, great work 👍

Here is a quick diff to use raw TF (as of 1.3) instead of slim :

-        state_in_OH = slim.one_hot_encoding(self.state_in, s_size)
-        output = slim.fully_connected(state_in_OH,
-                                      a_size,
-                                      biases_initializer=None,
-                                      activation_fn=tf.nn.sigmoid,
-                                      weights_initializer=ones)
+        state_in_OH = tf.one_hot(self.state_in, s_size)
+        output = tf.layers.dense(state_in_OH, a_size, tf.nn.sigmoid,
+                                 use_bias=False, kernel_initializer=ones)

@Riotpiaole OH = one hot [encoding]

According to my experiment (tensorflow 1.3), I suggest to use AdamOptimizer instead of GradientDescentOptimizer since GradientDescentOptimizer suffers from training stability issue.

@Riotpiaole I've re-implement the tutorial codes here, you may take a look at it.

can anyone explain to me why we do not use softmax instead of sigmoid? and also why we don't use bias?(I tried both and it wouldn't work)

@lipixun do you know the answer to my question? it would really help me thanks

@pooriaPoorsarvi as seen above we already got the responsible_weight variable, now we are getting the negative
Log likelihood to optimize for the maxium (tf only can optimize) no need to consider every other classes

Instead of using slim, can use tf as:
state_in_OH = tf.one_hot(self.state_in, s_size)
output = tf.layers.dense(state_in_OH, a_size, tf.nn.sigmoid, use_bias=False, kernel_initializer = tf.ones_initializer())

Thanks Arthur! this is helpful tutorial for beginers like me. Here is tensorflow 2 implementation may be helpful for someone

Thanks Arthur! this is helpful tutorial for beginers like me. Here is tensorflow 2 implementation may be helpful for someone

Thanks for the implementation. I wonder how is the implementation a policy network? I don't see policy gradient is used.