CS 7642 Reinforcement learning – MDP – Finding the Optimal State-Value Function



The game DieN is played in the following way:

  1. You will be given a die with N sides. You will know the size of N, and can assume that N is a value greater than 1 and less than or equal to 30.
  2. You will be given a bit mask vector isBadSide representing the sides of a die that will make you lose. The vector will be of size N, and 1 indexed. (there is no 0 side)
  3. You start with 0 dollars.
  4. At any time you have the option to roll the die or to quit the game
  5. If you decide to roll:
  6. And you roll a number not in isBadSide, you receive that many dollars. (eg.
  7. if you roll the number 2 and 2 is not active — meaning the second element of the vector is 0 — in isBadSide, then you receive 2 dollars)
  8. Repeat step 4.
  9. And you roll a number in isBadSide, then you lose all money obtained in previous rolls and the game ends.
  10. If you decide to quit:

i. You keep all money gained from previous rolls and the game ends.


  • ● For this problem, determine an optimal policy for playing the game DieN for N sides. You will be given N and the array isBadSide which indicates which sides are bad. As you will see, the optimal policy for this game will depend on your current bankroll.
  • ● You can try solving this problem either by creating an MDP of the game (state, action, transition, reward function, and assume a gamma of 1) and then calculating the optimal


state-value function or you can plug-in values and solve directly using the Bellman


● What is the expected amount of dollars for this game if you follow an optimal policy? That

is, what is the optimal state-value function for the initial state of the game? Provide answers for the problems you are given in the “Solve for Code” tab on the Heroku site. Your answer must be correct to 3 decimal places.


The following examples can be used to verify your calculation is correct.

  • ● Input: N = 21, isBadSide = {1,1,1,1,0,0,0,0,1,0,1,0,1,1,0,1,0,0,0,1,0}, Output: 7.3799
  • ● Input: N = 22, isBadSide = {1,1,1,1,1,1,0,1,0,1,1,0,1,0,1,0,0,1,0,0,1,0}, Output: 6.314
  • ● Input: N = 6, isBadSide = {1,1,1,0,0,0}, Output: 2.5833
  • Resources
  • The concepts explored in this homework are covered by:
  • ● Lectures

○ Lesson 1: Smoov & Curly’s Bogus Journey

● Readings

○ Littman (1996)(chapters 1-2)

page 65 in the link http://www.incompleteideas.net/book/RLbook2018.pdf would be of additional help

and aditionally you can also use Pymdptoolbox


Calculate the price of your order

550 words
We'll send you the first draft for approval by September 11, 2018 at 10:52 AM
Total price:
The price is based on these factors:
Academic level
Number of pages
Basic features
  • Free title page and bibliography
  • Unlimited revisions
  • Plagiarism-free guarantee
  • Money-back guarantee
  • 24/7 support
On-demand options
  • Writer’s samples
  • Part-by-part delivery
  • Overnight delivery
  • Copies of used sources
  • Expert Proofreading
Paper format
  • 275 words per page
  • 12 pt Arial/Times New Roman
  • Double line spacing
  • Any citation style (APA, MLA, Chicago/Turabian, Harvard)

Our guarantees

Delivering a high-quality product at a reasonable price is not enough anymore.
That’s why we have developed 5 beneficial guarantees that will make your experience with our service enjoyable, easy, and safe.

Money-back guarantee

You have to be 100% sure of the quality of your product to give a money-back guarantee. This describes us perfectly. Make sure that this guarantee is totally transparent.

Read more

Zero-plagiarism guarantee

Each paper is composed from scratch, according to your instructions. It is then checked by our plagiarism-detection software. There is no gap where plagiarism could squeeze in.

Read more

Free-revision policy

Thanks to our free revisions, there is no way for you to be unsatisfied. We will work on your paper until you are completely happy with the result.

Read more

Privacy policy

Your email is safe, as we store it according to international data protection rules. Your bank details are secure, as we use only reliable payment systems.

Read more

Fair-cooperation guarantee

By sending us your money, you buy the service we provide. Check out our terms and conditions if you prefer business talks to be laid out in official language.

Read more
error: Content is protected !!