Progress Report – June 2010

This month’s focus is RRL Code, Model based RL Video, and Incomplete data structures.  The RRL Code is my Prolog implementation.  The Model based RL Video is Michael Littman’s video lectures from NIPS  2009, which available from video lectures dot net.  Finally, I read Chapter 15 on Incomplete Data Structures of the Sterling and Shapiro (1994) book.  Each of these topics have their own blog entry.

From video lectures I have started watching an An Introduction to Statistical Relational Learning by Dr. Lise Getoor and Policy Gradient Reinforcement Learning by Dr. Douglass Aberdeen.  From the NIPS 2009 conference, I watched Bootstrapping from Game Tree Search, and consequently the article.

RRL Code

In the Relational Reinforcement Learning [Dzeroski et. al.1998 and Dzeroski et. al. 2001], I reviewed the prolog code segments.  By definition, the state is a list of grounded terms.  The pre/2 predicate is intended to show the predictions are met in a similar fashion as in the STRIPS implementation.  If the given state, the pre/2 produces the move/2 term.  The delta/3 predicate produces the next state given the current state and move term.  My objective is to take this prolog code segments and implement it into SWI-Prolog.

Progress Report – December 2009

During this month I focused on three items – completing my blocks world planning agent with environment, transfer learning, and reinforcement learning.  First, as part on my ongoing understanding towards RRL, finally I completed the blocks world planning agent using SWI-Prolog v5.8.1.  The test_environment clause is currently set to move three blocks.  The planner agent uses a depth-first search to find the correct plan.  It takes nine steps to complete the operation.  My next step is to study [Sutton and Barto 1998].  The combination between Relational Learning (RL) or Inductive Logic Programming (ILP) and Reinforcement Learning (RL) was suggested by Kaebling and Sutton in 1997, which lead to the Relational Reinforcement Learning (RRL).

Progress Report – November 2009

The month of November was very productive.  I began the month reviewing the RRL paper.  From there I reviewed from [Luger and Stubblefield 1993] the blocks world and its Prolog version of the blocks world planner.  Next step was to take the algorithm in Chapter 2, Figure 2.14 from [Russell and Norvig 1995], and to create an agent frame work in Prolog.  Upon review of [Covington, Nute, and Vellino 1997], I was able to apply advanced prolog tips to the agent frame work.  My initial objective is to have a functional blocks world planner in an agent frame work.  The basic planner agent has been created and returns an action.  However, the planner agent needs to generate a plan and return each action from the plan.  This needs further development; in other words, a work in progress.  The final blocks world planner agent will be posted upon completion.

Progress Report – October 2009

In this month, I continued my studies in Relational Reinforcement Learning by reviewing the article Towards Informed Reinforcement Learning from the proceedings of the 2004 Machine Learning workshop of Relational Reinforcement Learning.   Basically the articles summarizes that an agent with limited information can find an optimal policy and can achieve a goal or goal states with limited information about its environment.  The experiments reported seems to suggest this type of exploration is possible.  According to Google Scholar search, there are 11 subsequent articles that reference this one.  In the RRL arena, my goal is to repeat the block’s world experiment as reported in Relational Reinforcement Learning article by Dzeroski, De Raedt, and Blockeel.

Progress Report – August 2009

During the month of August, I focused on Relational Reinforcement Learning, a field that combines Relational Learning and Reinforcement Learning fields.  Please read my blog entry for Relational Reinforcement Learning.  Afterwards, I have read a number of different articles regarding RRL learning from different authors, including the RRL workshop at the ICML 2004 conference.  My initial preference is the work performed and researched by Dr. Eduardo F. Morales.

I posted my review and comments regarding part I of the book The Art of Prolog (Sterling and Shapiro 1994).  There are very few books regarding Logic Programming in recent years.  Since the mid 1990s, much of the effort in Logic Programming has been with Inductive Logic Programming and Relational Learning.  The emerging field of Statistical Relational Learning has become a new field of research. 

Relational Reinforcement Learning

I start this month reading the articles in Relational Reinforcement Learning.  It began as a curiosity due to a response in the Reinforcement Learning mailing list.  The authors point a speech by Rich Sutton and Leslie Pack Kaelbling at the IJCAI 1997 conference in Japan, in which they recommended the combination of Induction Logic Programming (aka Relational Learning) and Reinforcement Learning.  The authors of the paper demonstrate this concept by using the blocks world domain to illustrate the combination of both fields.  Perhaps, this is exactly what I needed since I have done much reading, studying, and video lectures with logic programming, ILP, and reinforcement learning.  Of course, this paper spawn much researching according to CiteSeerX and Google Scholarly.

I found the web site to the 2004 Workshop in Relational Reinforcement Learning and downloaded the proceedings.

I shall continue this path of research interest.