Progress Report – November 2009

The month of November was very productive.  I began the month reviewing the RRL paper.  From there I reviewed from [Luger and Stubblefield 1993] the blocks world and its Prolog version of the blocks world planner.  Next step was to take the algorithm in Chapter 2, Figure 2.14 from [Russell and Norvig 1995], and to create an agent frame work in Prolog.  Upon review of [Covington, Nute, and Vellino 1997], I was able to apply advanced prolog tips to the agent frame work.  My initial objective is to have a functional blocks world planner in an agent frame work.  The basic planner agent has been created and returns an action.  However, the planner agent needs to generate a plan and return each action from the plan.  This needs further development; in other words, a work in progress.  The final blocks world planner agent will be posted upon completion.

Advance Prolog Techniques

After doing some Prolog programming for a while, there are some useful advanced Prolog techniques. First, tail recursion is very powerful. With the tail recursion it allows the Prolog compiler can optimize code without having to save the pointer for backtracking.  This is useful since it does not build the stack with up and crash the program with stack overflow errors.

Second, the conversion of complex structures into a list using the Univ operator.  I have worked with some complex data structures in my agent code for the blocks world program.  By converting the data structure into a list, it was easy to process the required functors.  Then upon process completion, the list was converted back to a data structure.

Agent Framework

As part of my ongoing agent training, I re-read the agent environment algorithm in Chapter 2 of AIMA (Russell and Norvig, 1995). The key challenge is how to convert an algorithm with repeat and for loops into viable Prolog code. At first inspection, the implementation is best suited for a language with control loops built into the programming language (e.g., LISP). My primary Prolog books [Bratko 1990] and [Sterling and Shapiro 1994] did not provide the required help. However, upon rereading Prolog Programming In Depth (Covington, Nute, and Vellino 1997), I was able to figure out how to write loops.

Blocks World

In order to understand the RRL paper better, I began to review the blocks world.  So I went back to review [Luger & Stubblefield 1993] on the discussions on the blocks world.  It was mostly used to illustrate propositional and predicate calculus.  In addition, there was a section on robotic planning using the frame problem and STRIPS as tools.  I reviewed the code base for robotic planning written in prolog.  So I began to enumerate all the possible states in the three blocks world.  Afterwards, I designed an test on how to move from an initial state to goal state.  I wrote the initial state, the action, the next state, etc. until the goal state was achieved.  Indeed, this was very simple for three blocks.

Progress Report – October 2009

In this month, I continued my studies in Relational Reinforcement Learning by reviewing the article Towards Informed Reinforcement Learning from the proceedings of the 2004 Machine Learning workshop of Relational Reinforcement Learning.   Basically the articles summarizes that an agent with limited information can find an optimal policy and can achieve a goal or goal states with limited information about its environment.  The experiments reported seems to suggest this type of exploration is possible.  According to Google Scholar search, there are 11 subsequent articles that reference this one.  In the RRL arena, my goal is to repeat the block’s world experiment as reported in Relational Reinforcement Learning article by Dzeroski, De Raedt, and Blockeel.