After doing some Prolog programming for a while, there are some useful advanced Prolog techniques. First, tail recursion is very powerful. With the tail recursion it allows the Prolog compiler can optimize code without having to save the pointer for backtracking.  This is useful since it does not build the stack with up and crash the program with stack overflow errors.

Second, the conversion of complex structures into a list using the Univ operator.  I have worked with some complex data structures in my agent code for the blocks world program.  By converting the data structure into a list, it was easy to process the required functors.  Then upon process completion, the list was converted back to a data structure.

As part of my ongoing agent training, I re-read the agent environment algorithm in Chapter 2 of AIMA (Russell and Norvig, 1995). The key challenge is how to convert an algorithm with repeat and for loops into viable Prolog code. At first inspection, the implementation is best suited for a language with control loops built into the programming language (e.g., LISP). My primary Prolog books [Bratko 1990] and [Sterling and Shapiro 1994] did not provide the required help. However, upon rereading Prolog Programming In Depth (Covington, Nute, and Vellino 1997), I was able to figure out how to write loops.

Hence, I wrote a draft of the prolog code in paper. Next, I prepared my test Linux environment and using a text editor put the code into files. Next, I had to write a test file to test the individual functions. After much debugging, revising, and rewriting, I had a functional running environment for an agent framework.

Enclosed is my code (only displayed for educational, noncommercial purposes):

/*
 * blockenv.pl
 */

run_environment(State, Updatefn, Agents, Termination) :-
   get_percepts(Agents, State, Percepts),
   get_actions(Percepts, Actions),
   call(Updatefn, Actions, State, NewState),
   !,
   (call(Termination, NewState) -> true ;
   run_environment(NewState, Updatefn, Agents, Termination)).

get_percepts([], State, []).

get_percepts([Agent|Agents], State, Percepts) :-
    get_percepts(Agents, State, Percepts1),
    append([percept(Agent, State)], Percepts1, Percepts).

get_actions([], []).

get_actions([Percept|Percepts], Actions) :-
    get_actions(Percepts, Actions1),
    arg(1, Percept, Agent),
    arg(2, Percept, AgentPercept),
    call(Agent, AgentPercept, Action),
    append([action(Agent, Action)], Actions1, Actions).

/*
 * blktest.pl
 */

:- consult('../environment/blockenv.pl').

test_percepts(Percepts) :-
   get_percepts([planner], [handempty, clear(a), ontable(a), counter(1)], Percepts).

test_actions(Actions) :-
   test_percepts(Percepts),
   get_actions(Percepts, Actions).

planner(_, noaction).

update_counter([], []).

update_counter([H|Rest], NewState) :-
   update_counter(Rest, NewState1),
   ( counter(X) = H -> Y is X - 1 ,
     append([counter(Y)], NewState1, NewState)
   ;
     append([H], NewState1, NewState)
   ).

update_state(_, State, NewState) :-
   update_counter(State, NewState).

termination(State) :-
   member(counter(0), State).

test_environment :-
   run_environment([handempty, counter(2)], update_state, [planner],
           termination).

Enclosed is a test trace.

[trace] 5 ?- test_environment.
 T Call: (7) test_environment
   Call: (7) test_environment ? creep
   Call: (8) run_environment([handempty, counter(2)], update_state, [planner], termination) ? creep
   Call: (9) get_percepts([planner], [handempty, counter(2)], _L214) ? creep
   Call: (10) get_percepts([], [handempty, counter(2)], _L238) ? creep
   Exit: (10) get_percepts([], [handempty, counter(2)], []) ? creep
   Call: (10) lists:append([percept(planner, [handempty, counter(2)])], [], _L214) ? creep
   Exit: (10) lists:append([percept(planner, [handempty, counter(2)])], [], [percept(planner, [handempty, counter(2)])]) ? creep
   Exit: (9) get_percepts([planner], [handempty, counter(2)], [percept(planner, [handempty, counter(2)])]) ? creep
   Call: (9) get_actions([percept(planner, [handempty, counter(2)])], _L215) ? creep
   Call: (10) get_actions([], _L237) ? creep
   Exit: (10) get_actions([], []) ? creep
   Call: (10) arg(1, percept(planner, [handempty, counter(2)]), _L238) ? creep
   Exit: (10) arg(1, percept(planner, [handempty, counter(2)]), planner) ? creep
   Call: (10) arg(2, percept(planner, [handempty, counter(2)]), _L239) ? creep
   Exit: (10) arg(2, percept(planner, [handempty, counter(2)]), [handempty, counter(2)]) ? creep
   Call: (10) planner([handempty, counter(2)], _L240) ? creep
   Exit: (10) planner([handempty, counter(2)], noaction) ? creep
   Call: (10) lists:append([action(planner, noaction)], [], _L215) ? creep
   Exit: (10) lists:append([action(planner, noaction)], [], [action(planner, noaction)]) ? creep
   Exit: (9) get_actions([percept(planner, [handempty, counter(2)])], [action(planner, noaction)]) ? creep
   Call: (9) update_state([action(planner, noaction)], [handempty, counter(2)], _L216) ? creep
   Call: (10) update_counter([handempty, counter(2)], _L216) ? creep
   Call: (11) update_counter([counter(2)], _L255) ? creep
   Call: (12) update_counter([], _L278) ? creep
   Exit: (12) update_counter([], []) ? creep
   Call: (12) counter(_G536)=counter(2) ? creep
   Exit: (12) counter(2)=counter(2) ? creep
^  Call: (12) _L280 is 2-1 ? creep
^  Exit: (12) 1 is 2-1 ? creep
   Call: (12) lists:append([counter(1)], [], _L255) ? creep
   Exit: (12) lists:append([counter(1)], [], [counter(1)]) ? creep
   Exit: (11) update_counter([counter(2)], [counter(1)]) ? creep
   Call: (11) counter(_G549)=handempty ? creep
   Fail: (11) counter(_G549)=handempty ? creep
   Call: (11) lists:append([handempty], [counter(1)], _L216) ? creep
   Exit: (11) lists:append([handempty], [counter(1)], [handempty, counter(1)]) ? creep
   Exit: (10) update_counter([handempty, counter(2)], [handempty, counter(1)]) ? creep
   Exit: (9) update_state([action(planner, noaction)], [handempty, counter(2)], [handempty, counter(1)]) ? creep
   Call: (9) termination([handempty, counter(1)]) ? creep
   Call: (10) lists:member(counter(0), [handempty, counter(1)]) ? creep
   Fail: (10) lists:member(counter(0), [handempty, counter(1)]) ? creep
   Fail: (9) termination([handempty, counter(1)]) ? creep
   Call: (9) run_environment([handempty, counter(1)], update_state, [planner], termination) ? creep
   Call: (10) get_percepts([planner], [handempty, counter(1)], _L237) ? creep
   Call: (11) get_percepts([], [handempty, counter(1)], _L261) ? creep
   Exit: (11) get_percepts([], [handempty, counter(1)], []) ? creep
   Call: (11) lists:append([percept(planner, [handempty, counter(1)])], [], _L237) ? creep
   Exit: (11) lists:append([percept(planner, [handempty, counter(1)])], [], [percept(planner, [handempty, counter(1)])]) ? creep
   Exit: (10) get_percepts([planner], [handempty, counter(1)], [percept(planner, [handempty, counter(1)])]) ? creep
   Call: (10) get_actions([percept(planner, [handempty, counter(1)])], _L238) ? creep
   Call: (11) get_actions([], _L260) ? creep
   Exit: (11) get_actions([], []) ? creep
   Call: (11) arg(1, percept(planner, [handempty, counter(1)]), _L261) ? creep
   Exit: (11) arg(1, percept(planner, [handempty, counter(1)]), planner) ? creep
   Call: (11) arg(2, percept(planner, [handempty, counter(1)]), _L262) ? creep
   Exit: (11) arg(2, percept(planner, [handempty, counter(1)]), [handempty, counter(1)]) ? creep
   Call: (11) planner([handempty, counter(1)], _L263) ? creep
   Exit: (11) planner([handempty, counter(1)], noaction) ? creep
   Call: (11) lists:append([action(planner, noaction)], [], _L238) ? creep
   Exit: (11) lists:append([action(planner, noaction)], [], [action(planner, noaction)]) ? creep
   Exit: (10) get_actions([percept(planner, [handempty, counter(1)])], [action(planner, noaction)]) ? creep
   Call: (10) update_state([action(planner, noaction)], [handempty, counter(1)], _L239) ? creep
   Call: (11) update_counter([handempty, counter(1)], _L239) ? creep
   Call: (12) update_counter([counter(1)], _L278) ? creep
   Call: (13) update_counter([], _L301) ? creep
   Exit: (13) update_counter([], []) ? creep
   Call: (13) counter(_G573)=counter(1) ? creep
   Exit: (13) counter(1)=counter(1) ? creep
^  Call: (13) _L303 is 1-1 ? creep
^  Exit: (13) 0 is 1-1 ? creep
   Call: (13) lists:append([counter(0)], [], _L278) ? creep
   Exit: (13) lists:append([counter(0)], [], [counter(0)]) ? creep
   Exit: (12) update_counter([counter(1)], [counter(0)]) ? creep
   Call: (12) counter(_G586)=handempty ? creep
   Fail: (12) counter(_G586)=handempty ? creep
   Call: (12) lists:append([handempty], [counter(0)], _L239) ? creep
   Exit: (12) lists:append([handempty], [counter(0)], [handempty, counter(0)]) ? creep
   Exit: (11) update_counter([handempty, counter(1)], [handempty, counter(0)]) ? creep
   Exit: (10) update_state([action(planner, noaction)], [handempty, counter(1)], [handempty, counter(0)]) ? creep
   Call: (10) termination([handempty, counter(0)]) ? creep
   Call: (11) lists:member(counter(0), [handempty, counter(0)]) ? creep
   Exit: (11) lists:member(counter(0), [handempty, counter(0)]) ? creep
   Exit: (10) termination([handempty, counter(0)]) ? creep
   Call: (10) true ? creep
   Exit: (10) true ? creep
   Exit: (9) run_environment([handempty, counter(1)], update_state, [planner], termination) ? creep
   Exit: (8) run_environment([handempty, counter(2)], update_state, [planner], termination) ? creep
 T Exit: (7) test_environment
   Exit: (7) test_environment ? creep
true.

In order to understand the RRL paper better, I began to review the blocks world.  So I went back to review [Luger & Stubblefield 1993] on the discussions on the blocks world.  It was mostly used to illustrate propositional and predicate calculus.  In addition, there was a section on robotic planning using the frame problem and STRIPS as tools.  I reviewed the code base for robotic planning written in prolog.  So I began to enumerate all the possible states in the three blocks world.  Afterwards, I designed an test on how to move from an initial state to goal state.  I wrote the initial state, the action, the next state, etc. until the goal state was achieved.  Indeed, this was very simple for three blocks.

From there, I started my SWI Prolog interpreter, loaded the planner code, and tested using my initial state and goal state.  The planner returned with the expected path.  My next test was to setup a four block world using an initial state to a goal state.  The results were interesting because the from the initial state to the next few states the planner produced actions as expected.  Then the planner performed actions that are considered superfluous.  My conclusion is that reinforcement learning can help with by eliminating these additional actions and achieve the goal state sooner.

I present the 13 states for the three blocks world.

s0  = { handempty, clear(a), clear(b), clear(c), ontable(a), ontable(b), ontable(c) }
s1  = { handempty, clear(a), clear(b), on(b, c), ontable(a), ontable(c) }
s2  = { handempty, clear(a), clear(c), on(c, b), ontable(a), ontable(b) }
s3  = { handempty, clear(a), clear(b), on(a, c), ontable(b), ontable(c) }
s4  = { handempty, clear(a), clear(c), on(a, b), ontable(b), ontable(c) }
s5  = { handempty, clear(b), clear(c), on(b, a), ontable(a), ontable(c) }
s6  = { handempty, clear(b), clear(c), on(c, a), ontable(a), ontable(b) }
s7  = { handempty, clear(a), on(a, b), on(b, c), ontable(c) }
s8  = { handempty, clear(a), on(a, c), on(c, b), ontable(b) }
s9  = { handempty, clear(b), on(b, a), on(a, c), ontable(c) }
s10 = { handempty, clear(b), on(b, c), on(c, a), ontable(a) }
s11 = { handempty, clear(c), on(c, a), on(a, b), ontable(b) }
s12 = { handempty, clear(c), on(c, b), on(b, a), ontable(a) }

Enclosed is the states and actions from s10 to s7:

Initial state: s10
Action: pickup(b), putdown(b)
Next state: s6
Action: pickup(c), putdown(c)
Next state: s0
Action: pickup(b), stack(b, c)
Next state: s1
Action: pickup(a), stack(a, b)
Final state: s7

Starting at state [handempty, ontable(a), on(d,a), on(c,d), on(b,c), clear(b)] to final state [handempty, ontable(d), on(c,d), on(b,c), on(a,b), clear(a)],  the planner program produced the following actions:

pickup(b)
putdown(b)
pickup(c)
putdown(c)
pickup(d)
putdown(d)
pickup(a)
stack(a, d)
pickup(c)
stack(c, b)
pickup(a)
putdown(a)
pickup(c)
stack(c, a)
pickup(b)
stack(b, d)
pickup(c)
putdown(c)
pickup(b)
stack(b, a)
pickup(d)
stack(d, c)
pickup(b)
putdown(b)
pickup(a)
stack(a, b)
pickup(d)
putdown(d)
pickup(c)
stack(c, d)
pickup(a)
putdown(a)
pickup(b)
stack(b, c)
pickup(a)
stack(a, b)

In this month, I continued my studies in Relational Reinforcement Learning by reviewing the article Towards Informed Reinforcement Learning from the proceedings of the 2004 Machine Learning workshop of Relational Reinforcement Learning.   Basically the articles summarizes that an agent with limited information can find an optimal policy and can achieve a goal or goal states with limited information about its environment.  The experiments reported seems to suggest this type of exploration is possible.  According to Google Scholar search, there are 11 subsequent articles that reference this one.  In the RRL arena, my goal is to repeat the block’s world experiment as reported in Relational Reinforcement Learning article by Dzeroski, De Raedt, and Blockeel.

Also, I have been focused on Robert Winkler’s book An Introduction to Bayesian Analysis and Inference.  From the Winkler book, my goal is to gain a better understanding of Bayesian Inference to better understand the Decision Theoretic models of machine learning.  In the machine learning world, Bayesian approaches to various problems are resulting interesting solutions to various problems in Multiagent Reinforcement Learning.

There are some interesting articles in the Journal of Bayesian Analysis and in JAIR.

In another topic of interest, I am continuing my reintroduction of LISP by reading the Patrick Winston and Berthold Horn classic.

From a statistical point of view, this month achieve a new record number of hits with 806 hits, boosting my total for the current year to over 3200 hits.  The hotest pages are my About page, followed by Wumpus World and Wumpus World Revisited.

Thank you for your interest and support.

This month marks my first year in utilizing wordpress to host my blog.  It has been a great journey so far and hopefully will be better this coming year.

The month of September has been busy.  Unfortunately I did not write any blog entries due to my busy schedule.  However, from statistical point of view, my blog had the second highest total of number of views (362 in total).  Also, I covered various topics this month with Probability and Statistics, Bayesian Inference, Reinforcement Learning, and finally a LISP review.

I started to review the Reinforcement Learning book (Sutton and Barto 1997) only reading Part 1 and part of Part 2.  I did download the lisp code associated with the book.  I ran the tic-tac-toe program in my test environment.

In addition, I read Part 1 of the PAIP (Norving 1992).  What this book forced me to review LISP (Winston and Horn 1989), only covering the first three chapters in just a few days.

Also I completed the third chapter of Inductive Logic Programming (Bergadano and Gunetti 1996), which covered bottom up ILP methods.  Material is diverse since the authors integrated many papers together to discuss the important bottom up methods.

Next I completed the first four chapters of  Introduction to Bayesian Inference and Decision (Winkler 2003).  The books covers probability theory as well as the the introduction to Bayes Theorem.  Chapter 3 covers Bayesian Inference for Discrete Probability Models.  Very excellent material for those of you interested in Bayesian Inference and Decision as to gain insights into Decision Theoretic models.  The exercises are very good as well.  Looking forward to completing book followed by reading AIMA Part V.  For interested in Bayesian Methods, download the Journal of Bayesian Analysis.

All this material has been great.  I started thinking about some ideas such as using an agent approach for the farmer, wolf, goat, and cabbage problem.  Although it may seem a bit overkill, I was interested in seeing various methods to this problem with reinforcement learning methods.

The hottest page was my About page, followed by Wumpus World, AI Links page, the Warnsdorf algorithm, and last month’s progress report.

Thank you for your interest and support.

During the month of August, I focused on Relational Reinforcement Learning, a field that combines Relational Learning and Reinforcement Learning fields.  Please read my blog entry for Relational Reinforcement Learning.  Afterwards, I have read a number of different articles regarding RRL learning from different authors, including the RRL workshop at the ICML 2004 conference.  My initial preference is the work performed and researched by Dr. Eduardo F. Morales.

I posted my review and comments regarding part I of the book The Art of Prolog (Sterling and Shapiro 1994).  There are very few books regarding Logic Programming in recent years.  Since the mid 1990s, much of the effort in Logic Programming has been with Inductive Logic Programming and Relational Learning.  The emerging field of Statistical Relational Learning has become a new field of research.  In addition, the prolog language has been very stable since the ISO Prolog standard was established.  Some the prolog has minor differences with certain predefined predicates, but it has been compatible with Edinbrugh Prolog.  The current Prolog books cover some basic AI applications such as games, mini expert systems, and meta interpreters.  What is needed is agent/environment based applications using modern tools such as Q-Learning or TD-methods.

I finally completed Scott Sanner’s video lecture in Reinforcement Learning.  An excellent tutorial.  The lecture was discussing topics such as model based and model free methods.  The subjects MDPs, dynamic programming, Monte Carlo, temporal difference, and function approximations are discussed.

I also performed research on the Warnsdorf algorithm and posted my initial results.  Most of this work was done using procedural languages and with square boards of size five and greater.

In addition, I found the ICML-06 Workshop on Structural Knowledge Transfer for Machine Learning.

I finally completed the Summer Schools in Logic and Learning (SSLL 09) video lecture for Reinforcement Learning presented by Scott Sanner.  It was a good introduction into Reinforcement Learning.  It is a six part lecture.  Plan for time to watch the videos.  He starts with an introduction into the topic.  Next is model based solutions with Markov Decision Process (MDP) and dynamic programming (DP) with value iteration and policy iteration.  The Bellman equations are introduced.  Lastly, he reviews model free approaches from Chapters 5, 6, and 7 of the Reinforcement Learning: An Introduction (Sutton and Barto 1998).  He covers Monte Carlo and Temporal Difference (TD) algorithms, in particular TD lamdba and SARSA.  However due to large state spaces of problems, he discussion function approximators (both linear and nonlinear) along with gradient descend approaches.  A worth while video lecture.

As part of my follow research to my paper, I did some preliminary research for the Warnsdorf’s algorithm for finding knight’s tours.  I found three relevant papers on the subject (Pohl 1967, Pohl-Stockmeyer 2004, and Ganzfried 2004).  Basically, the Warnsdorf algorithm is to traverse the path with least degree.  In the case of a tie, the a path is randomly selected.  In reality, the Warnsdorf algorithm is not consistently successful as pointed out by (Ganzfried 2004).  The (Ganzfried 2004) paper modifies the Warnsdorf algorithm with key squares in which the move order changes, which improves finding knight tours.  The next step is generate a prolog program using the Pohl-Warnsdorf and Ganzfried algorithms.

The book The Art of Prolog (Sterling and Shapiro 1994) is an excellent treatise in the subject of Logic Programming and Prolog.  The book is divided into four parts – Logic Programming, The Prolog Language, Advance Prolog Programming Techniques, and Applications.

Beginning with Logic Programming, chapter one provides the best introduction into Logic Programming as well as the first definition of the abstract interpreter for logic programming.  The definitons for term, functor, compound term, clause or rules, logic programs, and the meaning for logic programs are presented.  The summary is worth reading.

Chapter Two goes into database programming and it connection to logic programming.

Next is Recursion in Chapter Three, which illustrates many techniques and links the original logic programming symbols.  For example the definition of a natural number is recursive; e.g. s(0), which is the successor function.  Many example programs are illustrated in this chapter.

The fourth chapter covers unification and expands on the simple abstract interpreter for logic programming.

The section ends with a chapter on Theory of Logic Programming by introducing terms such as the Hebrand Base and Universe of a Logic Program.  Each Chapter has a background section explaining the various directions and original papers on the subject matter.  So far, a very good book to read on logic programming.

I start this month reading the articles in Relational Reinforcement Learning.  It began as a curiosity due to a response in the Reinforcement Learning mailing list.  The authors point a speech by Rich Sutton and Leslie Pack Kaelbling at the IJCAI 1997 conference in Japan, in which they recommended the combination of Induction Logic Programming (aka Relational Learning) and Reinforcement Learning.  The authors of the paper demonstrate this concept by using the blocks world domain to illustrate the combination of both fields.  Perhaps, this is exactly what I needed since I have done much reading, studying, and video lectures with logic programming, ILP, and reinforcement learning.  Of course, this paper spawn much researching according to CiteSeerX and Google Scholarly.

I found the web site to the 2004 Workshop in Relational Reinforcement Learning and downloaded the proceedings.

I shall continue this path of research interest.