Progress Report – January 2010

In this month I wrote blog entries for Agent Design and EASSS09.  The agent design entry questions on the appropriate programming language to develop an agent and its environment.  I concluded that Java is an appropriate programming language.  Java has become a popular programming language for teaching computer science in academia, and industry has accepted Java as the appropriate programming language for developing and implementing web based applications as well as batch processing.

The EASSS09 entry refers to the European Agent Systems Summer School for learning agents.  The material that I have reviewed thus far has been excellent due to the broaden view of agent and multi-agent design.  The instructors provide excellent materials and references for the courses.

EASSS09

In this past summer in Turino, Italy, a summer school for Agent Systems was offered.  I reached this site when I was searching for other Google AI groups to join and found the news group for the European Agent System Laboratory.  The group had a list of recommended books for students to get in the study of Agents and Multi-Agent Systems.  Also, the site made a reference to the European Agent Systems Summer School.  I downloaded the course notes from the site with some interesting material.  The material was very excellent, and it is a worthwhile material to review to learn more about agent and multi-agent systems.

Agent Design

As I was surfing the web for articles with Google Scholar, I encounter a course syllabus regarding Designing Intelligent Agents using Java.  My first question was Java the appropriate choice for creating software agents.  Does Java offer certain advantages over the traditional programming languages such as Prolog and LISP?  Based on my experience with creating an agent using Prolog, the main disadvantage with Prolog is the lack of object oriented abstraction.  I had to take a structure and using the univ operator convert the structure into a list, which in my opinion is major performance hit.  Although LISP with the CLOS has the object oriented features, it lacks a multi-threading option for each software agents to run in its own thread.  Of course my other options are procedural languages of C++ and Java.  The C++ language requires the use of thread library that is not platform independent.  Hence, the natural choice is Java.  The down side of these procedural languages is lack of the power offered  by LISP and Prolog.  At this point I will rewrite my blocks world planner agent in Java.

Progress Report – December 2009

During this month I focused on three items – completing my blocks world planning agent with environment, transfer learning, and reinforcement learning.  First, as part on my ongoing understanding towards RRL, finally I completed the blocks world planning agent using SWI-Prolog v5.8.1.  The test_environment clause is currently set to move three blocks.  The planner agent uses a depth-first search to find the correct plan.  It takes nine steps to complete the operation.  My next step is to study [Sutton and Barto 1998].  The combination between Relational Learning (RL) or Inductive Logic Programming (ILP) and Reinforcement Learning (RL) was suggested by Kaebling and Sutton in 1997, which lead to the Relational Reinforcement Learning (RRL).

Transfer Learning

For some time in my research, I came across a hot topic called Transfer Learning.  The idea is not new, but has been a key question in Artificial Intelligence.  The main question behind transfer learning is that an agent can learn a policy or solution in one domain and transfer this knowledge into another domain as background knowledge.  Humans have the uncanny ability to learn solutions in one domain and transfer them to new domains.  Other important issues are the measurement of the agent’s performance with this background knowledge.  One potential problem to address is the issue of negative transfer in which the agent’s performance declines as a result of the new background knowledge.

Blocks World Planner

Finally, I have completed the blocks world planner agent code.  The code runs under SWI-Prolog version v5.8.1.  The three blocks world was tested.  Other block worlds world can used by simply modifying the initial and goal states.  The environment code was based on the AIMA environment algorithm and agent planner code was based on Luger’s implementation with some modifications due to the data structures and the separation between the agent planner and its update function.  This is the first step toward the RRL.  I will continue with RL experience by studying [Sutton and Barto 1998].

Progress Report – November 2009

The month of November was very productive.  I began the month reviewing the RRL paper.  From there I reviewed from [Luger and Stubblefield 1993] the blocks world and its Prolog version of the blocks world planner.  Next step was to take the algorithm in Chapter 2, Figure 2.14 from [Russell and Norvig 1995], and to create an agent frame work in Prolog.  Upon review of [Covington, Nute, and Vellino 1997], I was able to apply advanced prolog tips to the agent frame work.  My initial objective is to have a functional blocks world planner in an agent frame work.  The basic planner agent has been created and returns an action.  However, the planner agent needs to generate a plan and return each action from the plan.  This needs further development; in other words, a work in progress.  The final blocks world planner agent will be posted upon completion.

Advance Prolog Techniques

After doing some Prolog programming for a while, there are some useful advanced Prolog techniques. First, tail recursion is very powerful. With the tail recursion it allows the Prolog compiler can optimize code without having to save the pointer for backtracking.  This is useful since it does not build the stack with up and crash the program with stack overflow errors.

Second, the conversion of complex structures into a list using the Univ operator.  I have worked with some complex data structures in my agent code for the blocks world program.  By converting the data structure into a list, it was easy to process the required functors.  Then upon process completion, the list was converted back to a data structure.

Agent Framework

As part of my ongoing agent training, I re-read the agent environment algorithm in Chapter 2 of AIMA (Russell and Norvig, 1995). The key challenge is how to convert an algorithm with repeat and for loops into viable Prolog code. At first inspection, the implementation is best suited for a language with control loops built into the programming language (e.g., LISP). My primary Prolog books [Bratko 1990] and [Sterling and Shapiro 1994] did not provide the required help. However, upon rereading Prolog Programming In Depth (Covington, Nute, and Vellino 1997), I was able to figure out how to write loops.

Blocks World

In order to understand the RRL paper better, I began to review the blocks world.  So I went back to review [Luger & Stubblefield 1993] on the discussions on the blocks world.  It was mostly used to illustrate propositional and predicate calculus.  In addition, there was a section on robotic planning using the frame problem and STRIPS as tools.  I reviewed the code base for robotic planning written in prolog.  So I began to enumerate all the possible states in the three blocks world.  Afterwards, I designed an test on how to move from an initial state to goal state.  I wrote the initial state, the action, the next state, etc. until the goal state was achieved.  Indeed, this was very simple for three blocks.