TD Methods

In my quest in discovering if temporal difference methods can be used in games, I discovered the article Learning to Play Games Using Temporal Difference Methods (Wiering, Patist, Mannen 2005).  The authors used TD methods and Neural Network for function approximation to evaluate Backgammon, Chess, and Draughts.  In addition, the paper demonstrated three methods of function valuation from either self play, learning from expert, and database of human master games.  In the end, the authors concluded that learning from an expert or self-play the agent was able reach its maximum evaluation function in the neural net compared to the agent that learned from observing games stored in a database.  In a way this article was a demonstration of Transfer Learning.  Finally, the authors were aware of other function approximation methodologies such as support vector machines and gradient descent.

Progress Report – December 2009

During this month I focused on three items – completing my blocks world planning agent with environment, transfer learning, and reinforcement learning.  First, as part on my ongoing understanding towards RRL, finally I completed the blocks world planning agent using SWI-Prolog v5.8.1.  The test_environment clause is currently set to move three blocks.  The planner agent uses a depth-first search to find the correct plan.  It takes nine steps to complete the operation.  My next step is to study [Sutton and Barto 1998].  The combination between Relational Learning (RL) or Inductive Logic Programming (ILP) and Reinforcement Learning (RL) was suggested by Kaebling and Sutton in 1997, which lead to the Relational Reinforcement Learning (RRL).

Transfer Learning

For some time in my research, I came across a hot topic called Transfer Learning.  The idea is not new, but has been a key question in Artificial Intelligence.  The main question behind transfer learning is that an agent can learn a policy or solution in one domain and transfer this knowledge into another domain as background knowledge.  Humans have the uncanny ability to learn solutions in one domain and transfer them to new domains.  Other important issues are the measurement of the agent’s performance with this background knowledge.  One potential problem to address is the issue of negative transfer in which the agent’s performance declines as a result of the new background knowledge.