The month began by reviewing my college Probability and Statistics book (Devore 1991).  I needed to review standard probability concepts mentioned in articles and the Stanford machine learning class.  For example, the article Learning with a Mixture of Trees (Meila and Jordan 2000) the authors explain the new algorithm by using Joint Probabilities and probabilistic mathematics to describe the algorithm – a similar used by Dr. Ng in his machine learning lectures.

Another focus this month was the use of the predicate calculus to wumpus world in Part II titled logic based reasoning of the AIMA book.  Russell and Norvig (1995) demonstrated the power of predicate calculus (a.k.a. first order logic) over propositional calculus.  Using the wumpus world as the example, the authors illustrated the use of predicate calculus and its application to the situation calculus.  As a result I searched for articles related to the wumpus world in research.  I downloaded and read various articles using the wumpus world as the test bed.

Over 20 years ago one of my goals to develop an AI based chess playing program in Prolog.  However, what I did not realize that computer chess was making great strides with successful computer programs.  For example, Knight Cap was a successful computer program using an new technique called TD(leaf), which was tested against players in FICS and ICS chess servers with some outstanding results.  Then I decided to explore Othello since I was influenced by the code in Principles of Artificial Intelligence Programming (PAIP – Norvig 1991).  However, after reading The Evolution of Strong Othello Programs, I concluded that avenue has been explored.  Another possible area of exploration is Computer Go for which I have not played this game before.

I read some interesting articles on using the Wumpus World and other dynamic multiagent games.  The first was Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching.  This article focused on a self improving reactive agent with multiple adversarial agents on with the goal of getting 15 food objects in 8×8 grid.  Although the author of the work stated he could get all food objects, after many trial runs, the agent could obtain 13 food objects.

Next interesting article was Comparison of Techniques to Learning Agent Strategies in Adversarial Games.  In this article, the authors used a learning agent against adversary agent (the wumpus) in an 8×8 Wumpus World grid.  The authors explored four machine learning techniques  (e.g., Bayesian, CF4.5) and compared the results of  each technique for the agent.

From the ICML 1998 conference, I read the article Multiagent Reinforcement Learning: A Theoretical Framework and an Algorithm.  In this article, the authors discuss that in multiagent games, the agents achieve the optimal strategy by converging to a Nash equilibrium in a competitive Markov Decision Process.  Also, one agent has to observe other agents reward and track its Q values.  Then the authors developed the frame with many definitions, lemmas, assumptions, and theorems as well as illustrated an algorithm, which was not tested.

With the use of the citeseerx bibliography service, I was able to download additional conference papers, technical reports, and thesis citing the above article.  Also, I found Rich Sutton’s web pages on reinforcement learning as well as subscribed to the RL group and the machine learning mailing lists.

Another interesting article was Enhancing Computer Science Education with a Wireless Intelligent Simulation Environment (WISE).  Although the authors discussed this article from an educational point of view, I was interested because the authors discussed in the AI section the use of the WISE testbed for students to learn and develop the following types of agents:

• Simple Reflex Agent
• Reflex Agent with State
• Search-Based Agent
• Planning Agent
• Decision-Theoretic Agent
• Markov Decision Process (MDP) Agent
• Learning Agent
• Natural Language Agent
• Vision Processing Agent
• Multiagent Cooperation

Most published articles only explore a limited scope to a project.  Typically, there is discussion about potential future work, if it continues and ever gets funded.  As a non academic enthusiast I can explore this future work by determining if other researchers explored it and, if not, then it is potential opportunity of exploration.

Items of interest:

• Read article Learning with a Mixture of Trees, JMLR, 2000.
• Reviewing Probability and Statistics for Engineering and the Sciences by Devore.
• Read article Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching.
• Reviewed Machine Learning Lecture videos 12 to 14 from Dr. Andrew Ng.
• Read article Learning to Solve Game Trees.
• Reviewed PhD program for the Computer Science Department at Oregon State University.
• AIMA book – Read Chapters 7 and 8
• Downloaded and read article The Evolution of Strong Othello Programs.
• Downloaded articles using the wumpus world as test platform.
• Read article DFA Learning of Opponent Strategies.
• Read article Comparison of Techniques to Learning Agent Strategies in Adverisal Games.
• Read article Enhancing Computer Science Education with a Wireless Intelligent Simualtion Environment (WISE).
• Downloaded the WISE software from UTA.
• Downloaded and installed the latest JDK 1.6.11.
• Compiling and correcting the WISE software.
• Downloaded and read article Multiagent Reinforcement Learning: A Theoretical Framework and an Algorithm.
• Update the about page.
• Create new MARL blog website (under construction).

References:

Devore, J.L. (1991). Probability and Statistics for Engineering and the Sciences. 3rd ed. Pacific Grove, California: Brooks/Cole.

Stearn, D.; Herbrich, R.; Graepel, T. (2007).  Learning to Solve Game Trees.  In the Proceedings of International Conference in Machine Learining (ICML 2007).

Russell, S. and Norvig, P. (1995).  Artificial Intelligence: A Modern Approach.  Englewood Cliffs, NJ:  Prentice Hall.