Week 4

The topic of this week is spatial navigation in various animals, including rats, a few types of arthropods, and corvids.

First of all, I like the perspective (in Leising & Blaisdell, 2009) that you could actually view spatial navigation as a series of reinforcement learning (a “stimulus-response chain”). I have never really thought about animal navigation this way before, but this idea links to the Q learning model I talked about in Week 2, and the Q learning model actually seems to be able to explain well a few experiments mentioned in Leising and Blaisdell’s paper (2009). Here’s the formula again:

s: the state the agent is in
a: an available action the agent can take
r: the amount/value of reward
alpha: how fast the agent learns
gamma: how much the potential future rewards should be discounted; it can be 0 so the future states are not concerned
Q: the perceived value associated with this particular state (s-t) and action (a-t)

Now animal navigation seems to be very similar to a Roomba robot navigating a room while cleaning the floor, or a self-driving car following the roads to get to a destination. A simplified version of navigation, the grid world, can be described by this formula. Here, a state is the location of the agent (agent = the rat, the Roomba or the car); actions are the directions the agent could possibly take in a state; rewards are the consequences that may happen, such as getting the food for rats, or dirty floor getting cleaned for a Roomba, or getting to the destination without accidents for a car (and an accident would mean a negative reward). Some of the experiments in Leising and Blaisdell’s paper (2009) fit well to the idea of a Q learning model. In the kerplunk experiment, for example, after being trained with the same maze extensively, rats strictly followed their previous route after the maze was slightly altered, while ignoring the new barrier or food in the altered maze. The Q learning formula would simulate this habitual navigation behavior too, since even after the maze was altered, the rat/agent would still think they were in the same states as they were trained before, and choose the exactly same action/direction as before. One of the basic trade-offs in reinforcement learning problems, exploration vs exploitation, was also shown in other rat studies (Calhoun, 1962; Morris, 1981) as well as ant studies (Wehner & Srinivasan, 1981; Wehner & Wehner, 1990). The rats and ants demonstrated two types of navigation behaviors in the field/water maze/desert, one is to follow the most efficient path (i.e., exploitation of previously rewarded actions), the other is to follow longer and more random paths (i.e., exploration of new territories). In reinforcement learning problems, researchers would typically need to specify an ideal ratio of exploration vs exploitation in their robots, but maybe rats also have something similar specified in their genes?

Researchers also found other learning phenomena in spatial navigation, including blocking and overshadowing, where spatial cues or landmarks serve as the stimuli. But of course, real-life spatial navigations are more complicated than what reinforcement learning (RL) models can describe. Rats showed both response learning (which can be nicely modeled by Q learning) and place learning/goal-directed strategy in studies; real-world robots also have more advanced techniques when navigating the space. Simultaneous localization and mapping (SLAM) algorithms deals with real-world spatial navigation (such as in Roomba or self-driving cars) by constructing maps given sensor input, and it works somewhat similar to hippocampus cells, which constructs cognitive maps given sensory input. I think being able to process sensor/sensory input makes a major improvement from the simplified RL models/response learning, for both rats and robots.

The vector sum model explains goal-directed navigation in more details, arguing that animals navigate based on landmarks, and weight the landmarks differently by, for example, their proximity to the goal. The differential landmark weighting may also explain the learning phenomena such as blocking and overshadowing (redundant landmarks are weighted less heavily, salient landmarks are weighted more heavily). This reminds me of the A* search algorithm, where agents use heuristics to guide their graph search. Here the goal is the find the path to a specified node in a graph, and in the context of spatial navigation, nodes in the graph can be seen as landmarks. The agent basically preprocess the graph and weight every landmark/node differently (i.e., the heuristics), and when navigating, always go to the most heavily weighted node they could access as the next step. This parallels the vector sum model exactly and works well in many real-world applications.

Chapter 19 also discussed in more details about how arthropods build “cognitive maps” based on environmental or internal cues, such as the sky (to tell directions), the optic flow (for bees to estimate distances), step counting (for ants and crabs to estimate distances), body orientation and visual cues (for crabs to tell direction and distance), skyline, landmarks/image-matching, etc. The optic flow study with bees (Srinivasan et al., 1997) seems particularly well designed, ruling out many other cues that bees could rely on, including energy expenditure, time of flying, and number of stripes passes. That said, I’m not sure if I understand correctly what optic flow means — is it like when you are on a train, outside objects of different distances seem to move at different speeds (closer objects are faster), so they create different optic flows?

Chapter 19 also talked about global and local vectors for ants’ localization, which are very interesting and raise a lot of further questions. Global vectors sound like an estimate (probably based on step counting and direction estimates) of the current position relative to the start point; local vectors seem to be another estimate based on more recent experiences such as landmarks just passed. It is impressive that they showed the ants correcting their routes based on global vectors, after using local vectors incorrectly based on a changed landmark (channel direction). It seems like behaviors based on local vectors can be better described with RL models (landmark stimulus -> response), whereas behaviors based on global vectors may be better characterized by cognitive maps. There are also a lot more questions to ask about this: how are the two types of vectors integrated together to guide the ants’ route? When there is conflict information, how do ants (or humans/other animals) decide which type of vector take over the other? How do they “realize” they have gone wrong (or do they?) based on one type of vector and subsequently refer to the other vector?

Chapter 21 discussed the food-caching behaviors in corvids, which is less related to spatial navigation, but more related to spatial memory and social cognition. Across species and seasons, animals cache different amounts of food, and some bird are able to cache thousands (!) of food items and later find them. The social aspect of caching is particularly interesting. Corvids need avoid their food being pilfered by other birds, while potentially trying to pilfer other birds’ food without being found. This almost certainly requires theory of mind to some extent, particularly given evidence such as that they protected the food when a stranger was watching but not when their partner was watching (Dally et al., 2006), and that only birds with pilfering experiences developed better counter-pilfering strategies (Clayton, 2001). I wonder if there are more studies on birds’ social cognition…

Leave a Reply

Create your own website, free on WordPress.com.Create your own website.
Start free