Cholinergic Feedback Circuit Regulates Striatal Uncertainty and Reinforcement Learning

Addressing Peer Review of a Computational Model

This article addresses and clarifies the points raised during the peer review process of a research paper focusing on a computational model of the basal ganglia. The model explores how a cholinergic feedback circuit, specifically involving Tonic-Active Neurons (TANs), regulates uncertainty within the striatum and optimizes reinforcement learning. This response to reviewers details the improvements and clarifications made to the original manuscript to enhance its rigor, clarity, and impact.

Refining Model Explanation and Visualizations

Reviewers initially found the explanation of the model and its relation to previous work unclear. Figures were deemed incomplete, and explanations assumed excessive prior familiarity with the authors’ previous publications. To address this, significant revisions were made to improve clarity and accessibility.

Several new figures were incorporated to better illustrate the model’s assumptions, behavior, and connectivity. These include:

Schematic of TAN Signaling Effects on MSNs: A new figure (Figure 1) visually describes how TAN signaling impacts Medium Spiny Neurons (MSNs), the primary output neurons of the striatum. This schematic clarifies the fundamental interactions within the model.
Examples of Simulated Activity: To address the request for more detailed activity patterns, multiple examples of simulated activity for both MSNs and TANs were added (Figure 2). These examples provide concrete illustrations of neuronal behavior within the model under different conditions.
Synaptic Weight Changes: Figure 3 now depicts synaptic weight changes over time, illustrating their dependence on TAN pause duration and regulation by the proposed feedback mechanism. This visualization helps to understand the learning dynamics within the network.
TAN Behavior Changes: Further enhancing Figure 3, changes in TAN behavior as a consequence of the feedback mechanism are also visualized. This addition clarifies the adaptive role of the feedback loop in modulating TAN activity.

These visual enhancements, coupled with more detailed textual explanations, aim to make the model and its dynamics more readily understandable to a broader audience, even those not deeply familiar with the authors’ prior work.

Clarifying Relationship to Experimental Data

A crucial point raised by reviewers was the need to better define the relationship between the model and experimental findings. The revised manuscript now explicitly clarifies the assumptions and predictions of the model in the context of existing experimental data.

This includes:

Mechanistic Account of Behavioral Findings: The paper now provides a clearer mechanistic account of how the model aligns with and explains previous behavioral findings in reinforcement learning and basal ganglia function.
Discussion of Feedback Mechanism in Relation to Experiments: The proposed feedback mechanism is explicitly discussed in relation to relevant experimental findings, strengthening the model’s grounding in biological evidence.

By more clearly articulating the connections between the computational model and empirical data, the revised manuscript enhances its credibility and relevance to experimental neuroscience.

Defining Uncertainty and Entropy

The reviewers pointed out that the use of terms “entropy” and “uncertainty” was unclear and required better definition and motivation within the manuscript. In response, the authors have expanded their discussion of these concepts.

Key improvements include:

Concrete Definition in Model Terms: Uncertainty and entropy are now more concretely defined within the specific context of the computational model. This includes clarifying how the model represents and quantifies uncertainty in action selection.
Relating Entropy to MSN Population Activity: The manuscript now details how entropy relates to the activity distribution across the MSN population, providing a neural correlate for this abstract concept. A visual comparison between low and high entropy populations has been added to Figure 2 to further illustrate this point.

By providing clearer definitions and linking these concepts to observable aspects of the model, the revised manuscript addresses a significant point of confusion and strengthens the theoretical framework.

Implementing a Biologically Plausible Feedback Mechanism

One of the most significant advancements in the revised manuscript is the inclusion of new simulations demonstrating a biologically plausible mechanism for the feedback loop. The initial model assumed TANs had access to an analytical computation of entropy, which was critiqued as lacking biological realism.

To address this, the authors have:

Proposed a Local Mechanism: A novel mechanism is proposed where pairs of MSNs synapse closely on TAN dendrites, enabling the detection of coincident MSN activity. This coincident activity, representing alternative actions, is argued to approximate entropy.
Simulated AND Detection: This mechanism is implemented in the neural network using AND detection to modulate TAN pause duration. Simulations demonstrate that this biologically plausible mechanism yields similar adaptive behavior to the earlier model relying on analytical entropy computation (Figure 8).
Highlighting Plausibility and Alternatives: While emphasizing this as one plausible mechanism, the authors acknowledge other potential mechanisms and highlight that the core prediction remains: TAN pauses modulate striatal uncertainty and learning rate.

This addition of a biologically grounded feedback mechanism significantly strengthens the model’s plausibility and addresses a major concern raised by reviewers.

Addressing Specific Reviewer Concerns

The revised manuscript also includes detailed point-by-point responses to each reviewer’s comments, demonstrating a thorough engagement with the feedback. Key concerns and responses include:

TAN Input to MSN Population: Reviewer 1 requested clarification on how TAN input affects MSNs. Figures 1 and 2, along with more detailed text, now explicitly illustrate this interaction, including simulated activity patterns.
Entropy Computation Mechanism: Reviewer 1 strongly urged for a mechanistic implementation of entropy computation. The new simulations and Figure 8 directly address this by proposing and simulating a local, biologically plausible mechanism.
Definition of Uncertainty: Reviewer 2 raised concerns about the definition of uncertainty. The manuscript now carefully defines uncertainty in the context of action selection and clarifies its relation to MSN population activity and Shannon’s entropy.
Network Determinism and Entropy: Reviewer 2 questioned the presence of entropy in a seemingly deterministic system. The response clarifies that entropy arises from the distribution of activity across the MSN population, reflecting uncertainty in action selection, even in a deterministic network. Furthermore, the role of stochastic noise in the model is now explicitly discussed.
TAN Acronym and Clarity: Reviewer 2 criticized the use of the acronym TAN without definition and general clarity issues. The manuscript now explicitly defines TAN and has undergone extensive revisions for improved clarity and readability throughout.
Physiological Data Validation: Reviewer 2 questioned the validation of the model with physiological data. The response highlights substantial physiological evidence supporting the model’s assumptions and predictions, including MSN coding of action values and TAN pauses windowing dopamine signals.
Rebound Phase of Pause Response: Reviewer 3 raised a major concern about the model ignoring the rebound phase of the TAN pause response. The response addresses this by discussing how the rebound burst could further enhance dopamine signaling and plasticity, and by including new simulations exploring the effects of a post-pause rebound phase, detailed in a new section “Simulations of Post-Pause TAN Burst”.
MSN to ChI Feedback Loop Strength: Reviewer 3 questioned the strength of the MSN to Cholinergic Interneuron (ChI) feedback loop. The response clarifies that modulation may occur during the TAN pause when TANs are more sensitive to inhibition and acknowledges other potential mechanisms for uncertainty signaling to TANs.
Go/NoGo Unit Representation: Reviewer 3 questioned the use of Go/NoGo units and Shannon entropy in the context of action choices. The response clarifies the function of NoGo units and justifies the use of Shannon entropy over Go units as a measure of uncertainty in action selection.
Pause Duration Units: Reviewer 3 requested pause durations in milliseconds instead of arbitrary units. Figure legends now include pause durations in milliseconds for improved interpretability.

Conclusion: A Significantly Improved Model

The revisions made to the manuscript, as detailed in this response to reviewers, represent a significant improvement to the computational model. By enhancing clarity, providing more detailed visualizations, strengthening the link to experimental data, and implementing a biologically plausible feedback mechanism, the authors have addressed the major concerns raised during peer review. The resulting manuscript presents a more robust, accessible, and impactful model of cholinergic regulation of striatal uncertainty and its role in optimizing reinforcement learning. This refined model offers valuable insights into the neural mechanisms underlying adaptive behavior and decision-making.