Jan 16 Computational modeling

• Computational modeling → formalizing a theory
• symbolic
  • John Anderson's Adaptive Control of Thought (ACT)
• emergent phenomena
  • parallel constraint satisfaction & hopfield (energy minimization)
• connectionism → philosophical commitments
  • associationism: everything is activation-passing
  • Hard-core distributed representations: the only representation of everything is distributed
  • cognition is non-symbolic: no need for variable or variable binding
  • Hard-core Empiricism: everything is learned; nothing is innate (except learning algo)
• McCulloch-Pitts Net (1943)
  • To simulate parallel processing on a serial computer
    • Update neuron input
    • Update neuron activation
  • Threshold: $n_i \ge \theta_i$
  • Inhibition will shut the activation down

Feb 6 Hopfield network

https://julien-vitay.net/lecturenotes-neurocomputing/4-neurocomputing/2-Hopfield.html

• Auto-association
  • Every unit has a connection to and from every other unit
• Activation fn: $n_i^t = \sum_j w_{ij} a_j^{t-1}, a_i^t=[n_i^t > \theta_i]$
  • Note that it’s strict greater
  • Compare to McCulloch-Pitts Net: Inhibition only being summed up
  • Usually $\theta_i=0$
• Learning rule: $\Delta w_{ij}^p = (2 a_i^p -1)(2 a_j^p -1)$
  • Start with weight=0
  • Increase the weight if the activations are the same, otherwise decrease the weight
  • Kind of Habbian learning, but those not fire together also fire together
  • Training timestep=1 ⇒ Input pattern order invariant: $w_{ij} = \sum_{\text{pattern }p} (2 a_i^p -1)(2 a_j^p -1)$
• Inference
  • Activate a partial pattern
  • Run it a few timesteps without chaning weights
  • Run until stablized
  • Expecting it to fill in the rest of the pattern
• Analog
  • Simulates content-addressable memory (just like animal brain)
  • Signal cleanup / pattern completion
• Energy
  • $E=-\frac{1}{2} \sum_{i \ne j} w_{ij} a_i a_j$
    • $w_{ij} a_i a_j$: the goodness of a local match
  • During inference, the energy will never go up
  • Energy well: the region surrounding a local minimum of potential energy
  • Since every training pattern actually gives 2 patterns, therefore creating 2 energy wells (and the implicit well could be greater and deeper than the explicit well)
• Symmetry breaking
  • During inference, update only one unit at a time

Mar 5

• Linear Perceptron
  • 1-layer linear perceptron: hyper halfplane
  • 2-layer linear perceptron: convex
  • 3-layer linear perceptron: any function
• Hebbian learning
  • Simplest form: $\Delta w_{ij} = a_i a_j$