Neural Networks

• The Unreasonable Effectiveness of Recurrent Neural Networks
  
• https://colah.github.io/posts/2015-08-Understanding-LSTMs/
  
• http://karpathy.github.io/2015/05/21/rnn-effectiveness/
  
• https://ahmetozlu93.medium.com/long-short-term-memory-lstm-networks-in-a-nutshell-363cd470ccac
  
• Stanford CS 224N | Natural Language Processing with Deep Learning
  

• Has Generative AI Already Peaked? - Computerphile - YouTube
  

  We consistently find across our experiments that, across concepts, the frequency of a concept in the pretraining dataset is a strong predictor of the model’s performance on test examples containing that concept (see Fig. 2).
  Notably, model performance scales linearly as the concept frequency in pretraining data grows exponentially i.e., we observe a consistent log-linear scaling trend.
  We find that this log-linear trend is robust to controlling for correlated factors (similar samples in pretraining and test data [81]) and testing across different concept distributions along with samples generated entirely synthetically [52].
  

• AI can’t cross this line and we don’t know why. - YouTube
  

https://playground.tensorflow.org

OpenWorm is an international open science project to simulate the roundworm Caenorhabditis elegans at the cellular level as a simulation
Although the long-term goal is to model all 959 cells of the C. elegans, the first stage is to model the worm’s locomotion by simulating the 302 neurons and 95 muscle cells.
This bottom up simulation is being pursued by the OpenWorm community.
As of this writing, a physics engine called Sibernetic has been built for the project and models of the neural connectome and a muscle cell have been created in NeuroML format.
A 3D model of the worm anatomy can be accessed through the web via the OpenWorm browser.
The OpenWorm project is also contributing to develop Geppetto, a web-based multi-algorithm, multi-scale simulation platform engineered to support the simulation of the whole organism.

https://towardsdatascience.com/dropout-in-neural-networks-47a162d621d9

• How does Batch Size impact your model learning | by Devansh | Geek Culture | Medium
  
• Don’t Decay the Learning Rate, Increase the Batch Size - 1711.00489v2.pdf
  Decaying the learning rate is simulated annealing
  
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour - 1706.02677v2.pdf
  Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.
  
• Why Small Batch sizes lead to greater generalization in Deep Learning | by Devansh | Geek Culture | Medium
  
• [1609.04836] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
  

• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.
  
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.
  
• The Illustrated GPT-2 (Visualizing Transformer Language Models) – Jay Alammar – Visualizing machine learning one concept at a time.
  
  • The Illustrated GPT-2: Visualizing Transformer Language Models (2019) | Hacker News
    
  • The Illustrated Transformer is fantastic, but I would suggest that those going i… | Hacker News
    
  • LLM Visualization
    
• The Annotated Transformer
  
  • The Annotated Transformer
    
• Categories | Ketan Doshi Blog
  
• Transformers from scratch | peterbloem.nl
  multi-head attention: each head captures a “type” of relation
  mary,gave,roses,to,susan → who gave the roses? and who received them? need 2 heads of attention
  
• poloclub.github.io/transformer-explainer/
  
  • poloclub/transformer-explainer: Transformer Explained: Learn How LLM Transformer Models Work with Interactive Visualization
    
• Transformers from Scratch
  
• Transformers Explained Visually - Overview of Functionality | Ketan Doshi Blog
  
• Transformers Explained Visually - How it works, step-by-step | Ketan Doshi Blog
  
• Transformers Explained Visually - Multi-head Attention, deep dive | Ketan Doshi Blog
  
• Transformers Explained Visually - Not just how, but Why they work so well | Ketan Doshi Blog
  
• Transformer Models 101: Getting Started — Part 1 | by Nandini Bansal | Feb, 2023 | Towards Data Science
  
• https://projector.tensorflow.org/
  
• openai/transformer-debugger
  
  • OpenAI – transformer debugger release | Hacker News
    

How might LLMs store facts | Chapter 7, Deep Learning - YouTube Superposition Hypothesis

• The Johnson-Lindenstrauss bound for embedding with random projections — scikit-learn 1.6.0 documentation
  
• Johnson–Lindenstrauss lemma - Wikipedia
  You can encode \(\sim \exp(\epsilon N)\) vectors that are approximately perpendicular within an error of \(\epsilon\) in a $N$-dimensional space
  

• Toy Models of Superposition
  

  Compressed sensing. In general, if one projects a vector into a lower-dimensional space, one can’t reconstruct the original vector. However, this changes if one knows that the original vector is sparse. In this case, it is often possible to recover the original vector.
  

• [2411.00660v2] Physics in Next-token Prediction
  
• [2410.02724] Large Language Models as Markov Chains
  

https://multiversecomputing.com/
Model Compression: https://paperswithcode.com/task/model-compression
https://github.com/HuangOwen/Awesome-LLM-Compression
Utilizan mejor compresión de modelos: https://multiversecomputing.com/papers/compactifai-extreme-compression-of-large-language-models-using-quantum-inspired-tensor-networks
https://arxiv.org/abs/2401.14109
El mecanismo es Tensor Network: https://en.wikipedia.org/wiki/Tensor_network

https://medium.com/@researchgraph/compactifai-large-language-models-dont-actually-have-to-be-large-857548d053d2

They show that LLMs do not have to get larger to get better. Instead, they can perform just as well with just a fraction of the parameters. Moreover, the article also presented a novel compression technique that, unlike previous methods, is controllable and explainable.

También pueden editar (Knowledge Editing) y explicar la seguridad de la red (quitarle un concepto)

https://www.nature.com/articles/s42256-020-00237-3

https://youtu.be/CfAL_cL3SGQ
They are consecutive linear and non-linear transformations

https://link.medium.com/97kwPdKMSkb

1. Artificial NN are too simple (It takes 1000 artificial neurons to simulate 1 biological neuron at the single spike timescale)
   
2. We do not know how they work
   
3. We could compensate for the lack of complexity in artificial neurons with larger models, tons of computing power, and gigantic datasets, but that’s too inefficient to be the eventual last step of this quest.
   
4. ANNs should be more neuroscience-based for two reasons,
   
   1. (future) the difference in complexity between biological and artificial neurons will result in differences in outcome — AGI won’t come without a reform
      
   2. (present) the inefficiency with which we’re pursuing this goal is damaging our society and the planet
      

Lecture Notes: Neural Network Architectures

Old book: Neural Network Design
https://hagan.okstate.edu/NNDesign.pdf
https://github.com/estamos/Neural-Network-Design-Solutions-Manual
https://pypi.org/project/nndesigndemos/

• [2307.09793] On the Origin of LLMs: An Evolutionary Tree and Graph for 15,821 Large Language Models
  
• Constellation
  
• 
  
  • 37C3 - Self-cannibalizing AI
    
• llm evolutionary tree - Google Search
  
• Mooler0410/LLMsPracticalGuide: A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)
  https://www.reddit.com/r/MachineLearning/comments/13wkcn3/d_llm_evolutionare_tree_from_the_practical_guides/
  
• LLM Evolutionary Tree. LLM Proliferation. – blog.biocomm.ai
  

https://youtu.be/1L83tM8nwHU
Softer decision boundaries by mixing up data

https://youtu.be/9JpdAg6uMXs