Hardware meets AI!

Topics often include:

• faster ways to do things
• faster ways to implement things
• combinations of the above

list of papers

• 2/10 - Geoff Hinton
  • [x] https://www.cs.toronto.edu/~hinton/FFA13.pdf - forward forward
• 2/17 - Karthik
  • [ ] https://arxiv.org/abs/1912.01703 - PyTorch
  • [x] https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf - tensorflow
  • [ ] PipeDream: Generalized Pipeline Parallelism for DNN Training - PipeDream
• 2/24 - Prof. Deming Chen (UIUC)
  • FPGA
    • [ ] ScaleHLS: A New Scalable High-Level Synthesis Framework on Multi-Level Intermediate Representation
    • [ ] Hyperscale FPGA-as-a-Service Architecture for Large-Scale Distributed Graph Neural Network
  • Serverless
    • [ ] Cloud Programming Simplified: A Berkeley View on Serverless Computing (read this first)
    • [x] ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning
    • [ ] Serverless Computing on Heterogeneous Computers
    • [ ] Cirrus: a Serverless Framework for End-to-end ML Workflows
• assignment 1
  • Pruning
    • [x] [1] Lee, N., Ajanthan, T. and Torr, P.H., 2018. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340.
    • [x] [2] Wang, C., Zhang, G. and Grosse, R., 2020. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376.
    • [x] [3] Tanaka, H., Kunin, D., Yamins, D.L. and Ganguli, S., 2020. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05467.
    • [x] [4] Frankle, J., Dziugaite, G.K., Roy, D.M. and Carbin, M., 2020. Pruning Neural Networks at Initialization: Why are We Missing the Mark?. arXiv preprint arXiv:2009.08576.
  • [x] Bloom Filter: [1] Dai, Z. and Shrivastava, A., 2019. Adaptive learned Bloom filter (Ada-BF): Efficient utilization of the classifier. arXiv preprint arXiv:1910.09131.
• 3/3 - Jonathan Frankle (Mosaic)
  • [x] https://arxiv.org/pdf/1803.03635.pdf (Lottery Ticket)
  • [x] https://www.mosaicml.com/blog/methodology, https://www.mosaicml.com/blog/mosaic-resnet?src=hn, https://www.mosaicml.com/blog/mosaic-resnet-deep-dive (Mosaic blogs)
  • [ ] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  • [x] https://ieeexplore.ieee.org/abstract/document/9355301 (ZeRO)
  • [ ] https://arxiv.org/pdf/2211.00683.pdf (model recycling)
• 3/10 - Dr. Safeen Huda (Google Brain)
  • [ ] https://arxiv.org/pdf/1704.04760.pdf - TPU performance analysis
  • [ ] https://dl.acm.org/doi/10.1145/3360307 - TPU
  • [ ] https://www.computer.org/csdl/magazine/mi/5555/01/09351692/1r50VAsNljq - TPU design process
  • [ ] ⭐️ https://arxiv.org/pdf/2105.04663.pdf - parallelization bible
  • [ ] https://dl.acm.org/doi/pdf/10.1145/3470496.3533040 - Mozart reuse processor
  • [ ] https://dl.acm.org/doi/pdf/10.1145/2749469.2750380 - general purpose processors
  • [ ] https://dl.acm.org/doi/pdf/10.1145/3079856.3080255 - stream pipeline
• 3/24 - Danqi
  • [x] BERT, RoBERTa, GPT
  • [ ] https://arxiv.org/pdf/2012.15723.pdf - tools for LLM fine tuning
  • [ ] https://arxiv.org/pdf/1911.00172.pdf - Retrieval LMs
  • [ ] https://arxiv.org/pdf/2112.04426.pdf - RETRO
  • [ ] https://arxiv.org/pdf/2208.03299.pdf - Atlas
  • [ ] ⭐️ https://arxiv.org/pdf/2303.12712.pdf - GPT4 Microsoft comprehensive
• 3/31 - Entropy-Learned Hashing & Learnable Caching
  • [ ] https://dl.acm.org/doi/pdf/10.1145/3514221.3517894 - entropy-learned hashing
  • [ ] https://arxiv.org/pdf/1712.01208.pdf - learned index structures (google)
  • [ ] https://www.vldb.org/pvldb/vol16/p532-sabek.pdf - learnable hash functions
  • [ ] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9773195&tag=1 - learned caching via Belady’s algorithm
  • [ ] https://dl.acm.org/doi/abs/10.1145/3007787.3001146 - Belady’s algorithm
• 4/7 - Mixture of Experts & DLRM Scaling
  • [x] https://jmlr.org/papers/volume23/21-0998/21-0998.pdf - switch transformer
  • [x] https://arxiv.org/pdf/2206.03382.pdf - TUTEL adaptive MoE
  • [x] https://arxiv.org/pdf/2211.15841.pdf - MegaBlocks MoE
  • [ ] https://dl.acm.org/doi/pdf/10.1145/3470496.3533727 - DLRM training
  • [ ] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9065589 - DLRM phenomena
  • [ ] https://dl.acm.org/doi/pdf/10.1145/3326937.3341255 - XDL, DL framework for sparse data
  • [ ] https://dl.acm.org/doi/pdf/10.1145/3357384.3358045 - CTR prediction model
• 4/14 - TinyML and DataMUX
  • [x] https://arxiv.org/pdf/2206.15472.pdf - TinyML
  • [ ] https://ofa.mit.edu/ - train one network for many deployment scenarios
  • [ ] https://arxiv.org/pdf/2202.09318.pdf - DataMUX
  • [ ] https://arxiv.org/pdf/2302.12441.pdf - MUX pretrained LMs
• 4/21 - Snorkel / weakly-supervised learning
  • [ ] https://link.springer.com/article/10.1007/s00778-019-00552-1 - blog
  • [ ] https://dl.acm.org/doi/abs/10.1145/3299869.3314036 - snorkel drybell
  • [ ] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6879381/ - snuba
  • [ ] https://arxiv.org/abs/2208.14362 - weakly-supervised learning benchmark
• 4/28 - Federated Learning
  • [ ] https://research.googleblog.com/2017/04/federated-learning-collaborative.html - federated learning overview
  • [ ] https://www.nature.com/articles/s41467-022-33407-5 - example
  • [ ] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10025466 - gradient inversion attacks
  • [ ] https://arxiv.org/pdf/1607.00133.pdf - DL with differential privacy