Skip to content

Bag of Tricks in Machine Learning

Public Personal Notebook

  • Home
  • random
  • paper
  • list
  • nlp
  • reinforcementlearning
  • gan
  • cv

A Hitchhiker’s Guide On Distributed Training of Deep Neural Networks

January 14, 2019 by admin

https://arxiv.org/abs/1810.11787

Categories random Tags distributedtraining
Post navigation
Understanding binary cross-entropy (Daniel Godoy)
Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Best practice to follow this website

  1. In Feedly, click “add content”
  2. Input the url “bagoftricks.ml” and follow

ShujianFollow

Shujian
Retweet on TwitterShujian Retweeted
luyu_gaoLuyu Gao@luyu_gao·
22 Sep

Excited to share our EMNLP 2021 paper, Condenser: a Pre-training Architecture for Dense Retrieval. Condenser unsupervisedly learns information aggregation into dense representation through language model pre-training. https://arxiv.org/pdf/2104.08253.pdf with @JamieCallan

Reply on Twitter 1440705155385606145Retweet on Twitter 144070515538560614510Like on Twitter 144070515538560614568Twitter 1440705155385606145
Retweet on TwitterShujian Retweeted
stanfordnlpStanford NLP Group@stanfordnlp·
19 Sep

The result that the multimodal models much more effectively use vision for language than language for vision is intriguing (and somewhat surprising). https://twitter.com/ebugliarello/status/1437456748240121862

Emanuele Bugliarello@ebugliarello

Is your Vision-and-Language model really a Vision-AND-Language model? 👀

“Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers”

📄 https://arxiv.org/abs/2109.04448

🗣️ #EMNLP2021

Reply on Twitter 1439676722417319937Retweet on Twitter 143967672241731993736Like on Twitter 1439676722417319937178Twitter 1439676722417319937
Retweet on TwitterShujian Retweeted
karpathyAndrej Karpathy@karpathy·
27 Aug

Badly tuned LR decay schedules are an excellent way to silently shoot yourself in the foot. Models can often look like they are converging but it's just LR getting too low too fast. FixedLR (+optional warmup) with 1 manual decay of 10X on plateau is a safe strong baseline.

Reply on Twitter 1431380525759885313Retweet on Twitter 1431380525759885313191Like on Twitter 14313805257598853132114Twitter 1431380525759885313
Retweet on TwitterShujian Retweeted
sirbayesKevin Patrick Murphy@sirbayes·
6 Sep

For those that think that all we need to "solve AGI" is to "train bigger models" (eg GPT4?), I recommend this paper on causal inference in / for NLP as an antidote to such naive thinking: https://arxiv.org/abs/2109.00725

Reply on Twitter 1435025390326583298Retweet on Twitter 143502539032658329884Like on Twitter 1435025390326583298594Twitter 1435025390326583298
Retweet on TwitterShujian Retweeted
jacobeisensteinJacob Eisenstein@jacobeisenstein·
4 Sep

Our NLP+causality survey paper is finally on arxiv and I'm looking forward to a relaxing three day weekend! Cause and effect? Read the paper and you might find out. But if you *do* find out, is it because you read the paper? Read the paper and you might https://twitter.com/roireichart/status/1433884845235650567

Roi Reichart@roireichart

Since Jan. 2021 we, an interdisciplinary team of researchers interested in causal inference (CI) and text, have been working on a paper that provides a unified view on the intersection of CI and NLP.

I am happy to share it is now published on arxiv:

https://arxiv.org/abs/2109.00725

Reply on Twitter 1433985701003993088Retweet on Twitter 143398570100399308811Like on Twitter 1433985701003993088101Twitter 1433985701003993088
Load More...

Categories

  • ml (2)
  • nlp (12)
  • papers (1)
  • random (8)
  • reinforcementlearning (1)
  • Uncategorized (57)

Tags

anomaly (1) automl (1) ctr (1) cv (2) data (1) distributedtraining (1) gan (1) kaggle (1) list (2) ml (2) NER (1) nlp (10) nn (1) paper (1) random (3) reinforcementlearning (1) sql (1)

Recent Posts

  • Version mismatch in embedding-based retrieval
  • NER with small strongly labeled and large weakly labeled data
  • A Survey of Data Augmentation Approaches for NLP
  • Data Augmentation in NLP
  • PyTorch nn.DistributedDataParallel

Archives

  • July 2021 (2)
  • June 2021 (1)
  • May 2020 (1)
  • October 2019 (1)
  • August 2019 (11)
  • July 2019 (8)
  • June 2019 (6)
  • May 2019 (1)
  • April 2019 (11)
  • March 2019 (4)
  • February 2019 (2)
  • January 2019 (32)

Recent Comments

    January 2019
    M T W T F S S
     123456
    78910111213
    14151617181920
    21222324252627
    28293031  
        Feb »

    • 0
    • 1
    • 82,545
    • 40,384
    • 80
    • 0

    Recent Posts

    • Version mismatch in embedding-based retrieval
    • NER with small strongly labeled and large weakly labeled data
    • A Survey of Data Augmentation Approaches for NLP
    • Data Augmentation in NLP
    • PyTorch nn.DistributedDataParallel

    Categories

    • ml (2)
    • nlp (12)
    • papers (1)
    • random (8)
    • reinforcementlearning (1)
    • Uncategorized (57)

    Meta

    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org
    © 2021 Bag of Tricks in Machine Learning • Built with GeneratePress