Academic Publications: arXiv
Academic Publications · arXiv. Preprints in AI, computer science and related fields. This source has 14 entries admitted under this category's review policy; each links directly to the original.
Authority tiers in this category: A2–R.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Pretrains bidirectional text representations with masked language modeling, then fine-tunes them for tasks such as classification and question answering; an important original work for understanding encoder pretraining.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §3 method, §4 experiments, §5 ablations; version checked: v2.
Limitations: The paper's ablation on next-sentence prediction is limited to its training setup; it does not mean every later encoder must use that objective. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Pre-training · Self-supervised Learning · Embedding
Link last checked: 2026-09-23
Language Models are Few-Shot Learners
Studies in-context learning without weight updates through zero-shot, one-shot and few-shot experiments with GPT-3 at different scales.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §2 method, §3 results, §4 contamination, §5 limitations; version checked: v4.
Limitations: Some reasoning tasks still show clear weaknesses; results must be read with the contamination analysis, and parameter count is no guarantee of task capability. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Large Language Model (LLM) · In-context Learning (ICL)
Link last checked: 2026-09-23
Training Compute-Optimal Large Language Models
Re-estimates how to split a fixed compute budget between model size and training tokens, and validates the prediction with Chinchilla.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §3 three estimation approaches, §4 Chinchilla, §5 discussion; version checked: v1.
Limitations: Compute-optimal training is not the same as optimal total deployment cost including inference; the ratio depends on data and training conditions. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Scaling Laws · Pre-training
Link last checked: 2026-09-23
Deep Residual Learning for Image Recognition
Learns incremental mappings through residual branches and shortcut connections, giving a method and image recognition experiments that address training degradation in deep networks.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §3 residual learning, §4 ImageNet/CIFAR experiments; version checked: v1.
Limitations: Training degradation and overfitting are different problems; residual connections do not guarantee better results at any depth or on any task. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Residual Connection · Convolutional Neural Network (CNN)
Link last checked: 2026-09-23
Adam: A Method for Stochastic Optimization
Builds the Adam optimizer from moving estimates of the first and second moments of the gradient with bias correction, with the algorithm and experiments.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §2 algorithm, §3 bias correction, §6 experiments; version checked: v9.
Limitations: The paper does not prove that arbitrary non-convex training converges or beats SGD; hyperparameters still need validation. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Gradient Descent · Optimizers and Learning Rate Schedules
Link last checked: 2026-09-23
Generative Adversarial Networks
Sets up a generator and a discriminator for adversarial training, with the objective, an idealized analysis and initial generation experiments.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §3 adversarial nets, §4 theoretical conditions, §5–6 experiments and drawbacks; version checked: v1.
Limitations: The theoretical results rely on idealized conditions and do not guarantee stable training or the absence of mode collapse in practice. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Generative Adversarial Network (GAN) · Image Generation
Link last checked: 2026-09-23
Auto-Encoding Variational Bayes
Estimates the variational lower bound through reparameterization and trains an approximate posterior network, forming the core learning method of the variational autoencoder.
Evidence use: For checking the authors' method, evaluation design and scoped experimental results. Where to look: §2 method, §3 VAE, §5 experiments; version checked: v11.
Limitations: Approximate posteriors and latent-variable assumptions affect results; optimizing a lower bound is not exact posterior inference. This entry records the original paper link and an editorial summary; experiments were not reproduced, and arXiv listing does not imply peer review or state-of-the-art status.
Related concepts: Variational Autoencoder (VAE)
Link last checked: 2026-09-23
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
Recasts visual autoregressive generation from pixel-by-pixel or token-by-token prediction into coarse-to-fine "next-scale prediction", and shows its generation quality, efficiency and scaling behavior.
Evidence use: For understanding visual autoregressive generation, next-scale prediction and the related experiments. Where to look: method, ImageNet experiments and scaling analysis; version checked: v2.
Limitations: The evidence centers on the image generation settings tested in the paper; it does not show that VAR beats diffusion models on every task. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Image Generation
Link last checked: 2026-09-23
Guiding a Diffusion Model with a Bad Version of Itself
Evidence grade: for discovery only; must not by itself support a formal conclusion.
Proposes Autoguidance: replacing the unconditional model in classifier-free guidance with a weaker version of the same diffusion model, to improve the trade-off between generation quality and diversity.
Evidence use: For understanding guidance in diffusion models, Autoguidance and the quality–diversity trade-off. Where to look: method, ImageNet experiments and discussion; version checked: v3.
Limitations: The findings are mainly supported by the image diffusion models tested and cannot be extrapolated to every modality or diffusion model. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Diffusion Models · Image Generation
Link last checked: 2026-09-23
1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Studies scaling network depth in self-supervised contrastive RL, showing that deeper networks substantially improve goal-reaching and change the behaviors agents learn.
Evidence use: For understanding depth scaling in self-supervised contrastive RL and goal-conditioned learning. Where to look: §4.2, §4.5, §5 Limitations; version checked: v4.
Limitations: Scaling depth increases compute; results depend on contrastive RL and the environments tested and do not generalize to all RL algorithms. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Reinforcement Learning · Self-supervised Learning
Link last checked: 2026-09-23
Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training
Studies, through training dynamics, the two timescales on which diffusion models move from generalization to memorization, connecting generation quality, data size and overfitting through experiments and theory.
Evidence use: For understanding generalization, memorization and implicit dynamical regularization in diffusion model training. Where to look: §2, §4 Limitations and future works; version checked: v2.
Limitations: The theory uses a simplified model and the main experiments are unconditional diffusion; the title does not mean diffusion models never memorize training data. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Diffusion Models
Link last checked: 2026-09-23
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Systematically compares gated variants of softmax attention and finds that placing a per-head sigmoid gate after SDPA improves training stability, attention sinks and long-context extrapolation.
Evidence use: For understanding attention gating, attention sinks and long-context extrapolation. Where to look: §3.2, §4.3, Limitations; version checked: v1.
Limitations: The paper does not fully explain the long-context mechanism; the experimental gains are not guaranteed in every model configuration. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Attention · Context Window
Link last checked: 2026-09-23
The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
Shows that arbitrary-order generation in diffusion language models can bypass key reasoning forks and shrink the solution space, and proposes a more direct RL training scheme on that basis.
Evidence use: For understanding decoding order, reasoning limits and RL training in diffusion language models. Where to look: §3.2, §4, §7; version checked: v4.
Limitations: The findings are limited to the diffusion language models and math and code tasks tested; they do not show that arbitrary orderings are ineffective on all tasks. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Large Language Model (LLM) · Reinforcement Learning
Link last checked: 2026-09-23
High-accuracy sampling for diffusion models and log-concave distributions
Gives high-accuracy sampling algorithms for diffusion models and log-concave distributions, improving the dependence on target error to a polylogarithmic number of steps under the stated assumptions.
Evidence use: For understanding the theoretical complexity of diffusion sampling, dependence on high-accuracy error and the FORS method. Where to look: §3.1 FORS, §6; version checked: v2.
Limitations: This is a theoretical result that depends on assumptions such as score estimation; the paper leaves implementation and experiments to future work, so it should not be presented as a verified end-to-end engineering speedup. The paper and the official award evidence were checked, but the experiments were not independently reproduced.
Related concepts: Diffusion Models
Link last checked: 2026-09-23