Several comments: 1. image/video SSL methods differ by whether they use reconstruction or not. Joint...
Several comments:
1. image/video SSL methods differ by whether they use reconstruction or not. Joint embedding architectures Amy or may not have a predictor (JEA vs JEPA). They also differ by their anti-collapse method and data corruption/augmentation methods.
2. DINO *is* a kind of JEPA (technically a JEA). The difference with other JEPA models is the anti-collapse criterion (based on contrative clustering+ stopgradient/EMA), and the data augmentation scheme (image transformations, which is task specific),
3. V-JEPA 2.1, like DINO, is from Meta-FAIR (corporate lab), involved significant engineering teams, large/proprietary datasets, considerable computing resources, lots of fine-tuning, and number-pushing on many benchmarks.
4. Many JEPA papers are applications and/or fine-tuned versions of DINO, V-JEPA or I-JEPA.
5. LeJEPA, LeWM, and most other papers with "JEPA" in the title are *research* papers focused on the methodology, often done by students in academic labs, involving no engineering support, very little computing resources, and small/public training sets. They are generally tested on toy problems, simulated environments, or small/public benchmarks.
6. You simply cannot compare 5 with 2/3/4. 5 gives you ideas and new methods, not models. 2/3 give you well-engineered models, with lots of empirically validated tricks. Only rarely will 2/3 give you a new, well-principled method with room to spare. 5 give you empirical results on new applications.
7. What is published in 5 are methods that will eventually become like 2 and 3, whenever someone with resources picks it up and scales it.
DO NOT CONFUSE UPSTREAM FOUNDATIONAL RESEARCH ON METHODS AND DOWNSTREAM SCALING -> ENGINEERING -> NUMBER PUSHING -> MODELS -> PRODUCTS.
The former makes the rest possible, but it takes time.
1. image/video SSL methods differ by whether they use reconstruction or not. Joint embedding architectures Amy or may not have a predictor (JEA vs JEPA). They also differ by their anti-collapse method and data corruption/augmentation methods.
2. DINO *is* a kind of JEPA (technically a JEA). The difference with other JEPA models is the anti-collapse criterion (based on contrative clustering+ stopgradient/EMA), and the data augmentation scheme (image transformations, which is task specific),
3. V-JEPA 2.1, like DINO, is from Meta-FAIR (corporate lab), involved significant engineering teams, large/proprietary datasets, considerable computing resources, lots of fine-tuning, and number-pushing on many benchmarks.
4. Many JEPA papers are applications and/or fine-tuned versions of DINO, V-JEPA or I-JEPA.
5. LeJEPA, LeWM, and most other papers with "JEPA" in the title are *research* papers focused on the methodology, often done by students in academic labs, involving no engineering support, very little computing resources, and small/public training sets. They are generally tested on toy problems, simulated environments, or small/public benchmarks.
6. You simply cannot compare 5 with 2/3/4. 5 gives you ideas and new methods, not models. 2/3 give you well-engineered models, with lots of empirically validated tricks. Only rarely will 2/3 give you a new, well-principled method with room to spare. 5 give you empirical results on new applications.
7. What is published in 5 are methods that will eventually become like 2 and 3, whenever someone with resources picks it up and scales it.
DO NOT CONFUSE UPSTREAM FOUNDATIONAL RESEARCH ON METHODS AND DOWNSTREAM SCALING -> ENGINEERING -> NUMBER PUSHING -> MODELS -> PRODUCTS.
The former makes the rest possible, but it takes time.
💬6🔄12❤️114👀8462📊24