weijielyu commited on
Commit
918c308
·
1 Parent(s): ac3e055
Files changed (6) hide show
  1. VIEWER_DEBUG.md +0 -94
  2. VIEWER_DEBUG_GUIDE.md +0 -209
  3. VIEWER_IFRAME_SOLUTION.md +0 -208
  4. VIEWER_INTEGRATION.md +0 -149
  5. app.py +10 -5
  6. main.js +0 -1484
VIEWER_DEBUG.md DELETED
@@ -1,94 +0,0 @@
1
- # Viewer Debugging Guide
2
-
3
- The viewer now has extensive console logging to help diagnose issues.
4
-
5
- ## How to Debug
6
-
7
- 1. **Run the app:**
8
- ```bash
9
- python app.py
10
- ```
11
-
12
- 2. **Open the browser console:**
13
- - Chrome/Edge: Press `F12` or `Ctrl+Shift+J` (Windows) / `Cmd+Option+J` (Mac)
14
- - Firefox: Press `F12` or `Ctrl+Shift+K` (Windows) / `Cmd+Option+K` (Mac)
15
- - Safari: Enable Developer menu in Preferences, then press `Cmd+Option+C`
16
-
17
- 3. **Check the Console tab** and look for these key messages:
18
-
19
- ## Expected Console Messages
20
-
21
- ### On Page Load:
22
- ```
23
- DOM already loaded, initializing viewer in 100ms
24
- initViewer called, isInitialized: false
25
- Container found: <div...>
26
- Canvas created and added
27
- WebGL2 context created
28
- Worker created
29
- Starting render loop
30
- ✓ Viewer initialized successfully
31
- Rendering black screen, waiting for data... (repeats every 60 frames)
32
- ```
33
-
34
- ### After Generating a Model:
35
- ```
36
- === Starting PLY load ===
37
- URL: /gradio_api/file=outputs/splats/xxxxx/xxxxx.ply?v=xxxxx
38
- Initialized? true
39
- Worker? true
40
- Fetching: /gradio_api/file=...
41
- Response status: 200
42
- ArrayBuffer size: XXXXX bytes
43
- Magic bytes: Uint8Array(4) [112, 108, 121, 10] (this is correct for PLY files)
44
- PLY file sent to worker
45
-
46
- [Worker] Message received: {ply: ArrayBuffer}
47
- [Worker] Processing PLY buffer...
48
- [Worker] Buffer processed, vertex count: XXXXX
49
- [Worker] Generating texture...
50
- [Worker] Texture generated and sent
51
-
52
- Worker message received: {texdata: Uint32Array, ...}
53
- Texture data: {texwidth: 2048, texheight: XXX, vertexCount: XXXXX}
54
- ✓ Successfully loaded XXXXX gaussians
55
-
56
- First render with XXXXX vertices
57
- ```
58
-
59
- ## Common Issues and Solutions
60
-
61
- ### Issue 1: "Container #splat-container not found"
62
- **Problem:** The viewer container isn't in the DOM yet
63
- **Solution:** This should auto-retry, but if it persists, check that the Gradio interface loaded correctly
64
-
65
- ### Issue 2: "Failed to load: 404"
66
- **Problem:** The PLY file URL is incorrect or the file doesn't exist
67
- **Solution:** Check that the generation completed successfully and the file was saved
68
-
69
- ### Issue 3: Worker doesn't receive message
70
- **Problem:** The fetch failed or the ArrayBuffer is empty
71
- **Solution:** Check the ArrayBuffer size in the console - it should be > 0
72
-
73
- ### Issue 4: Worker error during processing
74
- **Problem:** The PLY file format might be incompatible
75
- **Solution:** Check the magic bytes - should be [112, 108, 121, 10] which is "ply\n"
76
-
77
- ### Issue 5: Black screen but no errors
78
- **Problem:** vertexCount might be 0 or rendering issue
79
- **Solution:** Check that "Successfully loaded X gaussians" appears with X > 0
80
-
81
- ## Quick Test
82
-
83
- Run this in the browser console after the page loads:
84
-
85
- ```javascript
86
- console.log('Testing viewer state:');
87
- console.log('Window function:', typeof window.__load_splat__); // should be "function"
88
- console.log('Container exists:', !!document.getElementById('splat-container')); // should be true
89
- ```
90
-
91
- ## Next Steps
92
-
93
- Copy the console output and we can diagnose the exact issue!
94
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
VIEWER_DEBUG_GUIDE.md DELETED
@@ -1,209 +0,0 @@
1
- # Viewer Debugging Guide
2
-
3
- ## How to Check Where the PLY File Is
4
-
5
- ### 1. Check the Debug Info Panel
6
-
7
- After generating a 3D model, look at the **"🔍 Debug Info"** textbox which will show:
8
-
9
- ```
10
- PLY Path: /path/to/outputs/splats/xxxxx/gaussians.ply
11
- PLY URL: /file=/path/to/outputs/splats/xxxxx/gaussians.ply
12
- File exists: True
13
- ```
14
-
15
- This tells you:
16
- - **PLY Path**: The actual file system path where the PLY was saved
17
- - **PLY URL**: The URL the viewer is trying to fetch
18
- - **File exists**: Whether the file actually exists on disk
19
-
20
- ### 2. Check Browser Console
21
-
22
- Open your browser's developer console (F12 or Cmd+Option+I) and look for:
23
-
24
- ```
25
- === PLY Loading Debug ===
26
- PLY URL: /file=/path/to/outputs/splats/xxxxx/gaussians.ply
27
- Current location: http://localhost:7860/
28
- Worker exists: true/false
29
- Fetch response status: 200 OK (or error)
30
- Response headers: [...]
31
- ✓ PLY loaded successfully!
32
- Buffer size: 245678 bytes
33
- ```
34
-
35
- ### 3. Check Loading Indicator
36
-
37
- The loading message in the viewer will update to show:
38
- - `"Loading 3D model..."` → Initial state
39
- - `"Fetching PLY from: /file=..."` → Fetch started
40
- - `"Processing 245.6 KB..."` → File loaded, processing
41
- - `"Error: [message]"` (in red) → Something went wrong
42
-
43
- ## Common Issues and Fixes
44
-
45
- ### Issue 1: File Not Found (404 Error)
46
-
47
- **Symptoms:**
48
- - Debug Info shows: `File exists: True`
49
- - Console shows: `HTTP 404: Not Found`
50
-
51
- **Problem:** Gradio's file serving path is incorrect
52
-
53
- **Fix:** Update the PLY URL format in `_generate_and_filter_outputs()`:
54
- ```python
55
- # Try different URL formats:
56
- ply_url = f"/file={ply_path}" # Current
57
- # OR
58
- ply_url = ply_path # Direct path
59
- # OR
60
- ply_url = f"/file/{Path(ply_path).name}" # Just filename
61
- ```
62
-
63
- ### Issue 2: Worker Not Initialized
64
-
65
- **Symptoms:**
66
- - Console shows: `Worker exists: false`
67
- - Loading shows: `"Error: Viewer worker not ready"`
68
-
69
- **Problem:** viewer.js didn't initialize properly
70
-
71
- **Possible causes:**
72
- 1. **viewer.js file is empty or corrupt**
73
- - Check: `ls -lh viewer.js` should show ~51KB
74
- - Fix: Re-download from GitHub
75
-
76
- 2. **viewer.js has syntax errors**
77
- - Check browser console for JavaScript errors
78
- - Fix: Verify file integrity
79
-
80
- 3. **Gradio HTML component limitations**
81
- - The embedded script might be too large
82
- - Fix: Serve viewer.js as a static file instead of embedding
83
-
84
- ### Issue 3: Stuck on "Loading 3D model..."
85
-
86
- **Symptoms:**
87
- - No console logs appear
88
- - Loading message never changes
89
-
90
- **Problem:** JavaScript not executing
91
-
92
- **Check:**
93
- 1. Is the viewer HTML actually being rendered?
94
- - Inspect the `out_viewer` component in browser DevTools
95
- - Should see `<canvas id="canvas">`, scripts, etc.
96
-
97
- 2. Are there Content Security Policy (CSP) errors?
98
- - Check console for CSP violations
99
- - Gradio might be blocking inline scripts
100
-
101
- ### Issue 4: CORS Error
102
-
103
- **Symptoms:**
104
- - Console shows: `CORS policy: No 'Access-Control-Allow-Origin'`
105
-
106
- **Problem:** Cross-origin resource sharing blocked
107
-
108
- **Fix:** Ensure PLY file is served from the same origin as the app
109
-
110
- ## Quick Diagnosis Checklist
111
-
112
- Run through these checks in order:
113
-
114
- - [ ] **Debug Info shows `File exists: True`**
115
- - ✓ File was created successfully
116
- - ✗ Check `generate_3d_head()` function
117
-
118
- - [ ] **Console shows `Worker exists: true`**
119
- - ✓ viewer.js loaded and initialized
120
- - ✗ Check viewer.js file and embedding
121
-
122
- - [ ] **Console shows fetch starting**
123
- - ✓ Script is executing
124
- - ✗ Check for JavaScript errors earlier in console
125
-
126
- - [ ] **Console shows `200 OK` response**
127
- - ✓ File is accessible
128
- - ✗ Check PLY URL format and Gradio file serving
129
-
130
- - [ ] **Console shows buffer size**
131
- - ✓ File downloaded successfully
132
- - ✗ Check fetch error message
133
-
134
- - [ ] **Loading indicator disappears**
135
- - ✓ Everything works!
136
- - ✗ Check worker message passing
137
-
138
- ## Alternative Approach: Static File Serving
139
-
140
- If embedding doesn't work, serve files statically:
141
-
142
- ```python
143
- # In app.py, after imports:
144
- import gradio as gr
145
-
146
- # Setup static file serving
147
- gr.set_static_paths(paths=["./outputs"])
148
-
149
- # Then in create_splat_viewer_html():
150
- def create_splat_viewer_html(ply_url: str) -> str:
151
- # Use external script tag instead of embedding
152
- return f"""
153
- <div id="splat-viewer-container" style="...">
154
- <canvas id="canvas"></canvas>
155
- ...
156
- </div>
157
- <script src="/file/viewer.js"></script>
158
- <script>
159
- // Load PLY
160
- fetch('{ply_url}')...
161
- </script>
162
- """
163
- ```
164
-
165
- ## Testing Locally
166
-
167
- To test the viewer independently:
168
-
169
- ```bash
170
- # 1. Generate a PLY file
171
- python app.py
172
- # Upload image, click generate
173
-
174
- # 2. Find the PLY path from Debug Info
175
- # Example: outputs/splats/xxxxx/gaussians.ply
176
-
177
- # 3. Test direct file access
178
- curl http://localhost:7860/file=outputs/splats/xxxxx/gaussians.ply
179
- # Should download the PLY file
180
-
181
- # 4. Test viewer.js loading
182
- curl http://localhost:7860/file/viewer.js
183
- # Should return JavaScript code
184
- ```
185
-
186
- ## Getting Help
187
-
188
- When reporting issues, provide:
189
- 1. **Debug Info** textbox contents
190
- 2. **Browser console** logs (full "=== PLY Loading Debug ===" section)
191
- 3. **Browser** and version
192
- 4. **File size** from `ls -lh outputs/splats/xxxxx/gaussians.ply`
193
- 5. Any **error messages** shown in red
194
-
195
- ## Next Steps
196
-
197
- After identifying the issue:
198
- 1. Fix the PLY URL format if it's a 404
199
- 2. Re-download viewer.js if it's corrupt
200
- 3. Switch to static file serving if embedding fails
201
- 4. Check Gradio documentation for HTML component limitations
202
-
203
- ---
204
-
205
- **Current Status**: Debugging mode enabled ✓
206
- **Debug Info**: Visible in UI
207
- **Console Logging**: Verbose
208
- **Error Display**: In viewer and console
209
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
VIEWER_IFRAME_SOLUTION.md DELETED
@@ -1,208 +0,0 @@
1
- # Viewer iFrame Solution for Hugging Face Spaces
2
-
3
- ## Problem
4
-
5
- The initial approach of embedding 51KB of viewer.js JavaScript directly in the Gradio HTML component didn't work because:
6
- 1. **Size limit**: Gradio's HTML component couldn't handle such large inline content
7
- 2. **CSP restrictions**: Hugging Face Spaces has Content Security Policy that may block large inline scripts
8
- 3. **Result**: Elements weren't rendering (`document.getElementById('canvas')` returned `null`)
9
-
10
- ## Solution
11
-
12
- Use an **iframe with srcdoc** attribute containing a complete HTML document that loads `viewer.js` as an external script.
13
-
14
- ### Key Changes
15
-
16
- #### 1. Serve viewer.js as Static File (lines 40-44)
17
-
18
- ```python
19
- # Copy viewer.js to outputs so it can be served as a static file
20
- VIEWER_JS_SRC = Path(__file__).parent / "viewer.js"
21
- if VIEWER_JS_SRC.exists():
22
- shutil.copy2(VIEWER_JS_SRC, OUTPUTS_DIR / "viewer.js")
23
- print(f"✓ Copied viewer.js to {OUTPUTS_DIR / 'viewer.js'}")
24
- ```
25
-
26
- This makes `viewer.js` accessible at `/file=outputs/viewer.js`
27
-
28
- #### 2. Create iFrame-Based Viewer (lines 335-468)
29
-
30
- ```python
31
- def create_splat_viewer_html(ply_url: str, viewer_id: str) -> str:
32
- """Create HTML with splat viewer that loads a PLY file via external script."""
33
- return f"""
34
- <iframe id="viewer-frame-{viewer_id}" style="..." srcdoc='
35
- <!DOCTYPE html>
36
- <html>
37
- <head>...</head>
38
- <body>
39
- <canvas id="canvas"></canvas>
40
- <div id="spinner">...</div>
41
- <div id="message"></div>
42
- <div id="fps"></div>
43
- <div id="camid"></div>
44
- <div id="controls-info">...</div>
45
-
46
- <script src="{ply_url.rsplit("/", 3)[0]}/viewer.js"></script>
47
- <script>
48
- // Auto-load PLY after viewer initializes
49
- ...
50
- </script>
51
- </body>
52
- </html>
53
- '></iframe>
54
- """
55
- ```
56
-
57
- ### How It Works
58
-
59
- 1. **iFrame Isolation**: Creates a sandboxed environment with its own DOM
60
- 2. **External Script**: Loads `viewer.js` via `<script src="...">` instead of embedding
61
- 3. **Automatic Loading**: Polls for `window.worker` then fetches and loads PLY
62
- 4. **Error Handling**: Shows clear messages if loading fails
63
-
64
- ### Advantages
65
-
66
- | Aspect | Previous (Embedded) | Current (iFrame) |
67
- |--------|---------------------|------------------|
68
- | **Size** | 51KB inline fails | References external file ✓ |
69
- | **DOM** | Shared with Gradio | Isolated in iframe ✓ |
70
- | **CSP** | May violate policy | Clean external script ✓ |
71
- | **Debugging** | Hard to inspect | Full iframe console ✓ |
72
- | **Reliability** | Failed to render | Renders successfully ✓ |
73
-
74
- ### File Structure
75
-
76
- ```
77
- FaceLift/
78
- ├── viewer.js # Source (51KB)
79
- ├── outputs/
80
- │ ├── viewer.js # Copied for serving
81
- │ └── splats/
82
- │ └── xxxxx/
83
- │ └── gaussians.ply # Generated PLY files
84
- └── app.py
85
- ```
86
-
87
- ### URL Pattern
88
-
89
- - **viewer.js**: `/file=outputs/viewer.js`
90
- - **PLY file**: `/file=outputs/splats/xxxxx/gaussians.ply`
91
-
92
- The script src dynamically extracts the base path from the PLY URL:
93
- ```javascript
94
- <script src="{ply_url.rsplit('/', 3)[0]}/viewer.js"></script>
95
- ```
96
-
97
- Example:
98
- - PLY URL: `/file=outputs/splats/xxxxx/gaussians.ply`
99
- - Base: `/file=outputs`
100
- - Viewer: `/file=outputs/viewer.js`
101
-
102
- ### Testing
103
-
104
- After deployment, check in browser console:
105
-
106
- ```javascript
107
- // Inside the iframe
108
- const iframe = document.querySelector('iframe[id^="viewer-frame"]');
109
- const iframeDoc = iframe.contentDocument;
110
-
111
- console.log('Canvas:', !!iframeDoc.getElementById('canvas')); // should be true
112
- console.log('Worker:', !!iframe.contentWindow.worker); // should be true
113
- ```
114
-
115
- ### Debug Info
116
-
117
- The UI now shows:
118
- ```
119
- PLY Path: /path/to/outputs/splats/xxxxx/gaussians.ply
120
- PLY URL: /file=/path/to/outputs/splats/xxxxx/gaussians.ply
121
- File exists: True
122
- Viewer ID: a1b2c3d4
123
- ```
124
-
125
- ### Console Output
126
-
127
- When working correctly:
128
- ```
129
- === Splat Viewer Init ===
130
- PLY URL: /file=outputs/splats/xxxxx/gaussians.ply
131
- ✓ Worker ready after 300 ms
132
- ✓ PLY loaded: 245678 bytes
133
- ✓ Sent to worker
134
- ```
135
-
136
- ### Common Issues
137
-
138
- #### Issue: "Viewer failed to initialize"
139
-
140
- **Cause**: viewer.js not accessible or failed to load
141
-
142
- **Check**:
143
- ```bash
144
- # Verify file was copied
145
- ls -lh outputs/viewer.js
146
-
147
- # Test direct access (when app is running)
148
- curl http://localhost:7860/file=outputs/viewer.js
149
- ```
150
-
151
- **Fix**: Ensure `VIEWER_JS_SRC` path is correct and file is copied
152
-
153
- #### Issue: "Error: HTTP 404"
154
-
155
- **Cause**: PLY file path incorrect or file doesn't exist
156
-
157
- **Check Debug Info**: Look at "File exists" field
158
-
159
- **Fix**: Verify `generate_3d_head()` creates the PLY file
160
-
161
- #### Issue: iframe shows blank/black
162
-
163
- **Cause**: CSP blocking iframe or script loading
164
-
165
- **Check**: Browser console for CSP errors
166
-
167
- **Fix**: May need to adjust Hugging Face Space settings (rare)
168
-
169
- ### Deployment Checklist
170
-
171
- - [x] `viewer.js` exists in project root (51KB)
172
- - [x] File copying code runs at startup
173
- - [x] Debug info visible in UI
174
- - [x] iframe contains complete HTML document
175
- - [x] External script reference uses correct path
176
- - [x] Error messages display clearly
177
- - [ ] Test on Hugging Face Spaces
178
- - [ ] Verify PLY files load correctly
179
- - [ ] Check interactive controls work
180
- - [ ] Test on mobile devices
181
-
182
- ### Next Steps
183
-
184
- 1. **Deploy to HF Spaces**:
185
- ```bash
186
- git add app.py viewer.js
187
- git commit -m "Fix viewer with iframe approach"
188
- git push
189
- ```
190
-
191
- 2. **Test the viewer**:
192
- - Generate a 3D model
193
- - Check Debug Info shows correct paths
194
- - Verify iframe loads and displays model
195
- - Test drag/zoom/pan controls
196
-
197
- 3. **Optional: Hide Debug Info** (after confirming it works):
198
- ```python
199
- out_debug = gr.Textbox(label="🔍 Debug Info", lines=3, visible=False)
200
- ```
201
-
202
- ---
203
-
204
- **Status**: ✅ Ready for deployment
205
- **Approach**: iFrame with external script
206
- **Size**: Minimal (~2KB HTML wrapper)
207
- **Compatibility**: Should work on HF Spaces
208
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
VIEWER_INTEGRATION.md DELETED
@@ -1,149 +0,0 @@
1
- # antimatter15/splat Embedded Viewer Integration
2
-
3
- ## Summary
4
-
5
- Successfully integrated the [antimatter15/splat](https://github.com/antimatter15/splat) WebGL Gaussian Splatting viewer as an **embedded component** directly in the Gradio interface - no downloads required!
6
-
7
- ## Changes Made
8
-
9
- ### 1. Files Added
10
- - **`viewer.js`** (51KB) - The complete antimatter15/splat rendering engine downloaded from GitHub
11
-
12
- ### 2. Files Removed
13
- - `VIEWER_README.md` - No longer needed (not distributing packages)
14
- - `viewer_template.html` - No longer needed (generating HTML dynamically)
15
- - `INTEGRATION_SUMMARY.md` - Outdated approach documentation
16
-
17
- ### 3. Code Changes in `app.py`
18
-
19
- #### a. Removed Download/Packaging Functionality
20
- - Removed `zipfile` import (line 20)
21
- - Removed `create_viewer_html()` function that generated standalone files
22
- - Removed `package_viewer()` function that created ZIP packages
23
- - Removed static file serving setup for viewer.js
24
- - Simplified outputs directory setup
25
-
26
- #### b. Added Embedded Viewer Function (lines 337-391)
27
- ```python
28
- def create_splat_viewer_html(ply_url: str) -> str:
29
- """Create HTML with embedded splat viewer that loads a PLY file."""
30
- ```
31
-
32
- This function:
33
- - Reads the entire `viewer.js` content
34
- - Embeds it directly in the HTML
35
- - Creates a styled canvas container with controls overlay
36
- - Adds loading indicator
37
- - Auto-fetches and loads the PLY file
38
- - Returns complete self-contained HTML
39
-
40
- #### c. Updated Gradio UI (lines 414-428)
41
- - Changed `out_viewer` from `gr.File` to `gr.HTML` for embedded viewing
42
- - Reordered outputs: viewer at top, followed by reconstruction, video, PLY download
43
- - Simplified instructions to reflect instant viewing
44
- - Added credit to antimatter15/splat
45
-
46
- #### d. Updated Generation Wrapper (lines 415-424)
47
- - Modified to return HTML content instead of file paths for viewer
48
- - Creates Gradio-accessible PLY URL using `/file=` prefix
49
- - Returns: `viewer_html, output_path, turntable_path, ply_path`
50
-
51
- #### e. Simplified Button Click Handler (lines 470-475)
52
- - Removed `.then()` chain for packaging
53
- - Direct mapping: `inputs` → `_generate_and_filter_outputs` → `outputs`
54
-
55
- ### 4. Documentation Updates
56
-
57
- #### `README.md`
58
- - Updated feature: "Embedded real-time WebGL viewer" (was: "Downloadable WebGL viewer")
59
- - Updated usage: "Interact directly with the 3D model in your browser" (was: "Download the ZIP package...")
60
-
61
- ## How It Works
62
-
63
- 1. User generates a 3D face model
64
- 2. System creates PLY file and gets its URL
65
- 3. `create_splat_viewer_html()` is called with the PLY URL
66
- 4. Function:
67
- - Reads the entire viewer.js (51KB)
68
- - Embeds it in a self-contained HTML string
69
- - Includes canvas, controls, loading indicator
70
- - Adds auto-load script for the PLY file
71
- 5. HTML is returned to `gr.HTML` component
72
- 6. User sees interactive 3D viewer immediately in the interface!
73
-
74
- ## Technical Details
75
-
76
- ### Viewer Features (from antimatter15/splat)
77
- - **WebGL 1.0** rendering
78
- - **No external dependencies** - pure JavaScript
79
- - **Progressive rendering** - sorts splats for efficient display
80
- - **Full camera controls**:
81
- - Mouse: Drag to rotate, right-drag to pan, scroll to zoom
82
- - Keyboard: WASD for camera, arrows for movement
83
- - Touch: One/two finger gestures
84
-
85
- ### Performance
86
- - ~60 FPS on modern hardware
87
- - Async processing via Web Workers
88
- - Efficient Gaussian Splatting shader
89
- - Handles large PLY files (typical face models: 100K-300K splats)
90
-
91
- ### Compatibility
92
- - Works in all modern browsers (Chrome, Firefox, Safari, Edge)
93
- - Mobile friendly with touch controls
94
- - No GPU acceleration required (but recommended)
95
-
96
- ## Advantages Over Previous Approach
97
-
98
- | Previous (Downloadable Package) | Current (Embedded Viewer) |
99
- |--------------------------------|---------------------------|
100
- | User must download ZIP | Instant viewing |
101
- | User must extract files | No extraction needed |
102
- | User must open HTML file | Works in Gradio directly |
103
- | 3 separate files to manage | Single component |
104
- | Requires file system access | Browser-only |
105
- | Potential security warnings | Trusted Gradio environment |
106
-
107
- ## Testing Checklist
108
-
109
- - [x] viewer.js file exists (51KB)
110
- - [x] Code compiles without syntax errors
111
- - [ ] App starts successfully (needs gradio installed)
112
- - [ ] 3D model generates
113
- - [ ] Viewer shows loading indicator
114
- - [ ] PLY file loads into viewer
115
- - [ ] Interactive controls work (drag, zoom, pan)
116
- - [ ] Works on mobile devices
117
- - [ ] PLY download still available
118
-
119
- ## Credits
120
-
121
- Huge thanks to **Kevin Kwok (@antimatter15)** for creating the excellent [splat viewer](https://github.com/antimatter15/splat) (MIT License).
122
-
123
- ## Next Steps
124
-
125
- 1. **Test locally**:
126
- ```bash
127
- pip install -r requirements.txt
128
- python app.py
129
- ```
130
-
131
- 2. **Deploy to Hugging Face**:
132
- ```bash
133
- git add app.py README.md viewer.js
134
- git commit -m "Add embedded antimatter15/splat viewer"
135
- git push
136
- ```
137
-
138
- 3. **Verify on HF Spaces**:
139
- - Generate a test model
140
- - Check viewer loads and displays
141
- - Test all controls
142
- - Verify mobile compatibility
143
-
144
- ---
145
-
146
- **Status**: ✅ Code complete, ready for testing!
147
- **File Size**: app.py is ~480 lines (clean and maintainable)
148
- **User Experience**: Seamless - no downloads, instant viewing!
149
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
app.py CHANGED
@@ -332,7 +332,7 @@ class FaceLiftPipeline:
332
  raise gr.Error(f"Generation failed: {str(e)}")
333
 
334
  # Create viewer HTML that references external viewer.js
335
- def create_splat_viewer_html(ply_url: str, viewer_id: str) -> str:
336
  """Create HTML with splat viewer that loads a PLY file via external script."""
337
 
338
  return f"""
@@ -418,12 +418,14 @@ def create_splat_viewer_html(ply_url: str, viewer_id: str) -> str:
418
  <strong>Controls:</strong> Drag: Rotate | Scroll: Zoom | Right-drag: Pan
419
  </div>
420
 
421
- <script src="{ply_url.rsplit("/", 3)[0]}/viewer.js"></script>
422
  <script>
423
  // Auto-load PLY after viewer initializes
424
  const plyUrl = "{ply_url}";
 
425
  console.log("=== Splat Viewer Init ===");
426
  console.log("PLY URL:", plyUrl);
 
427
 
428
  let attempts = 0;
429
  const checkAndLoad = setInterval(function() {{
@@ -457,7 +459,7 @@ def create_splat_viewer_html(ply_url: str, viewer_id: str) -> str:
457
  clearInterval(checkAndLoad);
458
  document.getElementById("spinner").style.display = "none";
459
  const msg = document.getElementById("message");
460
- msg.textContent = "Viewer failed to initialize";
461
  msg.style.display = "block";
462
  }}
463
  }}, 100);
@@ -487,12 +489,15 @@ def main():
487
  # Create Gradio-accessible URL for the PLY file
488
  ply_url = f"/file={ply_path}"
489
 
 
 
 
490
  # Generate unique viewer ID
491
  viewer_id = str(uuid.uuid4())[:8]
492
- viewer_html = create_splat_viewer_html(ply_url, viewer_id)
493
 
494
  # Debug info showing the paths
495
- debug_info = f"PLY Path: {ply_path}\nPLY URL: {ply_url}\nFile exists: {Path(ply_path).exists()}\nViewer ID: {viewer_id}"
496
 
497
  return viewer_html, output_path, turntable_path, ply_path, debug_info
498
 
 
332
  raise gr.Error(f"Generation failed: {str(e)}")
333
 
334
  # Create viewer HTML that references external viewer.js
335
+ def create_splat_viewer_html(ply_url: str, viewer_js_url: str, viewer_id: str) -> str:
336
  """Create HTML with splat viewer that loads a PLY file via external script."""
337
 
338
  return f"""
 
418
  <strong>Controls:</strong> Drag: Rotate | Scroll: Zoom | Right-drag: Pan
419
  </div>
420
 
421
+ <script src="{viewer_js_url}"></script>
422
  <script>
423
  // Auto-load PLY after viewer initializes
424
  const plyUrl = "{ply_url}";
425
+ const viewerJsUrl = "{viewer_js_url}";
426
  console.log("=== Splat Viewer Init ===");
427
  console.log("PLY URL:", plyUrl);
428
+ console.log("Viewer JS URL:", viewerJsUrl);
429
 
430
  let attempts = 0;
431
  const checkAndLoad = setInterval(function() {{
 
459
  clearInterval(checkAndLoad);
460
  document.getElementById("spinner").style.display = "none";
461
  const msg = document.getElementById("message");
462
+ msg.textContent = "Viewer failed to initialize. Check console for viewer.js errors.";
463
  msg.style.display = "block";
464
  }}
465
  }}, 100);
 
489
  # Create Gradio-accessible URL for the PLY file
490
  ply_url = f"/file={ply_path}"
491
 
492
+ # viewer.js is in the outputs directory
493
+ viewer_js_url = f"/file={OUTPUTS_DIR}/viewer.js"
494
+
495
  # Generate unique viewer ID
496
  viewer_id = str(uuid.uuid4())[:8]
497
+ viewer_html = create_splat_viewer_html(ply_url, viewer_js_url, viewer_id)
498
 
499
  # Debug info showing the paths
500
+ debug_info = f"PLY Path: {ply_path}\nPLY URL: {ply_url}\nViewer JS URL: {viewer_js_url}\nFile exists: {Path(ply_path).exists()}\nViewer ID: {viewer_id}"
501
 
502
  return viewer_html, output_path, turntable_path, ply_path, debug_info
503
 
main.js DELETED
@@ -1,1484 +0,0 @@
1
- let cameras = [
2
- {
3
- id: 0,
4
- img_name: "00001",
5
- width: 1959,
6
- height: 1090,
7
- position: [
8
- -3.0089893469241797, -0.11086489695181866, -3.7527640949141428,
9
- ],
10
- rotation: [
11
- [0.876134201218856, 0.06925962026449776, 0.47706599800804744],
12
- [-0.04747421839895102, 0.9972110940209488, -0.057586739349882114],
13
- [-0.4797239414934443, 0.027805376500959853, 0.8769787916452908],
14
- ],
15
- fy: 1164.6601287484507,
16
- fx: 1159.5880733038064,
17
- },
18
- {
19
- id: 1,
20
- img_name: "00009",
21
- width: 1959,
22
- height: 1090,
23
- position: [
24
- -2.5199776022057296, -0.09704735754873686, -3.6247725540304545,
25
- ],
26
- rotation: [
27
- [0.9982731285632193, -0.011928707708098955, -0.05751927260507243],
28
- [0.0065061360949636325, 0.9955928229282383, -0.09355533724430458],
29
- [0.058381769258182864, 0.09301955098900708, 0.9939511719154457],
30
- ],
31
- fy: 1164.6601287484507,
32
- fx: 1159.5880733038064,
33
- },
34
- {
35
- id: 2,
36
- img_name: "00017",
37
- width: 1959,
38
- height: 1090,
39
- position: [
40
- -0.7737533667465242, -0.3364271945329695, -2.9358969417573753,
41
- ],
42
- rotation: [
43
- [0.9998813418672372, 0.013742375651625236, -0.0069605529394208224],
44
- [-0.014268370388586709, 0.996512943252834, -0.08220929105659476],
45
- [0.00580653013657589, 0.08229885200307129, 0.9965907801935302],
46
- ],
47
- fy: 1164.6601287484507,
48
- fx: 1159.5880733038064,
49
- },
50
- {
51
- id: 3,
52
- img_name: "00025",
53
- width: 1959,
54
- height: 1090,
55
- position: [
56
- 1.2198221749590001, -0.2196687861401182, -2.3183162007028453,
57
- ],
58
- rotation: [
59
- [0.9208648867765482, 0.0012010625395201253, 0.389880004297208],
60
- [-0.06298204172269357, 0.987319521752825, 0.14571693239364383],
61
- [-0.3847611242348369, -0.1587410451475895, 0.9092635249821667],
62
- ],
63
- fy: 1164.6601287484507,
64
- fx: 1159.5880733038064,
65
- },
66
- {
67
- id: 4,
68
- img_name: "00033",
69
- width: 1959,
70
- height: 1090,
71
- position: [
72
- 1.742387858893817, -0.13848225198886954, -2.0566370113193146,
73
- ],
74
- rotation: [
75
- [0.24669889292141334, -0.08370189346592856, -0.9654706879349405],
76
- [0.11343747891376445, 0.9919082664242816, -0.05700815184573074],
77
- [0.9624300466054861, -0.09545671285663988, 0.2541976029815521],
78
- ],
79
- fy: 1164.6601287484507,
80
- fx: 1159.5880733038064,
81
- },
82
- {
83
- id: 5,
84
- img_name: "00041",
85
- width: 1959,
86
- height: 1090,
87
- position: [
88
- 3.6567309419223935, -0.16470990600750707, -1.3458085590422042,
89
- ],
90
- rotation: [
91
- [0.2341293058324528, -0.02968330457755884, -0.9717522161434825],
92
- [0.10270823606832301, 0.99469554638321, -0.005638106875665722],
93
- [0.9667649592295676, -0.09848690996657204, 0.2359360976431732],
94
- ],
95
- fy: 1164.6601287484507,
96
- fx: 1159.5880733038064,
97
- },
98
- {
99
- id: 6,
100
- img_name: "00049",
101
- width: 1959,
102
- height: 1090,
103
- position: [
104
- 3.9013554243203497, -0.2597500978038105, -0.8106154188297828,
105
- ],
106
- rotation: [
107
- [0.6717235545638952, -0.015718162115524837, -0.7406351366386528],
108
- [0.055627354673906296, 0.9980224478387622, 0.029270992841185218],
109
- [0.7387104058127439, -0.060861588786650656, 0.6712695459756353],
110
- ],
111
- fy: 1164.6601287484507,
112
- fx: 1159.5880733038064,
113
- },
114
- {
115
- id: 7,
116
- img_name: "00057",
117
- width: 1959,
118
- height: 1090,
119
- position: [4.742994605467533, -0.05591660945412069, 0.9500365976084458],
120
- rotation: [
121
- [-0.17042655709210375, 0.01207080756938, -0.9852964448542146],
122
- [0.1165090336695526, 0.9931575292530063, -0.00798543433078162],
123
- [0.9784581921120181, -0.1161568667478904, -0.1706667764862097],
124
- ],
125
- fy: 1164.6601287484507,
126
- fx: 1159.5880733038064,
127
- },
128
- {
129
- id: 8,
130
- img_name: "00065",
131
- width: 1959,
132
- height: 1090,
133
- position: [4.34676307626522, 0.08168160516967145, 1.0876221470355405],
134
- rotation: [
135
- [-0.003575447631888379, -0.044792503246552894, -0.9989899137764799],
136
- [0.10770152645126597, 0.9931680875192705, -0.04491693593046672],
137
- [0.9941768441149182, -0.10775333677534978, 0.0012732004866391048],
138
- ],
139
- fy: 1164.6601287484507,
140
- fx: 1159.5880733038064,
141
- },
142
- {
143
- id: 9,
144
- img_name: "00073",
145
- width: 1959,
146
- height: 1090,
147
- position: [3.264984351114202, 0.078974937336732, 1.0117200284114904],
148
- rotation: [
149
- [-0.026919994628162257, -0.1565891128261527, -0.9872968974090509],
150
- [0.08444552208239385, 0.983768234577625, -0.1583319754069128],
151
- [0.9960643893290491, -0.0876350978794554, -0.013259786205163005],
152
- ],
153
- fy: 1164.6601287484507,
154
- fx: 1159.5880733038064,
155
- },
156
- ];
157
-
158
- let camera = cameras[0];
159
-
160
- function getProjectionMatrix(fx, fy, width, height) {
161
- const znear = 0.2;
162
- const zfar = 200;
163
- return [
164
- [(2 * fx) / width, 0, 0, 0],
165
- [0, -(2 * fy) / height, 0, 0],
166
- [0, 0, zfar / (zfar - znear), 1],
167
- [0, 0, -(zfar * znear) / (zfar - znear), 0],
168
- ].flat();
169
- }
170
-
171
- function getViewMatrix(camera) {
172
- const R = camera.rotation.flat();
173
- const t = camera.position;
174
- const camToWorld = [
175
- [R[0], R[1], R[2], 0],
176
- [R[3], R[4], R[5], 0],
177
- [R[6], R[7], R[8], 0],
178
- [
179
- -t[0] * R[0] - t[1] * R[3] - t[2] * R[6],
180
- -t[0] * R[1] - t[1] * R[4] - t[2] * R[7],
181
- -t[0] * R[2] - t[1] * R[5] - t[2] * R[8],
182
- 1,
183
- ],
184
- ].flat();
185
- return camToWorld;
186
- }
187
- // function translate4(a, x, y, z) {
188
- // return [
189
- // ...a.slice(0, 12),
190
- // a[0] * x + a[4] * y + a[8] * z + a[12],
191
- // a[1] * x + a[5] * y + a[9] * z + a[13],
192
- // a[2] * x + a[6] * y + a[10] * z + a[14],
193
- // a[3] * x + a[7] * y + a[11] * z + a[15],
194
- // ];
195
- // }
196
-
197
- function multiply4(a, b) {
198
- return [
199
- b[0] * a[0] + b[1] * a[4] + b[2] * a[8] + b[3] * a[12],
200
- b[0] * a[1] + b[1] * a[5] + b[2] * a[9] + b[3] * a[13],
201
- b[0] * a[2] + b[1] * a[6] + b[2] * a[10] + b[3] * a[14],
202
- b[0] * a[3] + b[1] * a[7] + b[2] * a[11] + b[3] * a[15],
203
- b[4] * a[0] + b[5] * a[4] + b[6] * a[8] + b[7] * a[12],
204
- b[4] * a[1] + b[5] * a[5] + b[6] * a[9] + b[7] * a[13],
205
- b[4] * a[2] + b[5] * a[6] + b[6] * a[10] + b[7] * a[14],
206
- b[4] * a[3] + b[5] * a[7] + b[6] * a[11] + b[7] * a[15],
207
- b[8] * a[0] + b[9] * a[4] + b[10] * a[8] + b[11] * a[12],
208
- b[8] * a[1] + b[9] * a[5] + b[10] * a[9] + b[11] * a[13],
209
- b[8] * a[2] + b[9] * a[6] + b[10] * a[10] + b[11] * a[14],
210
- b[8] * a[3] + b[9] * a[7] + b[10] * a[11] + b[11] * a[15],
211
- b[12] * a[0] + b[13] * a[4] + b[14] * a[8] + b[15] * a[12],
212
- b[12] * a[1] + b[13] * a[5] + b[14] * a[9] + b[15] * a[13],
213
- b[12] * a[2] + b[13] * a[6] + b[14] * a[10] + b[15] * a[14],
214
- b[12] * a[3] + b[13] * a[7] + b[14] * a[11] + b[15] * a[15],
215
- ];
216
- }
217
-
218
- function invert4(a) {
219
- let b00 = a[0] * a[5] - a[1] * a[4];
220
- let b01 = a[0] * a[6] - a[2] * a[4];
221
- let b02 = a[0] * a[7] - a[3] * a[4];
222
- let b03 = a[1] * a[6] - a[2] * a[5];
223
- let b04 = a[1] * a[7] - a[3] * a[5];
224
- let b05 = a[2] * a[7] - a[3] * a[6];
225
- let b06 = a[8] * a[13] - a[9] * a[12];
226
- let b07 = a[8] * a[14] - a[10] * a[12];
227
- let b08 = a[8] * a[15] - a[11] * a[12];
228
- let b09 = a[9] * a[14] - a[10] * a[13];
229
- let b10 = a[9] * a[15] - a[11] * a[13];
230
- let b11 = a[10] * a[15] - a[11] * a[14];
231
- let det =
232
- b00 * b11 - b01 * b10 + b02 * b09 + b03 * b08 - b04 * b07 + b05 * b06;
233
- if (!det) return null;
234
- return [
235
- (a[5] * b11 - a[6] * b10 + a[7] * b09) / det,
236
- (a[2] * b10 - a[1] * b11 - a[3] * b09) / det,
237
- (a[13] * b05 - a[14] * b04 + a[15] * b03) / det,
238
- (a[10] * b04 - a[9] * b05 - a[11] * b03) / det,
239
- (a[6] * b08 - a[4] * b11 - a[7] * b07) / det,
240
- (a[0] * b11 - a[2] * b08 + a[3] * b07) / det,
241
- (a[14] * b02 - a[12] * b05 - a[15] * b01) / det,
242
- (a[8] * b05 - a[10] * b02 + a[11] * b01) / det,
243
- (a[4] * b10 - a[5] * b08 + a[7] * b06) / det,
244
- (a[1] * b08 - a[0] * b10 - a[3] * b06) / det,
245
- (a[12] * b04 - a[13] * b02 + a[15] * b00) / det,
246
- (a[9] * b02 - a[8] * b04 - a[11] * b00) / det,
247
- (a[5] * b07 - a[4] * b09 - a[6] * b06) / det,
248
- (a[0] * b09 - a[1] * b07 + a[2] * b06) / det,
249
- (a[13] * b01 - a[12] * b03 - a[14] * b00) / det,
250
- (a[8] * b03 - a[9] * b01 + a[10] * b00) / det,
251
- ];
252
- }
253
-
254
- function rotate4(a, rad, x, y, z) {
255
- let len = Math.hypot(x, y, z);
256
- x /= len;
257
- y /= len;
258
- z /= len;
259
- let s = Math.sin(rad);
260
- let c = Math.cos(rad);
261
- let t = 1 - c;
262
- let b00 = x * x * t + c;
263
- let b01 = y * x * t + z * s;
264
- let b02 = z * x * t - y * s;
265
- let b10 = x * y * t - z * s;
266
- let b11 = y * y * t + c;
267
- let b12 = z * y * t + x * s;
268
- let b20 = x * z * t + y * s;
269
- let b21 = y * z * t - x * s;
270
- let b22 = z * z * t + c;
271
- return [
272
- a[0] * b00 + a[4] * b01 + a[8] * b02,
273
- a[1] * b00 + a[5] * b01 + a[9] * b02,
274
- a[2] * b00 + a[6] * b01 + a[10] * b02,
275
- a[3] * b00 + a[7] * b01 + a[11] * b02,
276
- a[0] * b10 + a[4] * b11 + a[8] * b12,
277
- a[1] * b10 + a[5] * b11 + a[9] * b12,
278
- a[2] * b10 + a[6] * b11 + a[10] * b12,
279
- a[3] * b10 + a[7] * b11 + a[11] * b12,
280
- a[0] * b20 + a[4] * b21 + a[8] * b22,
281
- a[1] * b20 + a[5] * b21 + a[9] * b22,
282
- a[2] * b20 + a[6] * b21 + a[10] * b22,
283
- a[3] * b20 + a[7] * b21 + a[11] * b22,
284
- ...a.slice(12, 16),
285
- ];
286
- }
287
-
288
- function translate4(a, x, y, z) {
289
- return [
290
- ...a.slice(0, 12),
291
- a[0] * x + a[4] * y + a[8] * z + a[12],
292
- a[1] * x + a[5] * y + a[9] * z + a[13],
293
- a[2] * x + a[6] * y + a[10] * z + a[14],
294
- a[3] * x + a[7] * y + a[11] * z + a[15],
295
- ];
296
- }
297
-
298
- function createWorker(self) {
299
- let buffer;
300
- let vertexCount = 0;
301
- let viewProj;
302
- // 6*4 + 4 + 4 = 8*4
303
- // XYZ - Position (Float32)
304
- // XYZ - Scale (Float32)
305
- // RGBA - colors (uint8)
306
- // IJKL - quaternion/rot (uint8)
307
- const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
308
- let lastProj = [];
309
- let depthIndex = new Uint32Array();
310
- let lastVertexCount = 0;
311
-
312
- var _floatView = new Float32Array(1);
313
- var _int32View = new Int32Array(_floatView.buffer);
314
-
315
- function floatToHalf(float) {
316
- _floatView[0] = float;
317
- var f = _int32View[0];
318
-
319
- var sign = (f >> 31) & 0x0001;
320
- var exp = (f >> 23) & 0x00ff;
321
- var frac = f & 0x007fffff;
322
-
323
- var newExp;
324
- if (exp == 0) {
325
- newExp = 0;
326
- } else if (exp < 113) {
327
- newExp = 0;
328
- frac |= 0x00800000;
329
- frac = frac >> (113 - exp);
330
- if (frac & 0x01000000) {
331
- newExp = 1;
332
- frac = 0;
333
- }
334
- } else if (exp < 142) {
335
- newExp = exp - 112;
336
- } else {
337
- newExp = 31;
338
- frac = 0;
339
- }
340
-
341
- return (sign << 15) | (newExp << 10) | (frac >> 13);
342
- }
343
-
344
- function packHalf2x16(x, y) {
345
- return (floatToHalf(x) | (floatToHalf(y) << 16)) >>> 0;
346
- }
347
-
348
- function generateTexture() {
349
- if (!buffer) return;
350
- const f_buffer = new Float32Array(buffer);
351
- const u_buffer = new Uint8Array(buffer);
352
-
353
- var texwidth = 1024 * 2; // Set to your desired width
354
- var texheight = Math.ceil((2 * vertexCount) / texwidth); // Set to your desired height
355
- var texdata = new Uint32Array(texwidth * texheight * 4); // 4 components per pixel (RGBA)
356
- var texdata_c = new Uint8Array(texdata.buffer);
357
- var texdata_f = new Float32Array(texdata.buffer);
358
-
359
- // Here we convert from a .splat file buffer into a texture
360
- // With a little bit more foresight perhaps this texture file
361
- // should have been the native format as it'd be very easy to
362
- // load it into webgl.
363
- for (let i = 0; i < vertexCount; i++) {
364
- // x, y, z
365
- texdata_f[8 * i + 0] = f_buffer[8 * i + 0];
366
- texdata_f[8 * i + 1] = f_buffer[8 * i + 1];
367
- texdata_f[8 * i + 2] = f_buffer[8 * i + 2];
368
-
369
- // r, g, b, a
370
- texdata_c[4 * (8 * i + 7) + 0] = u_buffer[32 * i + 24 + 0];
371
- texdata_c[4 * (8 * i + 7) + 1] = u_buffer[32 * i + 24 + 1];
372
- texdata_c[4 * (8 * i + 7) + 2] = u_buffer[32 * i + 24 + 2];
373
- texdata_c[4 * (8 * i + 7) + 3] = u_buffer[32 * i + 24 + 3];
374
-
375
- // quaternions
376
- let scale = [
377
- f_buffer[8 * i + 3 + 0],
378
- f_buffer[8 * i + 3 + 1],
379
- f_buffer[8 * i + 3 + 2],
380
- ];
381
- let rot = [
382
- (u_buffer[32 * i + 28 + 0] - 128) / 128,
383
- (u_buffer[32 * i + 28 + 1] - 128) / 128,
384
- (u_buffer[32 * i + 28 + 2] - 128) / 128,
385
- (u_buffer[32 * i + 28 + 3] - 128) / 128,
386
- ];
387
-
388
- // Compute the matrix product of S and R (M = S * R)
389
- const M = [
390
- 1.0 - 2.0 * (rot[2] * rot[2] + rot[3] * rot[3]),
391
- 2.0 * (rot[1] * rot[2] + rot[0] * rot[3]),
392
- 2.0 * (rot[1] * rot[3] - rot[0] * rot[2]),
393
-
394
- 2.0 * (rot[1] * rot[2] - rot[0] * rot[3]),
395
- 1.0 - 2.0 * (rot[1] * rot[1] + rot[3] * rot[3]),
396
- 2.0 * (rot[2] * rot[3] + rot[0] * rot[1]),
397
-
398
- 2.0 * (rot[1] * rot[3] + rot[0] * rot[2]),
399
- 2.0 * (rot[2] * rot[3] - rot[0] * rot[1]),
400
- 1.0 - 2.0 * (rot[1] * rot[1] + rot[2] * rot[2]),
401
- ].map((k, i) => k * scale[Math.floor(i / 3)]);
402
-
403
- const sigma = [
404
- M[0] * M[0] + M[3] * M[3] + M[6] * M[6],
405
- M[0] * M[1] + M[3] * M[4] + M[6] * M[7],
406
- M[0] * M[2] + M[3] * M[5] + M[6] * M[8],
407
- M[1] * M[1] + M[4] * M[4] + M[7] * M[7],
408
- M[1] * M[2] + M[4] * M[5] + M[7] * M[8],
409
- M[2] * M[2] + M[5] * M[5] + M[8] * M[8],
410
- ];
411
-
412
- texdata[8 * i + 4] = packHalf2x16(4 * sigma[0], 4 * sigma[1]);
413
- texdata[8 * i + 5] = packHalf2x16(4 * sigma[2], 4 * sigma[3]);
414
- texdata[8 * i + 6] = packHalf2x16(4 * sigma[4], 4 * sigma[5]);
415
- }
416
-
417
- self.postMessage({ texdata, texwidth, texheight }, [texdata.buffer]);
418
- }
419
-
420
- function runSort(viewProj) {
421
- if (!buffer) return;
422
- const f_buffer = new Float32Array(buffer);
423
- if (lastVertexCount == vertexCount) {
424
- let dot =
425
- lastProj[2] * viewProj[2] +
426
- lastProj[6] * viewProj[6] +
427
- lastProj[10] * viewProj[10];
428
- if (Math.abs(dot - 1) < 0.01) {
429
- return;
430
- }
431
- } else {
432
- generateTexture();
433
- lastVertexCount = vertexCount;
434
- }
435
-
436
- console.time("sort");
437
- let maxDepth = -Infinity;
438
- let minDepth = Infinity;
439
- let sizeList = new Int32Array(vertexCount);
440
- for (let i = 0; i < vertexCount; i++) {
441
- let depth =
442
- ((viewProj[2] * f_buffer[8 * i + 0] +
443
- viewProj[6] * f_buffer[8 * i + 1] +
444
- viewProj[10] * f_buffer[8 * i + 2]) *
445
- 4096) |
446
- 0;
447
- sizeList[i] = depth;
448
- if (depth > maxDepth) maxDepth = depth;
449
- if (depth < minDepth) minDepth = depth;
450
- }
451
-
452
- // This is a 16 bit single-pass counting sort
453
- let depthInv = (256 * 256 - 1) / (maxDepth - minDepth);
454
- let counts0 = new Uint32Array(256 * 256);
455
- for (let i = 0; i < vertexCount; i++) {
456
- sizeList[i] = ((sizeList[i] - minDepth) * depthInv) | 0;
457
- counts0[sizeList[i]]++;
458
- }
459
- let starts0 = new Uint32Array(256 * 256);
460
- for (let i = 1; i < 256 * 256; i++)
461
- starts0[i] = starts0[i - 1] + counts0[i - 1];
462
- depthIndex = new Uint32Array(vertexCount);
463
- for (let i = 0; i < vertexCount; i++)
464
- depthIndex[starts0[sizeList[i]]++] = i;
465
-
466
- console.timeEnd("sort");
467
-
468
- lastProj = viewProj;
469
- self.postMessage({ depthIndex, viewProj, vertexCount }, [
470
- depthIndex.buffer,
471
- ]);
472
- }
473
-
474
- function processPlyBuffer(inputBuffer) {
475
- const ubuf = new Uint8Array(inputBuffer);
476
- // 10KB ought to be enough for a header...
477
- const header = new TextDecoder().decode(ubuf.slice(0, 1024 * 10));
478
- const header_end = "end_header\n";
479
- const header_end_index = header.indexOf(header_end);
480
- if (header_end_index < 0)
481
- throw new Error("Unable to read .ply file header");
482
- const vertexCount = parseInt(/element vertex (\d+)\n/.exec(header)[1]);
483
- console.log("Vertex Count", vertexCount);
484
- let row_offset = 0,
485
- offsets = {},
486
- types = {};
487
- const TYPE_MAP = {
488
- double: "getFloat64",
489
- int: "getInt32",
490
- uint: "getUint32",
491
- float: "getFloat32",
492
- short: "getInt16",
493
- ushort: "getUint16",
494
- uchar: "getUint8",
495
- };
496
- for (let prop of header
497
- .slice(0, header_end_index)
498
- .split("\n")
499
- .filter((k) => k.startsWith("property "))) {
500
- const [p, type, name] = prop.split(" ");
501
- const arrayType = TYPE_MAP[type] || "getInt8";
502
- types[name] = arrayType;
503
- offsets[name] = row_offset;
504
- row_offset += parseInt(arrayType.replace(/[^\d]/g, "")) / 8;
505
- }
506
- console.log("Bytes per row", row_offset, types, offsets);
507
-
508
- let dataView = new DataView(
509
- inputBuffer,
510
- header_end_index + header_end.length,
511
- );
512
- let row = 0;
513
- const attrs = new Proxy(
514
- {},
515
- {
516
- get(target, prop) {
517
- if (!types[prop]) throw new Error(prop + " not found");
518
- return dataView[types[prop]](
519
- row * row_offset + offsets[prop],
520
- true,
521
- );
522
- },
523
- },
524
- );
525
-
526
- console.time("calculate importance");
527
- let sizeList = new Float32Array(vertexCount);
528
- let sizeIndex = new Uint32Array(vertexCount);
529
- for (row = 0; row < vertexCount; row++) {
530
- sizeIndex[row] = row;
531
- if (!types["scale_0"]) continue;
532
- const size =
533
- Math.exp(attrs.scale_0) *
534
- Math.exp(attrs.scale_1) *
535
- Math.exp(attrs.scale_2);
536
- const opacity = 1 / (1 + Math.exp(-attrs.opacity));
537
- sizeList[row] = size * opacity;
538
- }
539
- console.timeEnd("calculate importance");
540
-
541
- console.time("sort");
542
- sizeIndex.sort((b, a) => sizeList[a] - sizeList[b]);
543
- console.timeEnd("sort");
544
-
545
- // 6*4 + 4 + 4 = 8*4
546
- // XYZ - Position (Float32)
547
- // XYZ - Scale (Float32)
548
- // RGBA - colors (uint8)
549
- // IJKL - quaternion/rot (uint8)
550
- const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
551
- const buffer = new ArrayBuffer(rowLength * vertexCount);
552
-
553
- console.time("build buffer");
554
- for (let j = 0; j < vertexCount; j++) {
555
- row = sizeIndex[j];
556
-
557
- const position = new Float32Array(buffer, j * rowLength, 3);
558
- const scales = new Float32Array(buffer, j * rowLength + 4 * 3, 3);
559
- const rgba = new Uint8ClampedArray(
560
- buffer,
561
- j * rowLength + 4 * 3 + 4 * 3,
562
- 4,
563
- );
564
- const rot = new Uint8ClampedArray(
565
- buffer,
566
- j * rowLength + 4 * 3 + 4 * 3 + 4,
567
- 4,
568
- );
569
-
570
- if (types["scale_0"]) {
571
- const qlen = Math.sqrt(
572
- attrs.rot_0 ** 2 +
573
- attrs.rot_1 ** 2 +
574
- attrs.rot_2 ** 2 +
575
- attrs.rot_3 ** 2,
576
- );
577
-
578
- rot[0] = (attrs.rot_0 / qlen) * 128 + 128;
579
- rot[1] = (attrs.rot_1 / qlen) * 128 + 128;
580
- rot[2] = (attrs.rot_2 / qlen) * 128 + 128;
581
- rot[3] = (attrs.rot_3 / qlen) * 128 + 128;
582
-
583
- scales[0] = Math.exp(attrs.scale_0);
584
- scales[1] = Math.exp(attrs.scale_1);
585
- scales[2] = Math.exp(attrs.scale_2);
586
- } else {
587
- scales[0] = 0.01;
588
- scales[1] = 0.01;
589
- scales[2] = 0.01;
590
-
591
- rot[0] = 255;
592
- rot[1] = 0;
593
- rot[2] = 0;
594
- rot[3] = 0;
595
- }
596
-
597
- position[0] = attrs.x;
598
- position[1] = attrs.y;
599
- position[2] = attrs.z;
600
-
601
- if (types["f_dc_0"]) {
602
- const SH_C0 = 0.28209479177387814;
603
- rgba[0] = (0.5 + SH_C0 * attrs.f_dc_0) * 255;
604
- rgba[1] = (0.5 + SH_C0 * attrs.f_dc_1) * 255;
605
- rgba[2] = (0.5 + SH_C0 * attrs.f_dc_2) * 255;
606
- } else {
607
- rgba[0] = attrs.red;
608
- rgba[1] = attrs.green;
609
- rgba[2] = attrs.blue;
610
- }
611
- if (types["opacity"]) {
612
- rgba[3] = (1 / (1 + Math.exp(-attrs.opacity))) * 255;
613
- } else {
614
- rgba[3] = 255;
615
- }
616
- }
617
- console.timeEnd("build buffer");
618
- return buffer;
619
- }
620
-
621
- const throttledSort = () => {
622
- if (!sortRunning) {
623
- sortRunning = true;
624
- let lastView = viewProj;
625
- runSort(lastView);
626
- setTimeout(() => {
627
- sortRunning = false;
628
- if (lastView !== viewProj) {
629
- throttledSort();
630
- }
631
- }, 0);
632
- }
633
- };
634
-
635
- let sortRunning;
636
- self.onmessage = (e) => {
637
- if (e.data.ply) {
638
- vertexCount = 0;
639
- runSort(viewProj);
640
- buffer = processPlyBuffer(e.data.ply);
641
- vertexCount = Math.floor(buffer.byteLength / rowLength);
642
- postMessage({ buffer: buffer, save: !!e.data.save });
643
- } else if (e.data.buffer) {
644
- buffer = e.data.buffer;
645
- vertexCount = e.data.vertexCount;
646
- } else if (e.data.vertexCount) {
647
- vertexCount = e.data.vertexCount;
648
- } else if (e.data.view) {
649
- viewProj = e.data.view;
650
- throttledSort();
651
- }
652
- };
653
- }
654
-
655
- const vertexShaderSource = `
656
- #version 300 es
657
- precision highp float;
658
- precision highp int;
659
-
660
- uniform highp usampler2D u_texture;
661
- uniform mat4 projection, view;
662
- uniform vec2 focal;
663
- uniform vec2 viewport;
664
-
665
- in vec2 position;
666
- in int index;
667
-
668
- out vec4 vColor;
669
- out vec2 vPosition;
670
-
671
- void main () {
672
- uvec4 cen = texelFetch(u_texture, ivec2((uint(index) & 0x3ffu) << 1, uint(index) >> 10), 0);
673
- vec4 cam = view * vec4(uintBitsToFloat(cen.xyz), 1);
674
- vec4 pos2d = projection * cam;
675
-
676
- float clip = 1.2 * pos2d.w;
677
- if (pos2d.z < -clip || pos2d.x < -clip || pos2d.x > clip || pos2d.y < -clip || pos2d.y > clip) {
678
- gl_Position = vec4(0.0, 0.0, 2.0, 1.0);
679
- return;
680
- }
681
-
682
- uvec4 cov = texelFetch(u_texture, ivec2(((uint(index) & 0x3ffu) << 1) | 1u, uint(index) >> 10), 0);
683
- vec2 u1 = unpackHalf2x16(cov.x), u2 = unpackHalf2x16(cov.y), u3 = unpackHalf2x16(cov.z);
684
- mat3 Vrk = mat3(u1.x, u1.y, u2.x, u1.y, u2.y, u3.x, u2.x, u3.x, u3.y);
685
-
686
- mat3 J = mat3(
687
- focal.x / cam.z, 0., -(focal.x * cam.x) / (cam.z * cam.z),
688
- 0., -focal.y / cam.z, (focal.y * cam.y) / (cam.z * cam.z),
689
- 0., 0., 0.
690
- );
691
-
692
- mat3 T = transpose(mat3(view)) * J;
693
- mat3 cov2d = transpose(T) * Vrk * T;
694
-
695
- float mid = (cov2d[0][0] + cov2d[1][1]) / 2.0;
696
- float radius = length(vec2((cov2d[0][0] - cov2d[1][1]) / 2.0, cov2d[0][1]));
697
- float lambda1 = mid + radius, lambda2 = mid - radius;
698
-
699
- if(lambda2 < 0.0) return;
700
- vec2 diagonalVector = normalize(vec2(cov2d[0][1], lambda1 - cov2d[0][0]));
701
- vec2 majorAxis = min(sqrt(2.0 * lambda1), 1024.0) * diagonalVector;
702
- vec2 minorAxis = min(sqrt(2.0 * lambda2), 1024.0) * vec2(diagonalVector.y, -diagonalVector.x);
703
-
704
- vColor = clamp(pos2d.z/pos2d.w+1.0, 0.0, 1.0) * vec4((cov.w) & 0xffu, (cov.w >> 8) & 0xffu, (cov.w >> 16) & 0xffu, (cov.w >> 24) & 0xffu) / 255.0;
705
- vPosition = position;
706
-
707
- vec2 vCenter = vec2(pos2d) / pos2d.w;
708
- gl_Position = vec4(
709
- vCenter
710
- + position.x * majorAxis / viewport
711
- + position.y * minorAxis / viewport, 0.0, 1.0);
712
-
713
- }
714
- `.trim();
715
-
716
- const fragmentShaderSource = `
717
- #version 300 es
718
- precision highp float;
719
-
720
- in vec4 vColor;
721
- in vec2 vPosition;
722
-
723
- out vec4 fragColor;
724
-
725
- void main () {
726
- float A = -dot(vPosition, vPosition);
727
- if (A < -4.0) discard;
728
- float B = exp(A) * vColor.a;
729
- fragColor = vec4(B * vColor.rgb, B);
730
- }
731
-
732
- `.trim();
733
-
734
- let defaultViewMatrix = [
735
- 0.47, 0.04, 0.88, 0, -0.11, 0.99, 0.02, 0, -0.88, -0.11, 0.47, 0, 0.07,
736
- 0.03, 6.55, 1,
737
- ];
738
- let viewMatrix = defaultViewMatrix;
739
- async function main() {
740
- let carousel = true;
741
- const params = new URLSearchParams(location.search);
742
- try {
743
- viewMatrix = JSON.parse(decodeURIComponent(location.hash.slice(1)));
744
- carousel = false;
745
- } catch (err) {}
746
- const url = new URL(
747
- // "nike.splat",
748
- // location.href,
749
- params.get("url") || "train.splat",
750
- "https://huggingface.co/cakewalk/splat-data/resolve/main/",
751
- );
752
- const req = await fetch(url, {
753
- mode: "cors", // no-cors, *cors, same-origin
754
- credentials: "omit", // include, *same-origin, omit
755
- });
756
- console.log(req);
757
- if (req.status != 200)
758
- throw new Error(req.status + " Unable to load " + req.url);
759
-
760
- const rowLength = 3 * 4 + 3 * 4 + 4 + 4;
761
- const reader = req.body.getReader();
762
- let splatData = new Uint8Array(req.headers.get("content-length"));
763
-
764
- const downsample =
765
- splatData.length / rowLength > 500000 ? 1 : 1 / devicePixelRatio;
766
- console.log(splatData.length / rowLength, downsample);
767
-
768
- const worker = new Worker(
769
- URL.createObjectURL(
770
- new Blob(["(", createWorker.toString(), ")(self)"], {
771
- type: "application/javascript",
772
- }),
773
- ),
774
- );
775
-
776
- const canvas = document.getElementById("canvas");
777
- const fps = document.getElementById("fps");
778
- const camid = document.getElementById("camid");
779
-
780
- let projectionMatrix;
781
-
782
- const gl = canvas.getContext("webgl2", {
783
- antialias: false,
784
- });
785
-
786
- const vertexShader = gl.createShader(gl.VERTEX_SHADER);
787
- gl.shaderSource(vertexShader, vertexShaderSource);
788
- gl.compileShader(vertexShader);
789
- if (!gl.getShaderParameter(vertexShader, gl.COMPILE_STATUS))
790
- console.error(gl.getShaderInfoLog(vertexShader));
791
-
792
- const fragmentShader = gl.createShader(gl.FRAGMENT_SHADER);
793
- gl.shaderSource(fragmentShader, fragmentShaderSource);
794
- gl.compileShader(fragmentShader);
795
- if (!gl.getShaderParameter(fragmentShader, gl.COMPILE_STATUS))
796
- console.error(gl.getShaderInfoLog(fragmentShader));
797
-
798
- const program = gl.createProgram();
799
- gl.attachShader(program, vertexShader);
800
- gl.attachShader(program, fragmentShader);
801
- gl.linkProgram(program);
802
- gl.useProgram(program);
803
-
804
- if (!gl.getProgramParameter(program, gl.LINK_STATUS))
805
- console.error(gl.getProgramInfoLog(program));
806
-
807
- gl.disable(gl.DEPTH_TEST); // Disable depth testing
808
-
809
- // Enable blending
810
- gl.enable(gl.BLEND);
811
- gl.blendFuncSeparate(
812
- gl.ONE_MINUS_DST_ALPHA,
813
- gl.ONE,
814
- gl.ONE_MINUS_DST_ALPHA,
815
- gl.ONE,
816
- );
817
- gl.blendEquationSeparate(gl.FUNC_ADD, gl.FUNC_ADD);
818
-
819
- const u_projection = gl.getUniformLocation(program, "projection");
820
- const u_viewport = gl.getUniformLocation(program, "viewport");
821
- const u_focal = gl.getUniformLocation(program, "focal");
822
- const u_view = gl.getUniformLocation(program, "view");
823
-
824
- // positions
825
- const triangleVertices = new Float32Array([-2, -2, 2, -2, 2, 2, -2, 2]);
826
- const vertexBuffer = gl.createBuffer();
827
- gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
828
- gl.bufferData(gl.ARRAY_BUFFER, triangleVertices, gl.STATIC_DRAW);
829
- const a_position = gl.getAttribLocation(program, "position");
830
- gl.enableVertexAttribArray(a_position);
831
- gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
832
- gl.vertexAttribPointer(a_position, 2, gl.FLOAT, false, 0, 0);
833
-
834
- var texture = gl.createTexture();
835
- gl.bindTexture(gl.TEXTURE_2D, texture);
836
-
837
- var u_textureLocation = gl.getUniformLocation(program, "u_texture");
838
- gl.uniform1i(u_textureLocation, 0);
839
-
840
- const indexBuffer = gl.createBuffer();
841
- const a_index = gl.getAttribLocation(program, "index");
842
- gl.enableVertexAttribArray(a_index);
843
- gl.bindBuffer(gl.ARRAY_BUFFER, indexBuffer);
844
- gl.vertexAttribIPointer(a_index, 1, gl.INT, false, 0, 0);
845
- gl.vertexAttribDivisor(a_index, 1);
846
-
847
- const resize = () => {
848
- gl.uniform2fv(u_focal, new Float32Array([camera.fx, camera.fy]));
849
-
850
- projectionMatrix = getProjectionMatrix(
851
- camera.fx,
852
- camera.fy,
853
- innerWidth,
854
- innerHeight,
855
- );
856
-
857
- gl.uniform2fv(u_viewport, new Float32Array([innerWidth, innerHeight]));
858
-
859
- gl.canvas.width = Math.round(innerWidth / downsample);
860
- gl.canvas.height = Math.round(innerHeight / downsample);
861
- gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);
862
-
863
- gl.uniformMatrix4fv(u_projection, false, projectionMatrix);
864
- };
865
-
866
- window.addEventListener("resize", resize);
867
- resize();
868
-
869
- worker.onmessage = (e) => {
870
- if (e.data.buffer) {
871
- splatData = new Uint8Array(e.data.buffer);
872
- if (e.data.save) {
873
- const blob = new Blob([splatData.buffer], {
874
- type: "application/octet-stream",
875
- });
876
- const link = document.createElement("a");
877
- link.download = "model.splat";
878
- link.href = URL.createObjectURL(blob);
879
- document.body.appendChild(link);
880
- link.click();
881
- }
882
- } else if (e.data.texdata) {
883
- const { texdata, texwidth, texheight } = e.data;
884
- // console.log(texdata)
885
- gl.bindTexture(gl.TEXTURE_2D, texture);
886
- gl.texParameteri(
887
- gl.TEXTURE_2D,
888
- gl.TEXTURE_WRAP_S,
889
- gl.CLAMP_TO_EDGE,
890
- );
891
- gl.texParameteri(
892
- gl.TEXTURE_2D,
893
- gl.TEXTURE_WRAP_T,
894
- gl.CLAMP_TO_EDGE,
895
- );
896
- gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
897
- gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
898
-
899
- gl.texImage2D(
900
- gl.TEXTURE_2D,
901
- 0,
902
- gl.RGBA32UI,
903
- texwidth,
904
- texheight,
905
- 0,
906
- gl.RGBA_INTEGER,
907
- gl.UNSIGNED_INT,
908
- texdata,
909
- );
910
- gl.activeTexture(gl.TEXTURE0);
911
- gl.bindTexture(gl.TEXTURE_2D, texture);
912
- } else if (e.data.depthIndex) {
913
- const { depthIndex, viewProj } = e.data;
914
- gl.bindBuffer(gl.ARRAY_BUFFER, indexBuffer);
915
- gl.bufferData(gl.ARRAY_BUFFER, depthIndex, gl.DYNAMIC_DRAW);
916
- vertexCount = e.data.vertexCount;
917
- }
918
- };
919
-
920
- let activeKeys = [];
921
- let currentCameraIndex = 0;
922
-
923
- window.addEventListener("keydown", (e) => {
924
- // if (document.activeElement != document.body) return;
925
- carousel = false;
926
- if (!activeKeys.includes(e.code)) activeKeys.push(e.code);
927
- if (/\d/.test(e.key)) {
928
- currentCameraIndex = parseInt(e.key);
929
- camera = cameras[currentCameraIndex];
930
- viewMatrix = getViewMatrix(camera);
931
- }
932
- if (["-", "_"].includes(e.key)) {
933
- currentCameraIndex =
934
- (currentCameraIndex + cameras.length - 1) % cameras.length;
935
- viewMatrix = getViewMatrix(cameras[currentCameraIndex]);
936
- }
937
- if (["+", "="].includes(e.key)) {
938
- currentCameraIndex = (currentCameraIndex + 1) % cameras.length;
939
- viewMatrix = getViewMatrix(cameras[currentCameraIndex]);
940
- }
941
- camid.innerText = "cam " + currentCameraIndex;
942
- if (e.code == "KeyV") {
943
- location.hash =
944
- "#" +
945
- JSON.stringify(
946
- viewMatrix.map((k) => Math.round(k * 100) / 100),
947
- );
948
- camid.innerText = "";
949
- } else if (e.code === "KeyP") {
950
- carousel = true;
951
- camid.innerText = "";
952
- }
953
- });
954
- window.addEventListener("keyup", (e) => {
955
- activeKeys = activeKeys.filter((k) => k !== e.code);
956
- });
957
- window.addEventListener("blur", () => {
958
- activeKeys = [];
959
- });
960
-
961
- window.addEventListener(
962
- "wheel",
963
- (e) => {
964
- carousel = false;
965
- e.preventDefault();
966
- const lineHeight = 10;
967
- const scale =
968
- e.deltaMode == 1
969
- ? lineHeight
970
- : e.deltaMode == 2
971
- ? innerHeight
972
- : 1;
973
- let inv = invert4(viewMatrix);
974
- if (e.shiftKey) {
975
- inv = translate4(
976
- inv,
977
- (e.deltaX * scale) / innerWidth,
978
- (e.deltaY * scale) / innerHeight,
979
- 0,
980
- );
981
- } else if (e.ctrlKey || e.metaKey) {
982
- // inv = rotate4(inv, (e.deltaX * scale) / innerWidth, 0, 0, 1);
983
- // inv = translate4(inv, 0, (e.deltaY * scale) / innerHeight, 0);
984
- // let preY = inv[13];
985
- inv = translate4(
986
- inv,
987
- 0,
988
- 0,
989
- (-10 * (e.deltaY * scale)) / innerHeight,
990
- );
991
- // inv[13] = preY;
992
- } else {
993
- let d = 4;
994
- inv = translate4(inv, 0, 0, d);
995
- inv = rotate4(inv, -(e.deltaX * scale) / innerWidth, 0, 1, 0);
996
- inv = rotate4(inv, (e.deltaY * scale) / innerHeight, 1, 0, 0);
997
- inv = translate4(inv, 0, 0, -d);
998
- }
999
-
1000
- viewMatrix = invert4(inv);
1001
- },
1002
- { passive: false },
1003
- );
1004
-
1005
- let startX, startY, down;
1006
- canvas.addEventListener("mousedown", (e) => {
1007
- carousel = false;
1008
- e.preventDefault();
1009
- startX = e.clientX;
1010
- startY = e.clientY;
1011
- down = e.ctrlKey || e.metaKey ? 2 : 1;
1012
- });
1013
- canvas.addEventListener("contextmenu", (e) => {
1014
- carousel = false;
1015
- e.preventDefault();
1016
- startX = e.clientX;
1017
- startY = e.clientY;
1018
- down = 2;
1019
- });
1020
-
1021
- canvas.addEventListener("mousemove", (e) => {
1022
- e.preventDefault();
1023
- if (down == 1) {
1024
- let inv = invert4(viewMatrix);
1025
- let dx = (5 * (e.clientX - startX)) / innerWidth;
1026
- let dy = (5 * (e.clientY - startY)) / innerHeight;
1027
- let d = 4;
1028
-
1029
- inv = translate4(inv, 0, 0, d);
1030
- inv = rotate4(inv, dx, 0, 1, 0);
1031
- inv = rotate4(inv, -dy, 1, 0, 0);
1032
- inv = translate4(inv, 0, 0, -d);
1033
- // let postAngle = Math.atan2(inv[0], inv[10])
1034
- // inv = rotate4(inv, postAngle - preAngle, 0, 0, 1)
1035
- // console.log(postAngle)
1036
- viewMatrix = invert4(inv);
1037
-
1038
- startX = e.clientX;
1039
- startY = e.clientY;
1040
- } else if (down == 2) {
1041
- let inv = invert4(viewMatrix);
1042
- // inv = rotateY(inv, );
1043
- // let preY = inv[13];
1044
- inv = translate4(
1045
- inv,
1046
- (-10 * (e.clientX - startX)) / innerWidth,
1047
- 0,
1048
- (10 * (e.clientY - startY)) / innerHeight,
1049
- );
1050
- // inv[13] = preY;
1051
- viewMatrix = invert4(inv);
1052
-
1053
- startX = e.clientX;
1054
- startY = e.clientY;
1055
- }
1056
- });
1057
- canvas.addEventListener("mouseup", (e) => {
1058
- e.preventDefault();
1059
- down = false;
1060
- startX = 0;
1061
- startY = 0;
1062
- });
1063
-
1064
- let altX = 0,
1065
- altY = 0;
1066
- canvas.addEventListener(
1067
- "touchstart",
1068
- (e) => {
1069
- e.preventDefault();
1070
- if (e.touches.length === 1) {
1071
- carousel = false;
1072
- startX = e.touches[0].clientX;
1073
- startY = e.touches[0].clientY;
1074
- down = 1;
1075
- } else if (e.touches.length === 2) {
1076
- // console.log('beep')
1077
- carousel = false;
1078
- startX = e.touches[0].clientX;
1079
- altX = e.touches[1].clientX;
1080
- startY = e.touches[0].clientY;
1081
- altY = e.touches[1].clientY;
1082
- down = 1;
1083
- }
1084
- },
1085
- { passive: false },
1086
- );
1087
- canvas.addEventListener(
1088
- "touchmove",
1089
- (e) => {
1090
- e.preventDefault();
1091
- if (e.touches.length === 1 && down) {
1092
- let inv = invert4(viewMatrix);
1093
- let dx = (4 * (e.touches[0].clientX - startX)) / innerWidth;
1094
- let dy = (4 * (e.touches[0].clientY - startY)) / innerHeight;
1095
-
1096
- let d = 4;
1097
- inv = translate4(inv, 0, 0, d);
1098
- // inv = translate4(inv, -x, -y, -z);
1099
- // inv = translate4(inv, x, y, z);
1100
- inv = rotate4(inv, dx, 0, 1, 0);
1101
- inv = rotate4(inv, -dy, 1, 0, 0);
1102
- inv = translate4(inv, 0, 0, -d);
1103
-
1104
- viewMatrix = invert4(inv);
1105
-
1106
- startX = e.touches[0].clientX;
1107
- startY = e.touches[0].clientY;
1108
- } else if (e.touches.length === 2) {
1109
- // alert('beep')
1110
- const dtheta =
1111
- Math.atan2(startY - altY, startX - altX) -
1112
- Math.atan2(
1113
- e.touches[0].clientY - e.touches[1].clientY,
1114
- e.touches[0].clientX - e.touches[1].clientX,
1115
- );
1116
- const dscale =
1117
- Math.hypot(startX - altX, startY - altY) /
1118
- Math.hypot(
1119
- e.touches[0].clientX - e.touches[1].clientX,
1120
- e.touches[0].clientY - e.touches[1].clientY,
1121
- );
1122
- const dx =
1123
- (e.touches[0].clientX +
1124
- e.touches[1].clientX -
1125
- (startX + altX)) /
1126
- 2;
1127
- const dy =
1128
- (e.touches[0].clientY +
1129
- e.touches[1].clientY -
1130
- (startY + altY)) /
1131
- 2;
1132
- let inv = invert4(viewMatrix);
1133
- // inv = translate4(inv, 0, 0, d);
1134
- inv = rotate4(inv, dtheta, 0, 0, 1);
1135
-
1136
- inv = translate4(inv, -dx / innerWidth, -dy / innerHeight, 0);
1137
-
1138
- // let preY = inv[13];
1139
- inv = translate4(inv, 0, 0, 3 * (1 - dscale));
1140
- // inv[13] = preY;
1141
-
1142
- viewMatrix = invert4(inv);
1143
-
1144
- startX = e.touches[0].clientX;
1145
- altX = e.touches[1].clientX;
1146
- startY = e.touches[0].clientY;
1147
- altY = e.touches[1].clientY;
1148
- }
1149
- },
1150
- { passive: false },
1151
- );
1152
- canvas.addEventListener(
1153
- "touchend",
1154
- (e) => {
1155
- e.preventDefault();
1156
- down = false;
1157
- startX = 0;
1158
- startY = 0;
1159
- },
1160
- { passive: false },
1161
- );
1162
-
1163
- let jumpDelta = 0;
1164
- let vertexCount = 0;
1165
-
1166
- let lastFrame = 0;
1167
- let avgFps = 0;
1168
- let start = 0;
1169
-
1170
- window.addEventListener("gamepadconnected", (e) => {
1171
- const gp = navigator.getGamepads()[e.gamepad.index];
1172
- console.log(
1173
- `Gamepad connected at index ${gp.index}: ${gp.id}. It has ${gp.buttons.length} buttons and ${gp.axes.length} axes.`,
1174
- );
1175
- });
1176
- window.addEventListener("gamepaddisconnected", (e) => {
1177
- console.log("Gamepad disconnected");
1178
- });
1179
-
1180
- let leftGamepadTrigger, rightGamepadTrigger;
1181
-
1182
- const frame = (now) => {
1183
- let inv = invert4(viewMatrix);
1184
- let shiftKey =
1185
- activeKeys.includes("Shift") ||
1186
- activeKeys.includes("ShiftLeft") ||
1187
- activeKeys.includes("ShiftRight");
1188
-
1189
- if (activeKeys.includes("ArrowUp")) {
1190
- if (shiftKey) {
1191
- inv = translate4(inv, 0, -0.03, 0);
1192
- } else {
1193
- inv = translate4(inv, 0, 0, 0.1);
1194
- }
1195
- }
1196
- if (activeKeys.includes("ArrowDown")) {
1197
- if (shiftKey) {
1198
- inv = translate4(inv, 0, 0.03, 0);
1199
- } else {
1200
- inv = translate4(inv, 0, 0, -0.1);
1201
- }
1202
- }
1203
- if (activeKeys.includes("ArrowLeft"))
1204
- inv = translate4(inv, -0.03, 0, 0);
1205
- //
1206
- if (activeKeys.includes("ArrowRight"))
1207
- inv = translate4(inv, 0.03, 0, 0);
1208
- // inv = rotate4(inv, 0.01, 0, 1, 0);
1209
- if (activeKeys.includes("KeyA")) inv = rotate4(inv, -0.01, 0, 1, 0);
1210
- if (activeKeys.includes("KeyD")) inv = rotate4(inv, 0.01, 0, 1, 0);
1211
- if (activeKeys.includes("KeyQ")) inv = rotate4(inv, 0.01, 0, 0, 1);
1212
- if (activeKeys.includes("KeyE")) inv = rotate4(inv, -0.01, 0, 0, 1);
1213
- if (activeKeys.includes("KeyW")) inv = rotate4(inv, 0.005, 1, 0, 0);
1214
- if (activeKeys.includes("KeyS")) inv = rotate4(inv, -0.005, 1, 0, 0);
1215
-
1216
- const gamepads = navigator.getGamepads ? navigator.getGamepads() : [];
1217
- let isJumping = activeKeys.includes("Space");
1218
- for (let gamepad of gamepads) {
1219
- if (!gamepad) continue;
1220
-
1221
- const axisThreshold = 0.1; // Threshold to detect when the axis is intentionally moved
1222
- const moveSpeed = 0.06;
1223
- const rotateSpeed = 0.02;
1224
-
1225
- // Assuming the left stick controls translation (axes 0 and 1)
1226
- if (Math.abs(gamepad.axes[0]) > axisThreshold) {
1227
- inv = translate4(inv, moveSpeed * gamepad.axes[0], 0, 0);
1228
- carousel = false;
1229
- }
1230
- if (Math.abs(gamepad.axes[1]) > axisThreshold) {
1231
- inv = translate4(inv, 0, 0, -moveSpeed * gamepad.axes[1]);
1232
- carousel = false;
1233
- }
1234
- if (gamepad.buttons[12].pressed || gamepad.buttons[13].pressed) {
1235
- inv = translate4(
1236
- inv,
1237
- 0,
1238
- -moveSpeed *
1239
- (gamepad.buttons[12].pressed -
1240
- gamepad.buttons[13].pressed),
1241
- 0,
1242
- );
1243
- carousel = false;
1244
- }
1245
-
1246
- if (gamepad.buttons[14].pressed || gamepad.buttons[15].pressed) {
1247
- inv = translate4(
1248
- inv,
1249
- -moveSpeed *
1250
- (gamepad.buttons[14].pressed -
1251
- gamepad.buttons[15].pressed),
1252
- 0,
1253
- 0,
1254
- );
1255
- carousel = false;
1256
- }
1257
-
1258
- // Assuming the right stick controls rotation (axes 2 and 3)
1259
- if (Math.abs(gamepad.axes[2]) > axisThreshold) {
1260
- inv = rotate4(inv, rotateSpeed * gamepad.axes[2], 0, 1, 0);
1261
- carousel = false;
1262
- }
1263
- if (Math.abs(gamepad.axes[3]) > axisThreshold) {
1264
- inv = rotate4(inv, -rotateSpeed * gamepad.axes[3], 1, 0, 0);
1265
- carousel = false;
1266
- }
1267
-
1268
- let tiltAxis = gamepad.buttons[6].value - gamepad.buttons[7].value;
1269
- if (Math.abs(tiltAxis) > axisThreshold) {
1270
- inv = rotate4(inv, rotateSpeed * tiltAxis, 0, 0, 1);
1271
- carousel = false;
1272
- }
1273
- if (gamepad.buttons[4].pressed && !leftGamepadTrigger) {
1274
- camera =
1275
- cameras[(cameras.indexOf(camera) + 1) % cameras.length];
1276
- inv = invert4(getViewMatrix(camera));
1277
- carousel = false;
1278
- }
1279
- if (gamepad.buttons[5].pressed && !rightGamepadTrigger) {
1280
- camera =
1281
- cameras[
1282
- (cameras.indexOf(camera) + cameras.length - 1) %
1283
- cameras.length
1284
- ];
1285
- inv = invert4(getViewMatrix(camera));
1286
- carousel = false;
1287
- }
1288
- leftGamepadTrigger = gamepad.buttons[4].pressed;
1289
- rightGamepadTrigger = gamepad.buttons[5].pressed;
1290
- if (gamepad.buttons[0].pressed) {
1291
- isJumping = true;
1292
- carousel = false;
1293
- }
1294
- if (gamepad.buttons[3].pressed) {
1295
- carousel = true;
1296
- }
1297
- }
1298
-
1299
- if (
1300
- ["KeyJ", "KeyK", "KeyL", "KeyI"].some((k) => activeKeys.includes(k))
1301
- ) {
1302
- let d = 4;
1303
- inv = translate4(inv, 0, 0, d);
1304
- inv = rotate4(
1305
- inv,
1306
- activeKeys.includes("KeyJ")
1307
- ? -0.05
1308
- : activeKeys.includes("KeyL")
1309
- ? 0.05
1310
- : 0,
1311
- 0,
1312
- 1,
1313
- 0,
1314
- );
1315
- inv = rotate4(
1316
- inv,
1317
- activeKeys.includes("KeyI")
1318
- ? 0.05
1319
- : activeKeys.includes("KeyK")
1320
- ? -0.05
1321
- : 0,
1322
- 1,
1323
- 0,
1324
- 0,
1325
- );
1326
- inv = translate4(inv, 0, 0, -d);
1327
- }
1328
-
1329
- viewMatrix = invert4(inv);
1330
-
1331
- if (carousel) {
1332
- let inv = invert4(defaultViewMatrix);
1333
-
1334
- const t = Math.sin((Date.now() - start) / 5000);
1335
- inv = translate4(inv, 2.5 * t, 0, 6 * (1 - Math.cos(t)));
1336
- inv = rotate4(inv, -0.6 * t, 0, 1, 0);
1337
-
1338
- viewMatrix = invert4(inv);
1339
- }
1340
-
1341
- if (isJumping) {
1342
- jumpDelta = Math.min(1, jumpDelta + 0.05);
1343
- } else {
1344
- jumpDelta = Math.max(0, jumpDelta - 0.05);
1345
- }
1346
-
1347
- let inv2 = invert4(viewMatrix);
1348
- inv2 = translate4(inv2, 0, -jumpDelta, 0);
1349
- inv2 = rotate4(inv2, -0.1 * jumpDelta, 1, 0, 0);
1350
- let actualViewMatrix = invert4(inv2);
1351
-
1352
- const viewProj = multiply4(projectionMatrix, actualViewMatrix);
1353
- worker.postMessage({ view: viewProj });
1354
-
1355
- const currentFps = 1000 / (now - lastFrame) || 0;
1356
- avgFps = avgFps * 0.9 + currentFps * 0.1;
1357
-
1358
- if (vertexCount > 0) {
1359
- document.getElementById("spinner").style.display = "none";
1360
- gl.uniformMatrix4fv(u_view, false, actualViewMatrix);
1361
- gl.clear(gl.COLOR_BUFFER_BIT);
1362
- gl.drawArraysInstanced(gl.TRIANGLE_FAN, 0, 4, vertexCount);
1363
- } else {
1364
- gl.clear(gl.COLOR_BUFFER_BIT);
1365
- document.getElementById("spinner").style.display = "";
1366
- start = Date.now() + 2000;
1367
- }
1368
- const progress = (100 * vertexCount) / (splatData.length / rowLength);
1369
- if (progress < 100) {
1370
- document.getElementById("progress").style.width = progress + "%";
1371
- } else {
1372
- document.getElementById("progress").style.display = "none";
1373
- }
1374
- fps.innerText = Math.round(avgFps) + " fps";
1375
- if (isNaN(currentCameraIndex)) {
1376
- camid.innerText = "";
1377
- }
1378
- lastFrame = now;
1379
- requestAnimationFrame(frame);
1380
- };
1381
-
1382
- frame();
1383
-
1384
- const isPly = (splatData) =>
1385
- splatData[0] == 112 &&
1386
- splatData[1] == 108 &&
1387
- splatData[2] == 121 &&
1388
- splatData[3] == 10;
1389
-
1390
- const selectFile = (file) => {
1391
- const fr = new FileReader();
1392
- if (/\.json$/i.test(file.name)) {
1393
- fr.onload = () => {
1394
- cameras = JSON.parse(fr.result);
1395
- viewMatrix = getViewMatrix(cameras[0]);
1396
- projectionMatrix = getProjectionMatrix(
1397
- camera.fx / downsample,
1398
- camera.fy / downsample,
1399
- canvas.width,
1400
- canvas.height,
1401
- );
1402
- gl.uniformMatrix4fv(u_projection, false, projectionMatrix);
1403
-
1404
- console.log("Loaded Cameras");
1405
- };
1406
- fr.readAsText(file);
1407
- } else {
1408
- stopLoading = true;
1409
- fr.onload = () => {
1410
- splatData = new Uint8Array(fr.result);
1411
- console.log("Loaded", Math.floor(splatData.length / rowLength));
1412
-
1413
- if (isPly(splatData)) {
1414
- // ply file magic header means it should be handled differently
1415
- worker.postMessage({ ply: splatData.buffer, save: true });
1416
- } else {
1417
- worker.postMessage({
1418
- buffer: splatData.buffer,
1419
- vertexCount: Math.floor(splatData.length / rowLength),
1420
- });
1421
- }
1422
- };
1423
- fr.readAsArrayBuffer(file);
1424
- }
1425
- };
1426
-
1427
- window.addEventListener("hashchange", (e) => {
1428
- try {
1429
- viewMatrix = JSON.parse(decodeURIComponent(location.hash.slice(1)));
1430
- carousel = false;
1431
- } catch (err) {}
1432
- });
1433
-
1434
- const preventDefault = (e) => {
1435
- e.preventDefault();
1436
- e.stopPropagation();
1437
- };
1438
- document.addEventListener("dragenter", preventDefault);
1439
- document.addEventListener("dragover", preventDefault);
1440
- document.addEventListener("dragleave", preventDefault);
1441
- document.addEventListener("drop", (e) => {
1442
- e.preventDefault();
1443
- e.stopPropagation();
1444
- selectFile(e.dataTransfer.files[0]);
1445
- });
1446
-
1447
- let bytesRead = 0;
1448
- let lastVertexCount = -1;
1449
- let stopLoading = false;
1450
-
1451
- while (true) {
1452
- const { done, value } = await reader.read();
1453
- if (done || stopLoading) break;
1454
-
1455
- splatData.set(value, bytesRead);
1456
- bytesRead += value.length;
1457
-
1458
- if (vertexCount > lastVertexCount) {
1459
- if (!isPly(splatData)) {
1460
- worker.postMessage({
1461
- buffer: splatData.buffer,
1462
- vertexCount: Math.floor(bytesRead / rowLength),
1463
- });
1464
- }
1465
- lastVertexCount = vertexCount;
1466
- }
1467
- }
1468
- if (!stopLoading) {
1469
- if (isPly(splatData)) {
1470
- // ply file magic header means it should be handled differently
1471
- worker.postMessage({ ply: splatData.buffer, save: false });
1472
- } else {
1473
- worker.postMessage({
1474
- buffer: splatData.buffer,
1475
- vertexCount: Math.floor(bytesRead / rowLength),
1476
- });
1477
- }
1478
- }
1479
- }
1480
-
1481
- main().catch((err) => {
1482
- document.getElementById("spinner").style.display = "none";
1483
- document.getElementById("message").innerText = err.toString();
1484
- });