Skip to content
BranchesTags

Repository files navigation

Smart Clipper

A Chrome extension for seamlessly transferring math problems (text, LaTeX formulas, and images) to Google Gemini.

Features

✨ Multi-Modal Capture

  • Text: Plain text with preserved formatting
  • LaTeX Formulas: Auto-detected and properly formatted
  • Images: Diagrams, graphs, and geometric figures

🎯 Smart Detection

  • Automatically detects mathematical content
  • Recognizes LaTeX in multiple formats ($...$, $$...$$, [...], etc.)
  • Extracts formulas from MathJax and KaTeX rendered content
  • Classifies images (geometry, graphs, formulas)

🚀 One-Click Transfer

  • Right-click context menu
  • Keyboard shortcut: Ctrl+Shift+G (or Cmd+Shift+G on Mac)
  • Preview before sending
  • Automatic tab management

🧠 Context Preservation

  • Maintains spatial relationships between text, formulas, and images
  • Preserves document structure
  • Extracts surrounding context for images

Installation

From Source

  1. Clone or download this repository
  2. Open Chrome and navigate to chrome://extensions/
  3. Enable "Developer mode" in the top right
  4. Click "Load unpacked"
  5. Select the smart-clipper directory
  6. The extension icon should appear in your toolbar

Icon Setup

The extension requires icon files. You can either:

  • Use the provided icon generation script (coming soon)
  • Create your own PNG icons in the following sizes and place them in assets/icons/:
    • icon16.png (16x16)
    • icon32.png (32x32)
    • icon48.png (48x48)
    • icon128.png (128x128)

For now, you can use any placeholder images with these dimensions.

Usage

Quick Start

  1. Select content on any webpage (text, formulas, images)
  2. Right-click and choose "Send to Gemini" OR press Ctrl+Shift+G
  3. Content automatically transfers to Gemini chat
  4. Review and submit in Gemini

Workflows

Smart Selection (Recommended)

1. Select entire problem area with mouse
2. Right-click → "Send to Gemini" OR Ctrl+Shift+G
3. Preview popup appears (optional)
4. Click "Send" → Content transfers to Gemini

Context Menu

1. Select content
2. Right-click
3. Choose "Send to Gemini" for instant transfer
   OR "Capture and Preview" to review first

Keyboard Shortcut

1. Select content
2. Press Ctrl+Shift+G (Cmd+Shift+G on Mac)
3. Content captures and preview appears

How It Works

Architecture

┌─────────────────────────────────────────────────┐
│           Content Script (Any Page)             │
│  • Listens for selections                       │
│  • Captures text, LaTeX, images                 │
│  • Shows preview popup                          │
└──────────────────┬──────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────┐
│          Background Service Worker              │
│  • Coordinates between tabs                     │
│  • Manages context menus                        │
│  • Handles keyboard shortcuts                   │
└──────────────────┬──────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────┐
│       Gemini Injector (Gemini Tab Only)         │
│  • Inserts text into chat input                 │
│  • Uploads images                               │
│  • Handles multi-part messages                  │
└─────────────────────────────────────────────────┘

Components

  • LaTeX Detector (lib/latex-detector.js): Detects and extracts LaTeX formulas
  • Image Extractor (lib/image-extractor.js): Extracts and processes images
  • Content Assembler (lib/content-assembler.js): Combines all content types
  • Content Script (content/content-script.js): Handles user interaction
  • Gemini Injector (content/gemini-injector.js): Inserts content into Gemini
  • Service Worker (background/service-worker.js): Coordinates everything

Examples

Example 1: Calculus Problem

Selected Content:

Find the area under the curve f(x) = x² from x=0 to x=2.

∫₀² x² dx

Sent to Gemini:

Find the area under the curve f(x) = x² from x=0 to x=2.

$$\int_0^2 x^2 dx$$

Example 2: Geometry Problem

Selected Content:

  • Text: "In triangle ABC: Angle A = 45°, Side AB = 10 cm, Side BC = 8 cm"
  • Image: Triangle diagram
  • Text: "Find the area of triangle ABC."

Sent to Gemini:

  • All text preserved in order
  • Triangle diagram attached as image
  • Maintains problem structure

Example 3: Complex Formula

Selected Content:

The quadratic formula: x = (-b ± √(b²-4ac)) / 2a

Sent to Gemini:

The quadratic formula: $$x = \frac{-b \pm \sqrt{b^2-4ac}}{2a}$$

Supported Formats

LaTeX Detection

  • Inline math: $formula$, \(formula\)
  • Display math: $$formula$$, \[formula\]
  • Environments: \begin{equation}...\end{equation}, \begin{align}...\end{align}
  • MathJax/KaTeX: Automatically extracts from rendered elements
  • Unicode math: Converts to LaTeX when appropriate

Image Formats

  • Standard images: <img> tags
  • SVG: Converted to data URL
  • Canvas: Captured as PNG
  • All formats uploaded to Gemini

Browser Compatibility

  • ✅ Chrome (v88+)
  • ✅ Edge (Chromium-based)
  • ⚠️ Other Chromium browsers (should work, not tested)
  • ❌ Firefox (requires porting to WebExtensions)
  • ❌ Safari (requires porting)

Development

Project Structure

smart-clipper/
├── manifest.json              # Extension configuration
├── background/
│   └── service-worker.js     # Background processing
├── content/
│   ├── content-script.js     # Page interaction
│   ├── content-style.css     # Styles for injected UI
│   └── gemini-injector.js    # Gemini-specific handler
├── lib/
│   ├── latex-detector.js     # LaTeX detection
│   ├── image-extractor.js    # Image processing
│   └── content-assembler.js  # Content assembly
├── popup/
│   ├── popup.html            # Extension popup
│   ├── popup.css             # Popup styles
│   └── popup.js              # Popup logic
├── assets/
│   └── icons/                # Extension icons
└── README.md

Building from Source

  1. Clone the repository:
git clone <repository-url>
cd smart-clipper

  1. Install dependencies (none currently - pure vanilla JS)

  2. Load in Chrome:

    • Navigate to chrome://extensions/
    • Enable "Developer mode"
    • Click "Load unpacked"
    • Select the project directory

Testing

  1. Test LaTeX Detection:

    • Visit any page with math formulas (Khan Academy, Wikipedia math articles)
    • Select formulas and test capture

  2. Test Image Capture:

    • Select content with embedded images
    • Verify images transfer correctly

  3. Test Gemini Integration:

    • Ensure you're logged into Gemini
    • Test transfer to chat input
    • Verify images upload correctly

Known Limitations

Current Version (v0.1.0)

  • ✅ Basic multi-modal capture
  • ✅ LaTeX detection and formatting
  • ✅ Image extraction
  • ✅ Gemini integration
  • ⚠️ Gemini UI selectors may need updates (Gemini is in active development)
  • ⚠️ Complex MathML conversion is simplified
  • ⚠️ No OCR for handwritten formulas (planned for Phase 5)
  • ⚠️ No edit capability yet (preview only)
  • ⚠️ No capture history

Planned Features

Phase 2 (v0.2.0):

  • Enhanced LaTeX detection patterns
  • Better image classification
  • Improved error handling

Phase 3 (v0.3.0):

  • Progressive capture mode
  • Multi-part problem assembly
  • Drag-to-reorder interface

Phase 4 (v0.4.0):

  • Clipboard monitor
  • Capture history
  • Edit before send
  • Template system

Phase 5 (v1.0.0):

  • OCR integration
  • Handwritten formula recognition
  • Advanced formatting options
  • Cloud sync

Troubleshooting

Content not capturing

  1. Make sure you have content selected
  2. Check browser console for errors (F12)
  3. Try reloading the page
  4. Verify extension is enabled

Not sending to Gemini

  1. Make sure you're logged into Gemini
  2. Check if Gemini tab opened correctly
  3. Try manually opening Gemini first
  4. Check for Gemini UI updates (selectors may need adjustment)

Images not uploading

  1. Verify images are within size limits
  2. Check browser console for errors
  3. Try with simpler images first
  4. Ensure Gemini supports image uploads

LaTeX not detected

  1. Check if formulas use supported delimiters
  2. Try wrapping in $...$ or $$...$$
  3. Report patterns that should be detected

Contributing

Contributions are welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Test thoroughly
  5. Submit a pull request

Privacy

Smart Clipper:

  • ✅ Processes all content locally
  • ✅ Only sends to Gemini when you explicitly request
  • ✅ No external servers or tracking
  • ✅ No data collection
  • ✅ Open source - audit the code yourself

License

MIT License - See LICENSE file for details

Support

  • Issues: Report bugs and request features via GitHub Issues
  • Documentation: See this README and code comments
  • Updates: Watch the repository for updates

Changelog

v0.1.0 (Current)

  • Initial release
  • Multi-modal capture (text, LaTeX, images)
  • Smart LaTeX detection
  • Image extraction with context
  • Gemini integration
  • Preview popup
  • Context menu and keyboard shortcuts

Acknowledgments

Built for enhanced learning with Google Gemini Guided Learning.

Special thanks to:

  • MathJax and KaTeX for LaTeX rendering inspiration
  • Chrome Extensions documentation
  • The mathematical education community

Happy Learning with Smart Clipper! 📐✨

About

Math Problem Clipper for Gemini Guided Learning

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages