SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

row-major matmul optimization

An innovative library for efficient LLM inference via low-bit quantization

Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). Five-model zoo, custom int8 kernels, no ML framework, no Python, no GPU.

258 KB WASM runtime for Needle a 26M-parameter tool-calling transformer. Runs in browser, Cloudflare Workers, and Node.js. No backend required.

🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-oriented durability with HNSW + DiskANN indexing for best-in-class performance.

⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.

Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.

rust library to write integer types of any bit length into a buffer - from `i1` to `i64`.

VKNN — Vulkan Neural Network: dependency-free C++17 on-device inference engine that runs ONNX models entirely on Android GPUs. CNNs, YOLO, LLMs, VLMs, 3D Gaussian Splatting; int4 quantization; compile once to .vxm.

Three ComfyUI nodes that show the real runtime precision of MODEL, CLIP, and VAE objects.

Post-training LLM weight compression for low-RAM machines: Q4/Q8 quantization, green-format repair, AVX2 CPU inference, optional CUDA. ~45% less RAM at ~99.9% quality.

Backprop-free learning study: spiking (LIF) neurons + Forward-Forward + JEPA + int4 QAT, with a full ablation notebook.

PyTorch implementation of TRIAD-PTQ (Trace-Router-Interaction-Aware Decomposition) — weight-only INT3/INT4 PTQ for compact LLMs and edge CNNs/ViTs, with real benchmarks on SmolLM/TinyLlama/MobileNetV2/EfficientNet-B0/MobileViT-S.

纯 Python 大模型高效推理参考工具包：INT8/INT4 训练后量化 + 分页 KV-cache + 投机解码，离线可跑

Quantize TinyLlama-1.1B-Chat from PyTorch to CoreML (float16, int8, int4) for efficient on-device inference on iOS 18+.

Benchmarks five KV cache quantization schemes (FP16→INT4) on GPT-2 with per-layer sensitivity analysis. Measures memory reduction vs perplexity tradeoff across sequence lengths without modifying model weights.

Utilities to rewrite ONNX convolution patterns into MatMul forms for optimal LLM-like int4 quantization (esp. Audio/Speech models).

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, deployed and benchmarked. Generates in 16 GB of RAM.