[SIGCOMM 2023] Lightning: A Reconfigurable Photonic-Electronic SmartNIC for Fast and Energy-Efficient Inference
-
Updated
Nov 17, 2023 - Verilog
#
inference-serving
Here are 17 public repositories matching this topic...
[Long Term Support] [SIGCOMM 2023] Lightning: A Reconfigurable Photonic-Electronic SmartNIC for Fast and Energy-Efficient Inference
verilog fpga-soc smartnic machine-learning-systems photonic-computing rfsoc inference-serving in-network-computing
-
Updated
Sep 20, 2024 - Verilog
The official repo for the paper "Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching"
-
Updated
Mar 17, 2025
Official repo for the ACSOS 2021 paper on how to manage many deep learning models at the edge!
-
Updated
Sep 29, 2021 - Python
NexusKV is a model-state intelligence layer for inference systems.
python rust golang distributed-systems inference radix-tree model-state kv-cache inference-serving ai-infrastructure state-reuse
-
Updated
Aug 21, 2026 - Python
Locus is an engine-neutral inference control plane for compute and model-state placement.
distributed-systems scheduling inference orchestration control-plane load-balancing model-serving inference-serving ai-infrastructure workload-placement state-aware-scheduling
-
Updated
Aug 22, 2026 - Rust
Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.
python benchmark machine-learning real-time latency websocket load-testing inference regression-testing performance-testing model-serving quality-gates mlops fastapi inference-serving llm-inference adaptive-batching
-
Updated
May 20, 2026 - Python
Reactive LLM inference.
inference incremental-computation inference-optimization kv-cache inference-serving llm llm-inference kvcache-reuse recurrent-memory
-
Updated
Jul 7, 2026 - Python
Reproducible Apple-Silicon (Apple MLX) deployment and one-token verification kit for the FreeToken edge-native MoE serving API — pinned, loopback-only, schema-validated evidence.
macos benchmark reproducibility mlx mixture-of-experts research-reproducibility apple-silicon inference-serving llm-inference freetoken
-
Updated
Aug 23, 2026 - Python
GPU-aware LLM serving runtime with async scheduling, dynamic batching, streaming, admission control, metrics, and mock/llama.cpp/vLLM backends.
-
Updated
Jul 21, 2026 - Python
Machine-readable companion to the IEEE OJ-CS survey 'Semantic Caching and Response Reuse for Large Language Model Services: A Survey' (Chukkapalli, Mishra, Naik, 2026): 21-work evidence matrix, systematic-search log, proposed benchmark trace schema, stdlib-only contract validator, and CPU pilot. Code MIT; data CC-BY-4.0.
-
Updated
Jul 11, 2026 - Python
GSM artifact for exact online GNN inference serving with degree-based scheduling and memory-aware batch division.
scheduling cuda memory-management graph-neural-networks gnn dgl inference-serving online-inference graph-inference
-
Updated
Jun 15, 2026 - Python
Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.
go kubernetes gpu prometheus nvidia cloud-native sre autoscaling observability gpu-monitoring finops gpu-utilization platform-engineering mlops kv-cache rke2 inference-serving vllm llm-inference dcgm
-
Updated
Aug 20, 2026 - Go
Put AI compute in the 82 million houses that already have a grid connection. An exact-arithmetic feasibility study of residential AI inference as an alternative to the data center buildout.
energy distributed-computing data-center edge-computing demand-response techno-economic-analysis feasibility-study waste-heat-recovery inference-serving exact-arithmetic llm-inference decentralized-ai small-language-models ai-infrastructure ai-energy gpu-economics grid-interconnection residential-compute
-
Updated
Aug 24, 2026 - HTML
cost-aware-inference-cluster: FastAPI inference-serving prototype with Redis queues, worker heartbeats, dynamic batching, autoscaling simulation, and local Docker Compose benchmark evidence.
docker kubernetes redis grafana prometheus autoscaling ml-infrastructure fastapi dynamic-batching inference-serving
-
Updated
Jun 29, 2026 - Python
Inference gateway for batching AI model requests through bounded micro-batching, queue backpressure, per-request result routing, and reproducible performance evaluation.
nodejs performance-engineering benchmarking typescript grpc inference systems-engineering backpressure model-serving ai-engineering inference-serving ai-infrastructure micro-batching
-
Updated
Aug 4, 2026 - TypeScript
Improve this page
Add a description, image, and links to the inference-serving topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the inference-serving topic, visit your repo's landing page and select "manage topics."