📝 Abstract:
Large Language Models (LLMs) achieve strong performance in many natural lan- guage processing tasks, but their high computational and memory requirements make hardware deployment challenging. Quantization reduces these costs by using low- precision weights and activations, yet practical accelerator design also requires mod- eling integer arithmetic, accumulator precision, re-scaling, quantization-domain tran- sitions, and data movement. This thesis presents a Python-based hardware-oriented modeling framework for quantized LLM inference based on SmoothQuant-style post- training quantization. The framework models key decoder stages, including Normal- ization layers, projection layers, attention matrix multiplications, MLP operations, element-wise operations, INT8 operands, INT32 accumulation, and output re-scaling. It is integrated into a Neural Compressor-based evaluation flow and evaluated on six model families (OPT, LLaMA, Qwen, StableLM, Gemma, and Mistral) using LAM- BADA, Winogrande, PIQA, and WikiText. The evaluations examine the effects of quantization on predictive quality, while integer-domain sparsity is analyzed to iden- tify opportunities for future sparse-aware hardware design.