Skip to content

CUDA: reduce MMQ stream-k overhead#22298

Merged
JohannesGaessler merged 2 commits intoggml-org:masterfrom
JohannesGaessler:cuda-mmq-fastdiv-8
Apr 25, 2026
Merged

CUDA: reduce MMQ stream-k overhead#22298
JohannesGaessler merged 2 commits intoggml-org:masterfrom
JohannesGaessler:cuda-mmq-fastdiv-8

Conversation

@JohannesGaessler
Copy link
Copy Markdown
Contributor

@JohannesGaessler JohannesGaessler commented Apr 23, 2026

This PR reduces the stream-k overhead in the MMQ kernel by using fastdiv which precomputes some values on the CPU to speed up integer divisions. Also, as originally suggested by @nisparks in #22170 and #22252 optionally use tiling rather than a stream-k decomposition. The implementation in this PR is different vs the ones linked: in those an extra variant of the kernel is being compiled that has the tiling hard-coded (as is done for relatively old GPUs), in this PR the number of CUDA blocks is scaled dynamically to the number of tiles so that each CUDA block works on exactly one tile; if it turns out that there is a meaningful performance difference it may make sense to still compile the extra kernels. The choice for whether or not to use stream-k does not explicitly depend on MoE in this PR, instead it is determined from the efficiency loss that would be incurred by tiling: if it is <= 10% tiling is used in order to skip the stream-k fixup.

Requirements

@JohannesGaessler JohannesGaessler requested a review from a team as a code owner April 23, 2026 19:13
@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 23, 2026

Quantization sweep NVIDIA dense
GPU Model Microbatch size Test t/s master t/s PR Speedup
P40 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 404.24 399.59 0.99
P40 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 552.73 547.70 0.99
P40 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 660.64 653.85 0.99
P40 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 747.62 743.90 1.00
P40 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 808.94 806.75 1.00
P40 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 829.51 827.94 1.00
P40 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 844.92 844.08 1.00
P40 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 850.44 844.68 0.99
P40 llama 8B IQ2_S - 2.5 bpw 16 pp2048 361.25 361.54 1.00
P40 llama 8B IQ2_S - 2.5 bpw 32 pp2048 477.63 471.11 0.99
P40 llama 8B IQ2_S - 2.5 bpw 64 pp2048 626.28 626.15 1.00
P40 llama 8B IQ2_S - 2.5 bpw 128 pp2048 708.41 707.07 1.00
P40 llama 8B IQ2_S - 2.5 bpw 256 pp2048 772.13 768.61 1.00
P40 llama 8B IQ2_S - 2.5 bpw 512 pp2048 793.41 789.35 0.99
P40 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 806.71 806.09 1.00
P40 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 805.31 805.45 1.00
P40 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 361.45 361.86 1.00
P40 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 465.81 460.18 0.99
P40 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 636.00 631.91 0.99
P40 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 712.06 710.88 1.00
P40 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 774.47 770.04 0.99
P40 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 796.21 792.90 1.00
P40 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 810.60 808.32 1.00
P40 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 816.24 813.00 1.00
P40 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 399.42 399.65 1.00
P40 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 551.85 545.93 0.99
P40 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 668.51 646.09 0.97
P40 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 756.31 740.35 0.98
P40 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 819.53 802.12 0.98
P40 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 843.83 826.40 0.98
P40 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 861.67 844.28 0.98
P40 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 862.49 848.36 0.98
P40 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 344.33 342.82 1.00
P40 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 517.38 516.96 1.00
P40 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 597.19 613.56 1.03
P40 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 693.58 707.21 1.02
P40 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 751.59 768.95 1.02
P40 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 774.66 792.74 1.02
P40 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 792.89 808.95 1.02
P40 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 796.10 811.23 1.02
P40 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 349.12 346.77 0.99
P40 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 505.61 505.82 1.00
P40 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 606.34 619.16 1.02
P40 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 701.44 711.20 1.01
P40 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 757.26 774.08 1.02
P40 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 778.09 793.16 1.02
P40 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 799.76 807.82 1.01
P40 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 802.06 815.64 1.02
P40 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 360.16 360.97 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 516.42 516.16 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 608.04 609.34 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 704.46 707.13 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 767.97 768.96 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 791.81 792.38 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 808.72 811.27 1.00
P40 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 808.16 815.66 1.01
P40 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 372.62 375.18 1.01
P40 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 511.36 509.05 1.00
P40 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 625.40 621.18 0.99
P40 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 716.88 712.74 0.99
P40 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 782.36 777.15 0.99
P40 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 804.13 800.31 1.00
P40 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 820.07 815.07 0.99
P40 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 821.17 819.92 1.00
P40 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 402.09 407.82 1.01
P40 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 570.81 571.31 1.00
P40 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 688.79 679.60 0.99
P40 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 772.13 766.05 0.99
P40 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 835.85 834.91 1.00
P40 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 859.77 856.55 1.00
P40 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 875.56 874.42 1.00
P40 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 880.06 877.65 1.00
P40 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 409.21 407.84 1.00
P40 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 565.91 571.95 1.01
P40 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 685.34 694.02 1.01
P40 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 770.15 774.19 1.01
P40 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 836.59 839.60 1.00
P40 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 857.70 859.93 1.00
P40 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 874.56 877.43 1.00
P40 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 879.11 881.37 1.00
P40 llama 8B Q2_K_M 16 pp2048 339.23 334.54 0.99
P40 llama 8B Q2_K_M 32 pp2048 463.14 466.49 1.01
P40 llama 8B Q2_K_M 64 pp2048 604.13 602.04 1.00
P40 llama 8B Q2_K_M 128 pp2048 670.36 674.66 1.01
P40 llama 8B Q2_K_M 256 pp2048 723.95 726.21 1.00
P40 llama 8B Q2_K_M 512 pp2048 743.76 746.11 1.00
P40 llama 8B Q2_K_M 1024 pp2048 761.45 763.38 1.00
P40 llama 8B Q2_K_M 2048 pp2048 770.06 768.56 1.00
P40 llama 8B Q3_K_S 16 pp2048 352.19 352.71 1.00
P40 llama 8B Q3_K_S 32 pp2048 444.58 447.92 1.01
P40 llama 8B Q3_K_S 64 pp2048 589.07 586.72 1.00
P40 llama 8B Q3_K_S 128 pp2048 644.88 639.63 0.99
P40 llama 8B Q3_K_S 256 pp2048 693.77 691.08 1.00
P40 llama 8B Q3_K_S 512 pp2048 710.12 706.34 0.99
P40 llama 8B Q3_K_S 1024 pp2048 722.38 719.07 1.00
P40 llama 8B Q3_K_S 2048 pp2048 725.76 721.46 0.99
P40 llama 8B Q4_0 16 pp2048 458.91 458.64 1.00
P40 llama 8B Q4_0 32 pp2048 547.44 555.68 1.02
P40 llama 8B Q4_0 64 pp2048 730.69 735.91 1.01
P40 llama 8B Q4_0 128 pp2048 830.58 845.45 1.02
P40 llama 8B Q4_0 256 pp2048 891.73 924.57 1.04
P40 llama 8B Q4_0 512 pp2048 913.62 944.30 1.03
P40 llama 8B Q4_0 1024 pp2048 914.23 957.12 1.05
P40 llama 8B Q4_0 2048 pp2048 914.45 951.30 1.04
P40 llama 8B Q4_1 16 pp2048 458.33 457.13 1.00
P40 llama 8B Q4_1 32 pp2048 544.57 548.72 1.01
P40 llama 8B Q4_1 64 pp2048 708.30 714.91 1.01
P40 llama 8B Q4_1 128 pp2048 790.75 800.94 1.01
P40 llama 8B Q4_1 256 pp2048 840.41 860.55 1.02
P40 llama 8B Q4_1 512 pp2048 859.02 877.17 1.02
P40 llama 8B Q4_1 1024 pp2048 874.41 888.05 1.02
P40 llama 8B Q4_1 2048 pp2048 879.25 889.69 1.01
P40 llama 8B Q4_K_S 16 pp2048 414.42 412.31 0.99
P40 llama 8B Q4_K_S 32 pp2048 508.12 506.61 1.00
P40 llama 8B Q4_K_S 64 pp2048 645.28 646.36 1.00
P40 llama 8B Q4_K_S 128 pp2048 723.84 724.17 1.00
P40 llama 8B Q4_K_S 256 pp2048 780.12 780.95 1.00
P40 llama 8B Q4_K_S 512 pp2048 801.78 801.48 1.00
P40 llama 8B Q4_K_S 1024 pp2048 818.96 820.79 1.00
P40 llama 8B Q4_K_S 2048 pp2048 823.06 821.50 1.00
P40 llama 8B Q5_0 16 pp2048 365.53 369.93 1.01
P40 llama 8B Q5_0 32 pp2048 515.73 523.52 1.02
P40 llama 8B Q5_0 64 pp2048 650.81 658.04 1.01
P40 llama 8B Q5_0 128 pp2048 728.20 731.57 1.00
P40 llama 8B Q5_0 256 pp2048 787.00 792.21 1.01
P40 llama 8B Q5_0 512 pp2048 805.82 807.87 1.00
P40 llama 8B Q5_0 1024 pp2048 819.24 826.27 1.01
P40 llama 8B Q5_0 2048 pp2048 813.56 827.30 1.02
P40 llama 8B Q5_1 16 pp2048 387.86 394.06 1.02
P40 llama 8B Q5_1 32 pp2048 530.20 535.54 1.01
P40 llama 8B Q5_1 64 pp2048 648.73 657.30 1.01
P40 llama 8B Q5_1 128 pp2048 721.42 729.30 1.01
P40 llama 8B Q5_1 256 pp2048 777.38 785.58 1.01
P40 llama 8B Q5_1 512 pp2048 795.06 803.11 1.01
P40 llama 8B Q5_1 1024 pp2048 809.89 818.84 1.01
P40 llama 8B Q5_1 2048 pp2048 812.93 818.49 1.01
P40 llama 8B Q5_K_S 16 pp2048 313.65 316.37 1.01
P40 llama 8B Q5_K_S 32 pp2048 462.86 454.51 0.98
P40 llama 8B Q5_K_S 64 pp2048 632.20 633.03 1.00
P40 llama 8B Q5_K_S 128 pp2048 693.03 693.56 1.00
P40 llama 8B Q5_K_S 256 pp2048 742.98 740.83 1.00
P40 llama 8B Q5_K_S 512 pp2048 763.67 759.60 0.99
P40 llama 8B Q5_K_S 1024 pp2048 780.00 771.32 0.99
P40 llama 8B Q5_K_S 2048 pp2048 781.57 780.96 1.00
P40 llama 8B Q6_K 16 pp2048 305.54 350.07 1.15
P40 llama 8B Q6_K 32 pp2048 452.99 452.73 1.00
P40 llama 8B Q6_K 64 pp2048 594.22 592.97 1.00
P40 llama 8B Q6_K 128 pp2048 662.63 660.73 1.00
P40 llama 8B Q6_K 256 pp2048 707.55 705.12 1.00
P40 llama 8B Q6_K 512 pp2048 723.24 720.28 1.00
P40 llama 8B Q6_K 1024 pp2048 739.63 734.33 0.99
P40 llama 8B Q6_K 2048 pp2048 741.53 739.93 1.00
P40 llama 8B Q8_0 16 pp2048 367.50 363.81 0.99
P40 llama 8B Q8_0 32 pp2048 539.38 545.87 1.01
P40 llama 8B Q8_0 64 pp2048 654.91 647.55 0.99
P40 llama 8B Q8_0 128 pp2048 739.99 738.49 1.00
P40 llama 8B Q8_0 256 pp2048 808.62 802.05 0.99
P40 llama 8B Q8_0 512 pp2048 829.49 826.96 1.00
P40 llama 8B Q8_0 1024 pp2048 847.96 842.70 0.99
P40 llama 8B Q8_0 2048 pp2048 850.64 849.65 1.00
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 1307.91 1350.61 1.03
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 2011.68 2088.65 1.04
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 2804.77 2835.08 1.01
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 3251.26 3266.44 1.00
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 3743.48 3765.01 1.01
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 3969.58 3958.04 1.00
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 3908.04 4074.99 1.04
RTX 3090 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 3868.97 4042.52 1.04
RTX 3090 llama 8B IQ2_S - 2.5 bpw 16 pp2048 1151.64 1178.22 1.02
RTX 3090 llama 8B IQ2_S - 2.5 bpw 32 pp2048 1773.69 1811.11 1.02
RTX 3090 llama 8B IQ2_S - 2.5 bpw 64 pp2048 2569.83 2593.26 1.01
RTX 3090 llama 8B IQ2_S - 2.5 bpw 128 pp2048 3034.05 2995.04 0.99
RTX 3090 llama 8B IQ2_S - 2.5 bpw 256 pp2048 3485.14 3444.16 0.99
RTX 3090 llama 8B IQ2_S - 2.5 bpw 512 pp2048 3646.92 3572.76 0.98
RTX 3090 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 3631.83 3635.32 1.00
RTX 3090 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 3498.78 3521.92 1.01
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 1151.95 1190.00 1.03
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 1763.49 1821.10 1.03
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 2505.11 2554.94 1.02
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 2921.58 2917.34 1.00
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 3336.75 3339.56 1.00
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 3500.59 3543.65 1.01
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 3517.49 3432.58 0.98
RTX 3090 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 3423.30 3582.94 1.05
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 1261.38 1306.68 1.04
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 1963.77 2026.34 1.03
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 2896.94 2982.92 1.03
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 3463.16 3483.68 1.01
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 3977.52 4007.11 1.01
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 4124.43 4178.12 1.01
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 4069.54 4313.63 1.06
RTX 3090 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 4006.18 4208.83 1.05
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 1090.75 1117.97 1.02
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 1740.07 1802.43 1.04
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 2748.48 2787.66 1.01
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 3430.01 3429.76 1.00
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 3988.38 3919.84 0.98
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 4151.09 4038.56 0.97
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 4113.58 4229.56 1.03
RTX 3090 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 3977.56 3971.31 1.00
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 1116.41 1146.12 1.03
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 1780.42 1859.09 1.04
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 2765.18 2819.95 1.02
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 3401.73 3413.23 1.00
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 3974.38 3944.15 0.99
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 4155.10 4135.53 1.00
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 4167.10 4251.37 1.02
RTX 3090 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 3966.58 4123.99 1.04
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 1131.99 1155.11 1.02
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 1807.34 1850.92 1.02
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 2802.10 2832.74 1.01
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 3443.44 3437.88 1.00
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 3993.38 3983.33 1.00
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 4211.32 4161.41 0.99
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 4170.15 4268.43 1.02
RTX 3090 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 3989.53 4085.66 1.02
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 1149.41 1167.74 1.02
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 1835.48 1857.39 1.01
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 2774.94 2804.81 1.01
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 3396.86 3343.79 0.98
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 3953.82 3882.03 0.98
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 4140.96 4083.15 0.99
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 4132.48 4130.07 1.00
RTX 3090 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 3983.29 3969.93 1.00
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 1263.21 1298.36 1.03
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 1988.80 2034.14 1.02
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 3005.28 3084.54 1.03
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 3629.31 3646.70 1.00
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 4250.69 4236.52 1.00
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 4411.98 4480.43 1.02
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 4439.10 4485.33 1.01
RTX 3090 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 4328.91 4433.43 1.02
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 1356.32 1422.10 1.05
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 2079.27 2153.80 1.04
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 3100.17 3174.61 1.02
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 3668.80 3668.12 1.00
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 4260.38 4271.32 1.00
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 4452.67 4457.90 1.00
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 4388.82 4500.34 1.03
RTX 3090 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 4241.74 4364.50 1.03
RTX 3090 llama 8B Q2_K_M 16 pp2048 1233.58 1289.99 1.05
RTX 3090 llama 8B Q2_K_M 32 pp2048 1786.46 1838.88 1.03
RTX 3090 llama 8B Q2_K_M 64 pp2048 2307.65 2372.68 1.03
RTX 3090 llama 8B Q2_K_M 128 pp2048 2414.32 2447.87 1.01
RTX 3090 llama 8B Q2_K_M 256 pp2048 2866.18 3044.79 1.06
RTX 3090 llama 8B Q2_K_M 512 pp2048 3038.78 3178.17 1.05
RTX 3090 llama 8B Q2_K_M 1024 pp2048 3017.48 3233.96 1.07
RTX 3090 llama 8B Q2_K_M 2048 pp2048 3019.22 3184.19 1.05
RTX 3090 llama 8B Q3_K_S 16 pp2048 1193.42 1246.59 1.04
RTX 3090 llama 8B Q3_K_S 32 pp2048 1799.81 1857.70 1.03
RTX 3090 llama 8B Q3_K_S 64 pp2048 2513.12 2572.04 1.02
RTX 3090 llama 8B Q3_K_S 128 pp2048 2930.05 2964.53 1.01
RTX 3090 llama 8B Q3_K_S 256 pp2048 3257.83 3382.06 1.04
RTX 3090 llama 8B Q3_K_S 512 pp2048 3342.75 3454.62 1.03
RTX 3090 llama 8B Q3_K_S 1024 pp2048 3342.98 3495.31 1.05
RTX 3090 llama 8B Q3_K_S 2048 pp2048 3305.40 3447.90 1.04
RTX 3090 llama 8B Q4_0 16 pp2048 1392.85 1423.18 1.02
RTX 3090 llama 8B Q4_0 32 pp2048 2126.46 2208.99 1.04
RTX 3090 llama 8B Q4_0 64 pp2048 3150.82 3199.91 1.02
RTX 3090 llama 8B Q4_0 128 pp2048 3752.07 3781.14 1.01
RTX 3090 llama 8B Q4_0 256 pp2048 4419.01 4410.34 1.00
RTX 3090 llama 8B Q4_0 512 pp2048 4660.01 4622.81 0.99
RTX 3090 llama 8B Q4_0 1024 pp2048 4783.57 4841.30 1.01
RTX 3090 llama 8B Q4_0 2048 pp2048 4709.87 4832.08 1.03
RTX 3090 llama 8B Q4_1 16 pp2048 1335.75 1379.54 1.03
RTX 3090 llama 8B Q4_1 32 pp2048 2150.39 2252.31 1.05
RTX 3090 llama 8B Q4_1 64 pp2048 2931.35 3010.31 1.03
RTX 3090 llama 8B Q4_1 128 pp2048 3405.12 3456.86 1.02
RTX 3090 llama 8B Q4_1 256 pp2048 3967.56 3977.49 1.00
RTX 3090 llama 8B Q4_1 512 pp2048 4198.34 4205.91 1.00
RTX 3090 llama 8B Q4_1 1024 pp2048 4227.43 4381.36 1.04
RTX 3090 llama 8B Q4_1 2048 pp2048 4131.52 4290.09 1.04
RTX 3090 llama 8B Q4_K_S 16 pp2048 1289.08 1334.11 1.03
RTX 3090 llama 8B Q4_K_S 32 pp2048 2062.84 2111.06 1.02
RTX 3090 llama 8B Q4_K_S 64 pp2048 2797.36 2828.84 1.01
RTX 3090 llama 8B Q4_K_S 128 pp2048 3273.46 3292.54 1.01
RTX 3090 llama 8B Q4_K_S 256 pp2048 3614.64 3736.80 1.03
RTX 3090 llama 8B Q4_K_S 512 pp2048 3647.43 3838.91 1.05
RTX 3090 llama 8B Q4_K_S 1024 pp2048 3633.62 3845.57 1.06
RTX 3090 llama 8B Q4_K_S 2048 pp2048 3658.26 3856.13 1.05
RTX 3090 llama 8B Q5_0 16 pp2048 1082.62 1112.41 1.03
RTX 3090 llama 8B Q5_0 32 pp2048 1835.23 1862.76 1.02
RTX 3090 llama 8B Q5_0 64 pp2048 2723.04 2812.12 1.03
RTX 3090 llama 8B Q5_0 128 pp2048 3377.48 3443.77 1.02
RTX 3090 llama 8B Q5_0 256 pp2048 3872.29 3945.45 1.02
RTX 3090 llama 8B Q5_0 512 pp2048 3943.86 4128.28 1.05
RTX 3090 llama 8B Q5_0 1024 pp2048 3917.80 4180.06 1.07
RTX 3090 llama 8B Q5_0 2048 pp2048 3815.57 4112.45 1.08
RTX 3090 llama 8B Q5_1 16 pp2048 1163.04 1209.10 1.04
RTX 3090 llama 8B Q5_1 32 pp2048 1908.33 1968.98 1.03
RTX 3090 llama 8B Q5_1 64 pp2048 2626.80 2722.52 1.04
RTX 3090 llama 8B Q5_1 128 pp2048 3119.17 3208.91 1.03
RTX 3090 llama 8B Q5_1 256 pp2048 3554.91 3676.80 1.03
RTX 3090 llama 8B Q5_1 512 pp2048 3579.17 3866.58 1.08
RTX 3090 llama 8B Q5_1 1024 pp2048 3550.68 3916.56 1.10
RTX 3090 llama 8B Q5_1 2048 pp2048 3473.49 3878.30 1.12
RTX 3090 llama 8B Q5_K_S 16 pp2048 1208.44 1238.99 1.03
RTX 3090 llama 8B Q5_K_S 32 pp2048 1908.82 1945.46 1.02
RTX 3090 llama 8B Q5_K_S 64 pp2048 2663.21 2698.14 1.01
RTX 3090 llama 8B Q5_K_S 128 pp2048 3166.88 3195.43 1.01
RTX 3090 llama 8B Q5_K_S 256 pp2048 3461.94 3541.61 1.02
RTX 3090 llama 8B Q5_K_S 512 pp2048 3447.27 3683.95 1.07
RTX 3090 llama 8B Q5_K_S 1024 pp2048 3441.83 3709.38 1.08
RTX 3090 llama 8B Q5_K_S 2048 pp2048 3418.16 3675.02 1.08
RTX 3090 llama 8B Q6_K 16 pp2048 1019.65 1054.29 1.03
RTX 3090 llama 8B Q6_K 32 pp2048 1607.95 1660.46 1.03
RTX 3090 llama 8B Q6_K 64 pp2048 2368.14 2395.12 1.01
RTX 3090 llama 8B Q6_K 128 pp2048 2902.90 2903.91 1.00
RTX 3090 llama 8B Q6_K 256 pp2048 3032.74 3102.12 1.02
RTX 3090 llama 8B Q6_K 512 pp2048 3054.42 3213.85 1.05
RTX 3090 llama 8B Q6_K 1024 pp2048 3024.11 3207.76 1.06
RTX 3090 llama 8B Q6_K 2048 pp2048 3020.34 3184.22 1.05
RTX 3090 llama 8B Q8_0 16 pp2048 1009.84 1057.60 1.05
RTX 3090 llama 8B Q8_0 32 pp2048 1845.51 1934.83 1.05
RTX 3090 llama 8B Q8_0 64 pp2048 2702.71 2818.67 1.04
RTX 3090 llama 8B Q8_0 128 pp2048 3343.00 3406.98 1.02
RTX 3090 llama 8B Q8_0 256 pp2048 3531.89 3761.51 1.07
RTX 3090 llama 8B Q8_0 512 pp2048 3625.91 3943.84 1.09
RTX 3090 llama 8B Q8_0 1024 pp2048 3646.27 4013.47 1.10
RTX 3090 llama 8B Q8_0 2048 pp2048 3603.41 3947.80 1.10
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 2051.06 2140.08 1.04
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 4246.78 4473.28 1.05
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 6275.23 6594.61 1.05
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 7801.87 8204.85 1.05
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 10387.55 10510.55 1.01
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 11762.69 11858.79 1.01
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 11817.50 11728.57 0.99
RTX 4090 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 11149.71 11051.63 0.99
RTX 4090 llama 8B IQ2_S - 2.5 bpw 16 pp2048 2152.15 2241.26 1.04
RTX 4090 llama 8B IQ2_S - 2.5 bpw 32 pp2048 3595.26 3797.73 1.06
RTX 4090 llama 8B IQ2_S - 2.5 bpw 64 pp2048 5701.23 5996.82 1.05
RTX 4090 llama 8B IQ2_S - 2.5 bpw 128 pp2048 7310.83 7583.19 1.04
RTX 4090 llama 8B IQ2_S - 2.5 bpw 256 pp2048 9431.00 9547.23 1.01
RTX 4090 llama 8B IQ2_S - 2.5 bpw 512 pp2048 10236.59 10316.21 1.01
RTX 4090 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 10015.84 9987.91 1.00
RTX 4090 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 9006.80 8941.80 0.99
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 2212.99 2311.92 1.04
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 3684.59 3889.42 1.06
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 5706.91 6021.43 1.06
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 7268.56 7546.72 1.04
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 9345.52 9393.69 1.01
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 10318.25 10412.10 1.01
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 10429.65 10409.09 1.00
RTX 4090 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 9948.05 9908.85 1.00
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 2444.92 2564.29 1.05
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 4069.08 4306.45 1.06
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 6451.07 6852.96 1.06
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 8326.73 8690.61 1.04
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 11046.55 11111.88 1.01
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 12474.25 12547.83 1.01
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 12480.25 12329.82 0.99
RTX 4090 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 11759.35 11557.06 0.98
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 1690.45 1768.28 1.05
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 3004.71 3165.76 1.05
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 5468.23 5807.97 1.06
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 7676.75 8006.51 1.04
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 10283.53 10342.69 1.01
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 11402.29 11449.99 1.00
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 11102.03 11032.27 0.99
RTX 4090 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 9891.13 9760.06 0.99
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 1711.79 1784.18 1.04
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 3053.39 3208.65 1.05
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 5490.47 5820.41 1.06
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 7660.17 7996.91 1.04
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 10250.32 10316.54 1.01
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 11418.21 11462.64 1.00
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 11114.65 11048.02 0.99
RTX 4090 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 9917.57 9770.41 0.99
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 1813.25 1886.35 1.04
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 3140.61 3297.22 1.05
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 5609.36 5936.03 1.06
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 7496.43 7886.36 1.05
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 10163.83 10431.51 1.03
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 11568.22 11638.58 1.01
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 11229.08 11140.86 0.99
RTX 4090 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 9996.89 9848.80 0.99
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 1920.61 1986.52 1.03
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 3239.57 3397.37 1.05
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 5630.39 5947.84 1.06
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 7419.23 7800.23 1.05
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 10159.54 10315.14 1.02
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 11455.59 11521.04 1.01
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 11127.08 11014.03 0.99
RTX 4090 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 9949.74 9793.03 0.98
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 1919.03 1992.74 1.04
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 3375.41 3495.87 1.04
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 6074.44 6489.78 1.07
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 8205.25 8656.49 1.05
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 11132.19 11271.70 1.01
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 12561.90 12647.02 1.01
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 12601.12 12508.26 0.99
RTX 4090 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 11638.50 11506.62 0.99
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 2045.85 2200.35 1.08
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 3667.76 3836.37 1.05
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 6295.39 6685.25 1.06
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 8333.01 8706.86 1.04
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 11218.46 11314.18 1.01
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 12737.58 12733.80 1.00
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 12843.75 12630.87 0.98
RTX 4090 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 12000.69 11728.27 0.98
RTX 4090 llama 8B Q2_K_M 16 pp2048 2225.71 2335.44 1.05
RTX 4090 llama 8B Q2_K_M 32 pp2048 3655.45 3810.78 1.04
RTX 4090 llama 8B Q2_K_M 64 pp2048 5351.91 5568.08 1.04
RTX 4090 llama 8B Q2_K_M 128 pp2048 5398.33 5585.71 1.03
RTX 4090 llama 8B Q2_K_M 256 pp2048 7455.22 7539.28 1.01
RTX 4090 llama 8B Q2_K_M 512 pp2048 9183.92 9256.55 1.01
RTX 4090 llama 8B Q2_K_M 1024 pp2048 9570.04 9485.35 0.99
RTX 4090 llama 8B Q2_K_M 2048 pp2048 9259.54 9137.58 0.99
RTX 4090 llama 8B Q3_K_S 16 pp2048 2046.31 2165.35 1.06
RTX 4090 llama 8B Q3_K_S 32 pp2048 3508.84 3656.20 1.04
RTX 4090 llama 8B Q3_K_S 64 pp2048 5627.15 5901.24 1.05
RTX 4090 llama 8B Q3_K_S 128 pp2048 7519.51 7840.05 1.04
RTX 4090 llama 8B Q3_K_S 256 pp2048 9609.18 9716.96 1.01
RTX 4090 llama 8B Q3_K_S 512 pp2048 10555.68 10603.18 1.00
RTX 4090 llama 8B Q3_K_S 1024 pp2048 10716.37 10698.36 1.00
RTX 4090 llama 8B Q3_K_S 2048 pp2048 10323.42 10260.99 0.99
RTX 4090 llama 8B Q4_0 16 pp2048 2038.28 2149.02 1.05
RTX 4090 llama 8B Q4_0 32 pp2048 3529.40 3781.64 1.07
RTX 4090 llama 8B Q4_0 64 pp2048 6035.11 6434.53 1.07
RTX 4090 llama 8B Q4_0 128 pp2048 8321.27 8805.03 1.06
RTX 4090 llama 8B Q4_0 256 pp2048 11265.30 11410.03 1.01
RTX 4090 llama 8B Q4_0 512 pp2048 12648.66 12726.13 1.01
RTX 4090 llama 8B Q4_0 1024 pp2048 12846.57 12684.68 0.99
RTX 4090 llama 8B Q4_0 2048 pp2048 12147.89 11988.03 0.99
RTX 4090 llama 8B Q4_1 16 pp2048 1899.04 1985.88 1.05
RTX 4090 llama 8B Q4_1 32 pp2048 3510.12 3747.85 1.07
RTX 4090 llama 8B Q4_1 64 pp2048 5897.90 6274.45 1.06
RTX 4090 llama 8B Q4_1 128 pp2048 7918.88 8413.06 1.06
RTX 4090 llama 8B Q4_1 256 pp2048 10660.16 10823.86 1.02
RTX 4090 llama 8B Q4_1 512 pp2048 11896.95 11977.46 1.01
RTX 4090 llama 8B Q4_1 1024 pp2048 12129.23 12035.36 0.99
RTX 4090 llama 8B Q4_1 2048 pp2048 11630.16 11454.49 0.98
RTX 4090 llama 8B Q4_K_S 16 pp2048 1984.97 2061.79 1.04
RTX 4090 llama 8B Q4_K_S 32 pp2048 3746.64 3934.79 1.05
RTX 4090 llama 8B Q4_K_S 64 pp2048 6012.14 6319.46 1.05
RTX 4090 llama 8B Q4_K_S 128 pp2048 8013.48 8379.56 1.05
RTX 4090 llama 8B Q4_K_S 256 pp2048 10724.40 10848.69 1.01
RTX 4090 llama 8B Q4_K_S 512 pp2048 12049.54 12119.00 1.01
RTX 4090 llama 8B Q4_K_S 1024 pp2048 12274.25 12176.15 0.99
RTX 4090 llama 8B Q4_K_S 2048 pp2048 11756.45 11606.67 0.99
RTX 4090 llama 8B Q5_0 16 pp2048 1716.52 1776.30 1.03
RTX 4090 llama 8B Q5_0 32 pp2048 3202.87 3347.68 1.05
RTX 4090 llama 8B Q5_0 64 pp2048 5362.77 5707.19 1.06
RTX 4090 llama 8B Q5_0 128 pp2048 7764.58 8200.72 1.06
RTX 4090 llama 8B Q5_0 256 pp2048 10539.71 10636.43 1.01
RTX 4090 llama 8B Q5_0 512 pp2048 11725.45 11787.09 1.01
RTX 4090 llama 8B Q5_0 1024 pp2048 11895.14 11899.45 1.00
RTX 4090 llama 8B Q5_0 2048 pp2048 11228.92 11171.73 0.99
RTX 4090 llama 8B Q5_1 16 pp2048 1682.27 1758.43 1.05
RTX 4090 llama 8B Q5_1 32 pp2048 3210.89 3385.18 1.05
RTX 4090 llama 8B Q5_1 64 pp2048 5403.87 5730.07 1.06
RTX 4090 llama 8B Q5_1 128 pp2048 7452.35 7891.63 1.06
RTX 4090 llama 8B Q5_1 256 pp2048 9955.96 10224.32 1.03
RTX 4090 llama 8B Q5_1 512 pp2048 11236.09 11330.23 1.01
RTX 4090 llama 8B Q5_1 1024 pp2048 11538.39 11498.30 1.00
RTX 4090 llama 8B Q5_1 2048 pp2048 10926.80 10806.31 0.99
RTX 4090 llama 8B Q5_K_S 16 pp2048 1806.60 1881.09 1.04
RTX 4090 llama 8B Q5_K_S 32 pp2048 3337.24 3488.51 1.05
RTX 4090 llama 8B Q5_K_S 64 pp2048 5577.67 5868.07 1.05
RTX 4090 llama 8B Q5_K_S 128 pp2048 7736.69 8084.18 1.04
RTX 4090 llama 8B Q5_K_S 256 pp2048 10330.38 10434.91 1.01
RTX 4090 llama 8B Q5_K_S 512 pp2048 11464.33 11532.39 1.01
RTX 4090 llama 8B Q5_K_S 1024 pp2048 11736.69 11687.72 1.00
RTX 4090 llama 8B Q5_K_S 2048 pp2048 11295.02 11160.93 0.99
RTX 4090 llama 8B Q6_K 16 pp2048 1519.87 1560.78 1.03
RTX 4090 llama 8B Q6_K 32 pp2048 2760.33 2881.17 1.04
RTX 4090 llama 8B Q6_K 64 pp2048 4849.10 5076.32 1.05
RTX 4090 llama 8B Q6_K 128 pp2048 6855.64 7146.23 1.04
RTX 4090 llama 8B Q6_K 256 pp2048 9226.49 9289.70 1.01
RTX 4090 llama 8B Q6_K 512 pp2048 10259.05 10261.84 1.00
RTX 4090 llama 8B Q6_K 1024 pp2048 10482.07 10497.81 1.00
RTX 4090 llama 8B Q6_K 2048 pp2048 9960.79 9880.67 0.99
RTX 4090 llama 8B Q8_0 16 pp2048 1386.19 1438.26 1.04
RTX 4090 llama 8B Q8_0 32 pp2048 2601.34 2710.53 1.04
RTX 4090 llama 8B Q8_0 64 pp2048 4515.20 4715.05 1.04
RTX 4090 llama 8B Q8_0 128 pp2048 6960.84 7281.37 1.05
RTX 4090 llama 8B Q8_0 256 pp2048 10051.84 10112.51 1.01
RTX 4090 llama 8B Q8_0 512 pp2048 11981.62 12003.96 1.00
RTX 4090 llama 8B Q8_0 1024 pp2048 12294.40 12582.10 1.02
RTX 4090 llama 8B Q8_0 2048 pp2048 11817.50 11960.96 1.01
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 2496.61 2669.11 1.07
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 3832.63 4121.04 1.08
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 5633.04 6087.73 1.08
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 7443.76 7678.81 1.03
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 9963.88 10068.22 1.01
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 12294.54 12374.13 1.01
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 13069.30 13306.25 1.02
RTX 5090 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 12858.76 12853.98 1.00
RTX 5090 llama 8B IQ2_S - 2.5 bpw 16 pp2048 2319.37 2458.42 1.06
RTX 5090 llama 8B IQ2_S - 2.5 bpw 32 pp2048 3546.79 3811.45 1.07
RTX 5090 llama 8B IQ2_S - 2.5 bpw 64 pp2048 5170.34 5535.29 1.07
RTX 5090 llama 8B IQ2_S - 2.5 bpw 128 pp2048 6692.06 6843.69 1.02
RTX 5090 llama 8B IQ2_S - 2.5 bpw 256 pp2048 8829.50 8878.40 1.01
RTX 5090 llama 8B IQ2_S - 2.5 bpw 512 pp2048 10511.14 10589.79 1.01
RTX 5090 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 10811.91 10810.38 1.00
RTX 5090 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 10028.71 9874.24 0.98
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 2343.22 2481.68 1.06
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 3544.80 3800.38 1.07
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 5137.91 5490.09 1.07
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 6588.22 6729.44 1.02
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 8681.77 8720.55 1.00
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 10518.30 10617.57 1.01
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 11252.00 11264.60 1.00
RTX 5090 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 11177.54 10986.41 0.98
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 2550.95 2720.15 1.07
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 3971.71 4273.48 1.08
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 5861.02 6358.68 1.08
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 7866.92 8028.89 1.02
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 10675.65 10815.04 1.01
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 13401.80 13482.98 1.01
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 14430.46 14567.97 1.01
RTX 5090 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 14264.27 14149.29 0.99
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 2188.28 2307.19 1.05
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 3512.81 3747.28 1.07
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 5312.31 5798.50 1.09
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 7495.62 7596.26 1.01
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 10075.42 10111.54 1.00
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 12343.28 12363.81 1.00
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 12676.24 12615.26 1.00
RTX 5090 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 11639.28 11384.37 0.98
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 2196.70 2320.47 1.06
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 3535.98 3769.09 1.07
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 5353.20 5845.29 1.09
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 7482.68 7631.04 1.02
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 10053.65 10098.01 1.00
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 12323.42 12345.50 1.00
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 12659.18 12609.51 1.00
RTX 5090 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 11626.41 11363.10 0.98
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 2256.03 2388.28 1.06
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 3608.63 3849.82 1.07
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 5464.58 5901.24 1.08
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 7556.71 7679.79 1.02
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 10161.63 10185.14 1.00
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 12506.66 12526.99 1.00
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 12842.50 12824.45 1.00
RTX 5090 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 11767.00 11560.94 0.98
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 2285.65 2426.41 1.06
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 3635.53 3865.49 1.06
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 5440.51 5857.38 1.08
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 7365.79 7518.44 1.02
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 10054.47 10094.53 1.00
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 12296.79 12310.97 1.00
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 12615.41 12645.23 1.00
RTX 5090 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 11553.29 11436.79 0.99
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 2411.07 2573.99 1.07
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 3810.36 4125.07 1.08
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 5751.64 6217.35 1.08
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 7650.71 7847.87 1.03
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 10628.84 10713.78 1.01
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 13423.04 13582.02 1.01
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 14361.47 14381.25 1.00
RTX 5090 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 14061.36 13866.25 0.99
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 2453.67 2649.67 1.08
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 3830.15 4098.21 1.07
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 5778.99 6229.85 1.08
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 7704.08 7906.08 1.03
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 10647.48 10725.77 1.01
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 13363.77 13570.83 1.02
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 14194.78 14399.00 1.01
RTX 5090 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 14018.08 14019.72 1.00
RTX 5090 llama 8B Q2_K_M 16 pp2048 2389.47 2514.73 1.05
RTX 5090 llama 8B Q2_K_M 32 pp2048 3527.23 3765.43 1.07
RTX 5090 llama 8B Q2_K_M 64 pp2048 4938.19 5271.32 1.07
RTX 5090 llama 8B Q2_K_M 128 pp2048 5918.08 6027.20 1.02
RTX 5090 llama 8B Q2_K_M 256 pp2048 7755.61 7718.32 1.00
RTX 5090 llama 8B Q2_K_M 512 pp2048 9396.16 9385.64 1.00
RTX 5090 llama 8B Q2_K_M 1024 pp2048 10186.56 10156.11 1.00
RTX 5090 llama 8B Q2_K_M 2048 pp2048 10190.90 10079.26 0.99
RTX 5090 llama 8B Q3_K_S 16 pp2048 2330.32 2421.37 1.04
RTX 5090 llama 8B Q3_K_S 32 pp2048 3533.36 3783.07 1.07
RTX 5090 llama 8B Q3_K_S 64 pp2048 5132.92 5501.25 1.07
RTX 5090 llama 8B Q3_K_S 128 pp2048 6808.85 6914.53 1.02
RTX 5090 llama 8B Q3_K_S 256 pp2048 8945.88 8907.58 1.00
RTX 5090 llama 8B Q3_K_S 512 pp2048 10927.60 10870.72 0.99
RTX 5090 llama 8B Q3_K_S 1024 pp2048 11612.50 11505.02 0.99
RTX 5090 llama 8B Q3_K_S 2048 pp2048 11698.27 11334.80 0.97
RTX 5090 llama 8B Q4_0 16 pp2048 2381.65 2579.37 1.08
RTX 5090 llama 8B Q4_0 32 pp2048 3776.92 4098.18 1.09
RTX 5090 llama 8B Q4_0 64 pp2048 5833.95 6265.67 1.07
RTX 5090 llama 8B Q4_0 128 pp2048 7763.89 8033.39 1.03
RTX 5090 llama 8B Q4_0 256 pp2048 10753.90 10890.13 1.01
RTX 5090 llama 8B Q4_0 512 pp2048 13712.04 13769.89 1.00
RTX 5090 llama 8B Q4_0 1024 pp2048 14864.50 14840.43 1.00
RTX 5090 llama 8B Q4_0 2048 pp2048 15045.80 14747.20 0.98
RTX 5090 llama 8B Q4_1 16 pp2048 2354.58 2517.58 1.07
RTX 5090 llama 8B Q4_1 32 pp2048 3707.93 4027.02 1.09
RTX 5090 llama 8B Q4_1 64 pp2048 5705.13 6094.33 1.07
RTX 5090 llama 8B Q4_1 128 pp2048 7442.28 7699.66 1.03
RTX 5090 llama 8B Q4_1 256 pp2048 10137.14 10312.79 1.02
RTX 5090 llama 8B Q4_1 512 pp2048 12737.15 12737.61 1.00
RTX 5090 llama 8B Q4_1 1024 pp2048 13729.90 13613.88 0.99
RTX 5090 llama 8B Q4_1 2048 pp2048 13837.26 13597.66 0.98
RTX 5090 llama 8B Q4_K_S 16 pp2048 2402.38 2540.27 1.06
RTX 5090 llama 8B Q4_K_S 32 pp2048 3776.61 4016.67 1.06
RTX 5090 llama 8B Q4_K_S 64 pp2048 5657.64 6120.04 1.08
RTX 5090 llama 8B Q4_K_S 128 pp2048 7550.28 7767.69 1.03
RTX 5090 llama 8B Q4_K_S 256 pp2048 10235.53 10305.99 1.01
RTX 5090 llama 8B Q4_K_S 512 pp2048 12794.76 12856.07 1.00
RTX 5090 llama 8B Q4_K_S 1024 pp2048 13792.70 13758.04 1.00
RTX 5090 llama 8B Q4_K_S 2048 pp2048 13875.87 13609.89 0.98
RTX 5090 llama 8B Q5_0 16 pp2048 2124.60 2264.17 1.07
RTX 5090 llama 8B Q5_0 32 pp2048 3432.66 3652.98 1.06
RTX 5090 llama 8B Q5_0 64 pp2048 5414.43 5772.73 1.07
RTX 5090 llama 8B Q5_0 128 pp2048 7428.22 7681.88 1.03
RTX 5090 llama 8B Q5_0 256 pp2048 10098.86 10327.67 1.02
RTX 5090 llama 8B Q5_0 512 pp2048 12706.72 12976.60 1.02
RTX 5090 llama 8B Q5_0 1024 pp2048 13652.16 13894.98 1.02
RTX 5090 llama 8B Q5_0 2048 pp2048 13562.94 13613.52 1.00
RTX 5090 llama 8B Q5_1 16 pp2048 2162.04 2297.17 1.06
RTX 5090 llama 8B Q5_1 32 pp2048 3512.81 3783.52 1.08
RTX 5090 llama 8B Q5_1 64 pp2048 5465.39 5722.03 1.05
RTX 5090 llama 8B Q5_1 128 pp2048 7137.47 7426.51 1.04
RTX 5090 llama 8B Q5_1 256 pp2048 9629.42 9781.15 1.02
RTX 5090 llama 8B Q5_1 512 pp2048 12068.91 12122.91 1.00
RTX 5090 llama 8B Q5_1 1024 pp2048 12872.44 12864.01 1.00
RTX 5090 llama 8B Q5_1 2048 pp2048 12786.78 12556.41 0.98
RTX 5090 llama 8B Q5_K_S 16 pp2048 2269.41 2392.22 1.05
RTX 5090 llama 8B Q5_K_S 32 pp2048 3618.89 3839.88 1.06
RTX 5090 llama 8B Q5_K_S 64 pp2048 5370.34 5767.50 1.07
RTX 5090 llama 8B Q5_K_S 128 pp2048 7316.37 7503.74 1.03
RTX 5090 llama 8B Q5_K_S 256 pp2048 9837.46 9966.31 1.01
RTX 5090 llama 8B Q5_K_S 512 pp2048 12376.44 12446.69 1.01
RTX 5090 llama 8B Q5_K_S 1024 pp2048 13353.43 13325.32 1.00
RTX 5090 llama 8B Q5_K_S 2048 pp2048 13465.70 13185.79 0.98
RTX 5090 llama 8B Q6_K 16 pp2048 2016.07 2107.02 1.05
RTX 5090 llama 8B Q6_K 32 pp2048 3240.83 3435.43 1.06
RTX 5090 llama 8B Q6_K 64 pp2048 4836.80 5166.19 1.07
RTX 5090 llama 8B Q6_K 128 pp2048 6401.53 6458.14 1.01
RTX 5090 llama 8B Q6_K 256 pp2048 8335.20 8338.29 1.00
RTX 5090 llama 8B Q6_K 512 pp2048 10213.68 10140.66 0.99
RTX 5090 llama 8B Q6_K 1024 pp2048 10994.32 10853.87 0.99
RTX 5090 llama 8B Q6_K 2048 pp2048 10964.21 10642.39 0.97
RTX 5090 llama 8B Q8_0 16 pp2048 1805.64 1908.17 1.06
RTX 5090 llama 8B Q8_0 32 pp2048 3074.15 3288.35 1.07
RTX 5090 llama 8B Q8_0 64 pp2048 5142.13 5444.45 1.06
RTX 5090 llama 8B Q8_0 128 pp2048 7273.50 7510.55 1.03
RTX 5090 llama 8B Q8_0 256 pp2048 10082.06 10166.27 1.01
RTX 5090 llama 8B Q8_0 512 pp2048 13045.94 12913.98 0.99
RTX 5090 llama 8B Q8_0 1024 pp2048 14380.70 14256.86 0.99
RTX 5090 llama 8B Q8_0 2048 pp2048 14610.92 14220.46 0.97
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 16 pp2048 655.49 684.67 1.04
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 32 pp2048 967.63 1005.61 1.04
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 64 pp2048 485.38 484.93 1.00
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 128 pp2048 891.24 890.63 1.00
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 256 pp2048 1519.66 1508.61 0.99
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 512 pp2048 2431.86 2434.22 1.00
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 3022.79 3122.42 1.03
V100-PCIE-32GB llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 3533.89 3569.50 1.01
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 16 pp2048 598.99 618.52 1.03
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 32 pp2048 915.16 948.53 1.04
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 64 pp2048 475.79 472.65 0.99
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 128 pp2048 868.34 867.66 1.00
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 256 pp2048 1505.15 1509.90 1.00
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 512 pp2048 2291.61 2347.58 1.02
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 1024 pp2048 3026.94 3010.84 0.99
V100-PCIE-32GB llama 8B IQ2_S - 2.5 bpw 2048 pp2048 3437.98 3425.61 1.00
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 618.75 634.97 1.03
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 929.75 964.24 1.04
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 482.64 479.07 0.99
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 874.94 869.97 0.99
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 1496.33 1513.46 1.01
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 2302.05 2338.30 1.02
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 3027.17 3020.78 1.00
V100-PCIE-32GB llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 3558.96 3535.63 0.99
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 661.18 684.70 1.04
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 992.85 1022.87 1.03
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 479.74 479.77 1.00
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 881.36 874.05 0.99
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 1544.67 1560.01 1.01
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 2309.21 2346.99 1.02
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 3042.72 3083.61 1.01
V100-PCIE-32GB llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 3560.97 3534.67 0.99
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 16 pp2048 596.85 623.38 1.04
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 32 pp2048 929.12 939.98 1.01
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 64 pp2048 470.09 469.18 1.00
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 128 pp2048 849.23 867.02 1.02
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 256 pp2048 1502.09 1514.30 1.01
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 512 pp2048 2303.28 2362.60 1.03
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 3005.38 3033.61 1.01
V100-PCIE-32GB llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 3435.01 3480.02 1.01
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 596.40 628.90 1.05
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 921.14 944.40 1.03
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 482.10 481.22 1.00
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 866.71 884.81 1.02
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 1549.84 1553.52 1.00
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 2317.55 2390.51 1.03
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 3042.88 3062.55 1.01
V100-PCIE-32GB llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 3499.88 3512.30 1.00
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 16 pp2048 586.62 616.85 1.05
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 32 pp2048 912.54 940.13 1.03
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 64 pp2048 472.32 471.07 1.00
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 128 pp2048 846.54 872.10 1.03
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 256 pp2048 1504.49 1505.10 1.00
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 512 pp2048 2286.07 2341.09 1.02
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 2994.52 3059.08 1.02
V100-PCIE-32GB llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 3391.72 3486.46 1.03
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 590.00 617.23 1.05
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 919.98 945.98 1.03
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 472.26 470.41 1.00
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 858.44 854.71 1.00
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 1506.03 1529.67 1.02
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 2304.59 2303.91 1.00
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 3036.28 3029.16 1.00
V100-PCIE-32GB llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 3500.51 3446.84 0.98
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 16 pp2048 705.73 734.21 1.04
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 32 pp2048 1034.88 1063.42 1.03
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 64 pp2048 592.11 601.51 1.02
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 128 pp2048 1073.78 1089.67 1.01
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 256 pp2048 1880.66 1918.13 1.02
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 512 pp2048 2817.84 2788.97 0.99
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 3425.77 3449.86 1.01
V100-PCIE-32GB llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 3736.12 3821.48 1.02
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 16 pp2048 681.62 712.63 1.05
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 32 pp2048 1041.40 1080.57 1.04
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 64 pp2048 594.60 601.80 1.01
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 128 pp2048 1072.47 1089.33 1.02
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 256 pp2048 1824.67 1879.29 1.03
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 512 pp2048 2796.94 2770.91 0.99
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 3417.21 3433.05 1.00
V100-PCIE-32GB llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 3813.11 3847.11 1.01
V100-PCIE-32GB llama 8B Q2_K_M 16 pp2048 617.88 658.76 1.07
V100-PCIE-32GB llama 8B Q2_K_M 32 pp2048 863.64 895.81 1.04
V100-PCIE-32GB llama 8B Q2_K_M 64 pp2048 693.15 695.04 1.00
V100-PCIE-32GB llama 8B Q2_K_M 128 pp2048 1176.05 1191.37 1.01
V100-PCIE-32GB llama 8B Q2_K_M 256 pp2048 2002.07 2021.02 1.01
V100-PCIE-32GB llama 8B Q2_K_M 512 pp2048 2996.20 2952.60 0.99
V100-PCIE-32GB llama 8B Q2_K_M 1024 pp2048 3527.85 3550.10 1.01
V100-PCIE-32GB llama 8B Q2_K_M 2048 pp2048 3838.89 3888.94 1.01
V100-PCIE-32GB llama 8B Q3_K_S 16 pp2048 661.52 672.83 1.02
V100-PCIE-32GB llama 8B Q3_K_S 32 pp2048 940.43 957.75 1.02
V100-PCIE-32GB llama 8B Q3_K_S 64 pp2048 583.31 583.40 1.00
V100-PCIE-32GB llama 8B Q3_K_S 128 pp2048 1112.00 1113.52 1.00
V100-PCIE-32GB llama 8B Q3_K_S 256 pp2048 1898.19 1924.09 1.01
V100-PCIE-32GB llama 8B Q3_K_S 512 pp2048 2891.21 2873.91 0.99
V100-PCIE-32GB llama 8B Q3_K_S 1024 pp2048 3454.52 3498.17 1.01
V100-PCIE-32GB llama 8B Q3_K_S 2048 pp2048 3887.75 3910.21 1.01
V100-PCIE-32GB llama 8B Q4_0 16 pp2048 756.45 785.83 1.04
V100-PCIE-32GB llama 8B Q4_0 32 pp2048 1131.56 1164.31 1.03
V100-PCIE-32GB llama 8B Q4_0 64 pp2048 640.70 640.84 1.00
V100-PCIE-32GB llama 8B Q4_0 128 pp2048 1173.21 1173.07 1.00
V100-PCIE-32GB llama 8B Q4_0 256 pp2048 1997.19 2002.62 1.00
V100-PCIE-32GB llama 8B Q4_0 512 pp2048 2968.04 3001.87 1.01
V100-PCIE-32GB llama 8B Q4_0 1024 pp2048 3592.48 3633.81 1.01
V100-PCIE-32GB llama 8B Q4_0 2048 pp2048 3983.25 4012.23 1.01
V100-PCIE-32GB llama 8B Q4_1 16 pp2048 755.86 797.66 1.06
V100-PCIE-32GB llama 8B Q4_1 32 pp2048 1105.06 1148.55 1.04
V100-PCIE-32GB llama 8B Q4_1 64 pp2048 616.96 626.76 1.02
V100-PCIE-32GB llama 8B Q4_1 128 pp2048 1102.96 1134.78 1.03
V100-PCIE-32GB llama 8B Q4_1 256 pp2048 1897.20 1920.41 1.01
V100-PCIE-32GB llama 8B Q4_1 512 pp2048 2826.98 2917.87 1.03
V100-PCIE-32GB llama 8B Q4_1 1024 pp2048 3480.44 3514.25 1.01
V100-PCIE-32GB llama 8B Q4_1 2048 pp2048 3903.74 3927.13 1.01
V100-PCIE-32GB llama 8B Q4_K_S 16 pp2048 722.28 747.47 1.03
V100-PCIE-32GB llama 8B Q4_K_S 32 pp2048 1050.15 1080.58 1.03
V100-PCIE-32GB llama 8B Q4_K_S 64 pp2048 576.15 583.32 1.01
V100-PCIE-32GB llama 8B Q4_K_S 128 pp2048 1066.36 1072.76 1.01
V100-PCIE-32GB llama 8B Q4_K_S 256 pp2048 1892.17 1878.19 0.99
V100-PCIE-32GB llama 8B Q4_K_S 512 pp2048 2754.31 2804.04 1.02
V100-PCIE-32GB llama 8B Q4_K_S 1024 pp2048 3413.79 3420.30 1.00
V100-PCIE-32GB llama 8B Q4_K_S 2048 pp2048 3857.34 3872.48 1.00
V100-PCIE-32GB llama 8B Q5_0 16 pp2048 628.88 649.32 1.03
V100-PCIE-32GB llama 8B Q5_0 32 pp2048 981.73 1014.83 1.03
V100-PCIE-32GB llama 8B Q5_0 64 pp2048 678.18 712.32 1.05
V100-PCIE-32GB llama 8B Q5_0 128 pp2048 1274.27 1295.41 1.02
V100-PCIE-32GB llama 8B Q5_0 256 pp2048 2166.93 2216.31 1.02
V100-PCIE-32GB llama 8B Q5_0 512 pp2048 3062.22 3100.79 1.01
V100-PCIE-32GB llama 8B Q5_0 1024 pp2048 3688.66 3704.33 1.00
V100-PCIE-32GB llama 8B Q5_0 2048 pp2048 3969.28 3989.81 1.01
V100-PCIE-32GB llama 8B Q5_1 16 pp2048 669.87 709.07 1.06
V100-PCIE-32GB llama 8B Q5_1 32 pp2048 1014.51 1006.65 0.99
V100-PCIE-32GB llama 8B Q5_1 64 pp2048 650.09 668.57 1.03
V100-PCIE-32GB llama 8B Q5_1 128 pp2048 1225.07 1256.79 1.03
V100-PCIE-32GB llama 8B Q5_1 256 pp2048 2188.03 2186.77 1.00
V100-PCIE-32GB llama 8B Q5_1 512 pp2048 3039.24 3078.78 1.01
V100-PCIE-32GB llama 8B Q5_1 1024 pp2048 3643.33 3672.06 1.01
V100-PCIE-32GB llama 8B Q5_1 2048 pp2048 3989.95 3954.89 0.99
V100-PCIE-32GB llama 8B Q5_K_S 16 pp2048 673.47 672.84 1.00
V100-PCIE-32GB llama 8B Q5_K_S 32 pp2048 977.79 997.36 1.02
V100-PCIE-32GB llama 8B Q5_K_S 64 pp2048 605.24 619.31 1.02
V100-PCIE-32GB llama 8B Q5_K_S 128 pp2048 1198.91 1183.14 0.99
V100-PCIE-32GB llama 8B Q5_K_S 256 pp2048 2044.41 2075.26 1.02
V100-PCIE-32GB llama 8B Q5_K_S 512 pp2048 2934.96 2939.96 1.00
V100-PCIE-32GB llama 8B Q5_K_S 1024 pp2048 3570.98 3634.38 1.02
V100-PCIE-32GB llama 8B Q5_K_S 2048 pp2048 4010.28 3997.54 1.00
V100-PCIE-32GB llama 8B Q6_K 16 pp2048 628.67 646.12 1.03
V100-PCIE-32GB llama 8B Q6_K 32 pp2048 934.63 947.05 1.01
V100-PCIE-32GB llama 8B Q6_K 64 pp2048 636.59 648.46 1.02
V100-PCIE-32GB llama 8B Q6_K 128 pp2048 1226.70 1227.54 1.00
V100-PCIE-32GB llama 8B Q6_K 256 pp2048 2175.75 2167.15 1.00
V100-PCIE-32GB llama 8B Q6_K 512 pp2048 2919.27 3003.11 1.03
V100-PCIE-32GB llama 8B Q6_K 1024 pp2048 3623.16 3595.27 0.99
V100-PCIE-32GB llama 8B Q6_K 2048 pp2048 3955.47 3920.11 0.99
V100-PCIE-32GB llama 8B Q8_0 16 pp2048 624.07 646.04 1.04
V100-PCIE-32GB llama 8B Q8_0 32 pp2048 1013.73 987.08 0.97
V100-PCIE-32GB llama 8B Q8_0 64 pp2048 688.05 720.08 1.05
V100-PCIE-32GB llama 8B Q8_0 128 pp2048 1378.97 1408.62 1.02
V100-PCIE-32GB llama 8B Q8_0 256 pp2048 2424.42 2418.16 1.00
V100-PCIE-32GB llama 8B Q8_0 512 pp2048 3397.63 3366.14 0.99
V100-PCIE-32GB llama 8B Q8_0 1024 pp2048 3915.87 3948.88 1.01
V100-PCIE-32GB llama 8B Q8_0 2048 pp2048 4181.36 4185.98 1.00
Quantization sweep NVIDIA dense
GPU Model Microbatch size Test t/s master t/s 1b2cf95 Speedup
P40 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 824.38 823.69 1.00
P40 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 1224.06 1230.95 1.01
P40 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 1553.69 1564.53 1.01
P40 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 2350.79 2368.05 1.01
P40 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 3085.36 3104.34 1.01
P40 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 3512.50 3526.83 1.00
P40 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 3913.13 3933.90 1.01
P40 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 4006.41 4029.21 1.01
P40 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 806.08 816.74 1.01
P40 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 1163.85 1167.78 1.00
P40 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 1583.78 1592.94 1.01
P40 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 2349.25 2360.74 1.00
P40 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 3050.04 3061.39 1.00
P40 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 3465.65 3478.98 1.00
P40 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 3853.52 3867.74 1.00
P40 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 3910.16 3915.32 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 807.87 816.82 1.01
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 1163.21 1163.19 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 1574.50 1578.74 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 2333.17 2340.75 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 3043.40 3052.09 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 3475.65 3485.95 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 3880.02 3894.10 1.00
P40 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 3962.55 3978.94 1.00
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 837.90 850.86 1.02
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 1250.02 1261.43 1.01
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 1609.86 1589.39 0.99
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 2394.60 2376.97 0.99
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 3126.08 3108.97 0.99
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 3562.19 3533.21 0.99
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 3974.34 3951.84 0.99
P40 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 4057.39 4042.74 1.00
P40 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 792.62 792.44 1.00
P40 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 1232.80 1239.09 1.01
P40 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 1576.86 1605.57 1.02
P40 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 2348.76 2379.03 1.01
P40 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 3052.19 3085.25 1.01
P40 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 3448.96 3488.44 1.01
P40 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 3844.03 3878.92 1.01
P40 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 3907.54 3931.65 1.01
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 779.26 783.78 1.01
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 1190.43 1203.73 1.01
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 1520.57 1552.72 1.02
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 2296.02 2330.70 1.02
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 3015.63 3052.96 1.01
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 3440.97 3478.08 1.01
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 3842.11 3878.93 1.01
P40 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 3911.42 3939.98 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 793.65 801.43 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 1214.49 1229.63 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 1574.87 1589.72 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 2348.07 2365.47 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 3053.29 3071.54 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 3466.30 3484.62 1.01
P40 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 3864.42 3878.91 1.00
P40 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 3929.56 3941.20 1.00
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 819.42 834.79 1.02
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 1243.64 1255.57 1.01
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 1615.95 1608.59 1.00
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 2388.05 2387.06 1.00
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 3088.28 3086.97 1.00
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 3499.87 3496.85 1.00
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 3889.68 3883.87 1.00
P40 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 3952.29 3943.71 1.00
P40 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 868.51 881.96 1.02
P40 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 1339.93 1346.77 1.01
P40 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 1723.22 1727.01 1.00
P40 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 2523.38 2524.59 1.00
P40 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 3234.78 3238.74 1.00
P40 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 3649.01 3652.38 1.00
P40 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 4045.38 4053.78 1.00
P40 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 4118.22 4112.88 1.00
P40 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 875.54 881.61 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 1333.86 1346.18 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 1716.90 1734.80 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 2516.13 2541.96 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 3222.51 3254.96 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 3635.37 3668.74 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 4034.82 4069.15 1.01
P40 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 4111.32 4138.48 1.01
P40 granitemoe 3B Q2_K_M 16 pp2048 746.29 755.06 1.01
P40 granitemoe 3B Q2_K_M 32 pp2048 1084.13 1106.16 1.02
P40 granitemoe 3B Q2_K_M 64 pp2048 1389.06 1397.05 1.01
P40 granitemoe 3B Q2_K_M 128 pp2048 2105.48 2120.44 1.01
P40 granitemoe 3B Q2_K_M 256 pp2048 2811.75 2831.35 1.01
P40 granitemoe 3B Q2_K_M 512 pp2048 3274.31 3294.04 1.01
P40 granitemoe 3B Q2_K_M 1024 pp2048 3689.86 3687.33 1.00
P40 granitemoe 3B Q2_K_M 2048 pp2048 3725.82 3789.75 1.02
P40 granitemoe 3B Q3_K_S 16 pp2048 804.99 815.23 1.01
P40 granitemoe 3B Q3_K_S 32 pp2048 1135.02 1152.32 1.02
P40 granitemoe 3B Q3_K_S 64 pp2048 1502.95 1507.70 1.00
P40 granitemoe 3B Q3_K_S 128 pp2048 2220.24 2216.98 1.00
P40 granitemoe 3B Q3_K_S 256 pp2048 2843.20 2858.36 1.01
P40 granitemoe 3B Q3_K_S 512 pp2048 3248.19 3272.19 1.01
P40 granitemoe 3B Q3_K_S 1024 pp2048 3610.53 3638.29 1.01
P40 granitemoe 3B Q3_K_S 2048 pp2048 3700.37 3725.19 1.01
P40 granitemoe 3B Q4_0 16 pp2048 918.60 920.43 1.00
P40 granitemoe 3B Q4_0 32 pp2048 1284.81 1305.47 1.02
P40 granitemoe 3B Q4_0 64 pp2048 1722.13 1744.00 1.01
P40 granitemoe 3B Q4_0 128 pp2048 2534.30 2565.84 1.01
P40 granitemoe 3B Q4_0 256 pp2048 3254.92 3299.77 1.01
P40 granitemoe 3B Q4_0 512 pp2048 3675.09 3703.93 1.01
P40 granitemoe 3B Q4_0 1024 pp2048 4058.63 4100.49 1.01
P40 granitemoe 3B Q4_0 2048 pp2048 4097.14 4187.56 1.02
P40 granitemoe 3B Q4_1 16 pp2048 922.30 921.72 1.00
P40 granitemoe 3B Q4_1 32 pp2048 1287.50 1299.31 1.01
P40 granitemoe 3B Q4_1 64 pp2048 1709.13 1710.00 1.00
P40 granitemoe 3B Q4_1 128 pp2048 2510.90 2516.28 1.00
P40 granitemoe 3B Q4_1 256 pp2048 3229.30 3238.44 1.00
P40 granitemoe 3B Q4_1 512 pp2048 3655.74 3654.23 1.00
P40 granitemoe 3B Q4_1 1024 pp2048 4039.78 4044.54 1.00
P40 granitemoe 3B Q4_1 2048 pp2048 4115.91 4126.19 1.00
P40 granitemoe 3B Q4_K_S 16 pp2048 861.78 865.14 1.00
P40 granitemoe 3B Q4_K_S 32 pp2048 1211.37 1223.31 1.01
P40 granitemoe 3B Q4_K_S 64 pp2048 1588.73 1605.70 1.01
P40 granitemoe 3B Q4_K_S 128 pp2048 2369.78 2394.70 1.01
P40 granitemoe 3B Q4_K_S 256 pp2048 3080.26 3107.04 1.01
P40 granitemoe 3B Q4_K_S 512 pp2048 3489.11 3525.04 1.01
P40 granitemoe 3B Q4_K_S 1024 pp2048 3871.10 3920.31 1.01
P40 granitemoe 3B Q4_K_S 2048 pp2048 3963.45 4015.70 1.01
P40 granitemoe 3B Q5_0 16 pp2048 804.76 813.85 1.01
P40 granitemoe 3B Q5_0 32 pp2048 1238.23 1241.31 1.00
P40 granitemoe 3B Q5_0 64 pp2048 1634.10 1637.86 1.00
P40 granitemoe 3B Q5_0 128 pp2048 2417.48 2421.47 1.00
P40 granitemoe 3B Q5_0 256 pp2048 3111.74 3103.43 1.00
P40 granitemoe 3B Q5_0 512 pp2048 3505.99 3503.90 1.00
P40 granitemoe 3B Q5_0 1024 pp2048 3826.71 3887.07 1.02
P40 granitemoe 3B Q5_0 2048 pp2048 3883.94 3978.66 1.02
P40 granitemoe 3B Q5_1 16 pp2048 822.22 841.91 1.02
P40 granitemoe 3B Q5_1 32 pp2048 1261.11 1266.04 1.00
P40 granitemoe 3B Q5_1 64 pp2048 1616.20 1625.09 1.01
P40 granitemoe 3B Q5_1 128 pp2048 2396.65 2410.09 1.01
P40 granitemoe 3B Q5_1 256 pp2048 3087.01 3083.20 1.00
P40 granitemoe 3B Q5_1 512 pp2048 3478.34 3481.46 1.00
P40 granitemoe 3B Q5_1 1024 pp2048 3849.95 3876.69 1.01
P40 granitemoe 3B Q5_1 2048 pp2048 3931.74 3886.06 0.99
P40 granitemoe 3B Q5_K_S 16 pp2048 753.94 762.92 1.01
P40 granitemoe 3B Q5_K_S 32 pp2048 1142.97 1138.56 1.00
P40 granitemoe 3B Q5_K_S 64 pp2048 1515.59 1529.45 1.01
P40 granitemoe 3B Q5_K_S 128 pp2048 2277.43 2294.58 1.01
P40 granitemoe 3B Q5_K_S 256 pp2048 2966.80 2986.43 1.01
P40 granitemoe 3B Q5_K_S 512 pp2048 3389.64 3402.31 1.00
P40 granitemoe 3B Q5_K_S 1024 pp2048 3717.16 3790.98 1.02
P40 granitemoe 3B Q5_K_S 2048 pp2048 3869.10 3894.21 1.01
P40 granitemoe 3B Q6_K 16 pp2048 742.73 789.64 1.06
P40 granitemoe 3B Q6_K 32 pp2048 1126.40 1132.26 1.01
P40 granitemoe 3B Q6_K 64 pp2048 1483.96 1502.91 1.01
P40 granitemoe 3B Q6_K 128 pp2048 2205.21 2227.10 1.01
P40 granitemoe 3B Q6_K 256 pp2048 2850.03 2900.96 1.02
P40 granitemoe 3B Q6_K 512 pp2048 3265.57 3308.58 1.01
P40 granitemoe 3B Q6_K 1024 pp2048 3636.07 3651.15 1.00
P40 granitemoe 3B Q6_K 2048 pp2048 3722.18 3713.87 1.00
P40 granitemoe 3B Q8_0 16 pp2048 787.01 786.82 1.00
P40 granitemoe 3B Q8_0 32 pp2048 1236.75 1246.64 1.01
P40 granitemoe 3B Q8_0 64 pp2048 1639.38 1637.43 1.00
P40 granitemoe 3B Q8_0 128 pp2048 2441.47 2441.91 1.00
P40 granitemoe 3B Q8_0 256 pp2048 3159.49 3158.21 1.00
P40 granitemoe 3B Q8_0 512 pp2048 3570.76 3544.57 0.99
P40 granitemoe 3B Q8_0 1024 pp2048 3987.18 3976.16 1.00
P40 granitemoe 3B Q8_0 2048 pp2048 4080.25 4082.51 1.00
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 1886.35 2210.88 1.17
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 3067.81 3427.05 1.12
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 4377.14 4820.17 1.10
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 5461.81 5992.29 1.10
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 7862.29 8869.61 1.13
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 9847.04 12298.24 1.25
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 11083.57 14279.69 1.29
RTX 3090 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 11703.84 15659.94 1.34
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 1771.91 2072.46 1.17
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 2835.64 3304.31 1.17
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 4133.22 4566.10 1.10
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 5141.58 5610.63 1.09
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 7154.91 8392.86 1.17
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 9030.21 11506.08 1.27
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 10010.44 13136.91 1.31
RTX 3090 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 10494.04 13834.71 1.32
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 1801.96 2101.09 1.17
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 2859.77 3318.40 1.16
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 4078.25 4512.08 1.11
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 5008.05 5496.93 1.10
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 7203.13 8207.79 1.14
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 9051.83 11384.12 1.26
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 10443.19 13388.71 1.28
RTX 3090 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 10849.60 14590.48 1.34
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 1877.38 2205.11 1.17
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 3028.94 3502.95 1.16
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 4556.12 5216.74 1.14
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 5813.30 6319.61 1.09
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 8256.06 9341.14 1.13
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 10431.24 12792.62 1.23
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 11753.91 14817.95 1.26
RTX 3090 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 12344.58 15945.51 1.29
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 1753.29 2030.45 1.16
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 2826.96 3289.36 1.16
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 4366.53 4870.29 1.12
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 5796.95 6324.11 1.09
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 8239.07 9365.33 1.14
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 10034.46 12571.58 1.25
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 11128.75 14157.13 1.27
RTX 3090 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 11457.23 14832.56 1.29
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 1753.65 2031.52 1.16
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 2831.37 3281.89 1.16
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 4337.94 4829.35 1.11
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 5654.27 6138.64 1.09
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 8023.24 9097.11 1.13
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 9978.67 12349.15 1.24
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 11110.06 14139.17 1.27
RTX 3090 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 11337.23 14899.22 1.31
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 1776.41 2069.89 1.17
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 2878.71 3340.93 1.16
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 4389.97 4910.63 1.12
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 5728.76 6271.40 1.09
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 8131.34 9271.07 1.14
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 10090.06 12508.81 1.24
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 11135.20 14202.04 1.28
RTX 3090 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 11459.31 14693.76 1.28
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 1804.07 2086.38 1.16
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 2941.47 3374.81 1.15
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 4397.93 4904.02 1.12
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 5663.63 6209.21 1.10
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 8048.52 9166.89 1.14
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 9991.83 12382.56 1.24
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 11095.93 14123.53 1.27
RTX 3090 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 11421.94 14907.90 1.31
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 1845.96 2219.94 1.20
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 3013.17 3565.39 1.18
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 4574.43 5208.09 1.14
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 5964.37 6617.76 1.11
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 8530.69 9821.20 1.15
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 10664.34 13220.37 1.24
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 11889.47 15072.46 1.27
RTX 3090 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 12404.41 16161.44 1.30
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 1961.51 2276.23 1.16
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 3164.11 3665.37 1.16
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 4781.92 5333.95 1.12
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 6161.31 6713.93 1.09
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 8817.97 9934.50 1.13
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 10889.99 13362.06 1.23
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 12205.31 15354.55 1.26
RTX 3090 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 12822.16 16603.73 1.29
RTX 3090 granitemoe 3B Q2_K_M 16 pp2048 1854.86 2131.37 1.15
RTX 3090 granitemoe 3B Q2_K_M 32 pp2048 2855.02 3205.69 1.12
RTX 3090 granitemoe 3B Q2_K_M 64 pp2048 3808.88 4148.37 1.09
RTX 3090 granitemoe 3B Q2_K_M 128 pp2048 4377.92 4985.14 1.14
RTX 3090 granitemoe 3B Q2_K_M 256 pp2048 6382.69 7439.92 1.17
RTX 3090 granitemoe 3B Q2_K_M 512 pp2048 8080.64 10426.54 1.29
RTX 3090 granitemoe 3B Q2_K_M 1024 pp2048 9286.91 12365.60 1.33
RTX 3090 granitemoe 3B Q2_K_M 2048 pp2048 10046.22 13717.45 1.37
RTX 3090 granitemoe 3B Q3_K_S 16 pp2048 1838.35 2118.71 1.15
RTX 3090 granitemoe 3B Q3_K_S 32 pp2048 2937.79 3353.38 1.14
RTX 3090 granitemoe 3B Q3_K_S 64 pp2048 4156.22 4554.99 1.10
RTX 3090 granitemoe 3B Q3_K_S 128 pp2048 5183.11 5652.74 1.09
RTX 3090 granitemoe 3B Q3_K_S 256 pp2048 7393.74 8388.97 1.13
RTX 3090 granitemoe 3B Q3_K_S 512 pp2048 9147.24 11530.24 1.26
RTX 3090 granitemoe 3B Q3_K_S 1024 pp2048 10381.18 13497.41 1.30
RTX 3090 granitemoe 3B Q3_K_S 2048 pp2048 10908.34 14856.11 1.36
RTX 3090 granitemoe 3B Q4_0 16 pp2048 1862.21 2188.56 1.18
RTX 3090 granitemoe 3B Q4_0 32 pp2048 3008.66 3545.84 1.18
RTX 3090 granitemoe 3B Q4_0 64 pp2048 4546.84 5094.07 1.12
RTX 3090 granitemoe 3B Q4_0 128 pp2048 5835.19 6437.98 1.10
RTX 3090 granitemoe 3B Q4_0 256 pp2048 8419.48 9545.73 1.13
RTX 3090 granitemoe 3B Q4_0 512 pp2048 10385.04 12840.97 1.24
RTX 3090 granitemoe 3B Q4_0 1024 pp2048 11723.97 14898.08 1.27
RTX 3090 granitemoe 3B Q4_0 2048 pp2048 12263.62 16222.65 1.32
RTX 3090 granitemoe 3B Q4_1 16 pp2048 1891.18 2209.84 1.17
RTX 3090 granitemoe 3B Q4_1 32 pp2048 3123.09 3591.06 1.15
RTX 3090 granitemoe 3B Q4_1 64 pp2048 4475.56 4991.88 1.12
RTX 3090 granitemoe 3B Q4_1 128 pp2048 5590.28 6129.15 1.10
RTX 3090 granitemoe 3B Q4_1 256 pp2048 8063.43 9130.43 1.13
RTX 3090 granitemoe 3B Q4_1 512 pp2048 10042.05 12367.42 1.23
RTX 3090 granitemoe 3B Q4_1 1024 pp2048 11412.03 14368.49 1.26
RTX 3090 granitemoe 3B Q4_1 2048 pp2048 12117.66 15894.07 1.31
RTX 3090 granitemoe 3B Q4_K_S 16 pp2048 1959.73 2301.36 1.17
RTX 3090 granitemoe 3B Q4_K_S 32 pp2048 3204.44 3616.49 1.13
RTX 3090 granitemoe 3B Q4_K_S 64 pp2048 4529.78 5015.29 1.11
RTX 3090 granitemoe 3B Q4_K_S 128 pp2048 5638.05 6220.84 1.10
RTX 3090 granitemoe 3B Q4_K_S 256 pp2048 8136.40 9239.79 1.14
RTX 3090 granitemoe 3B Q4_K_S 512 pp2048 10104.70 12520.46 1.24
RTX 3090 granitemoe 3B Q4_K_S 1024 pp2048 11496.52 14541.02 1.26
RTX 3090 granitemoe 3B Q4_K_S 2048 pp2048 12232.63 16037.85 1.31
RTX 3090 granitemoe 3B Q5_0 16 pp2048 1673.88 1981.95 1.18
RTX 3090 granitemoe 3B Q5_0 32 pp2048 2870.43 3207.53 1.12
RTX 3090 granitemoe 3B Q5_0 64 pp2048 4301.38 4778.20 1.11
RTX 3090 granitemoe 3B Q5_0 128 pp2048 5614.13 6158.56 1.10
RTX 3090 granitemoe 3B Q5_0 256 pp2048 8120.45 9180.71 1.13
RTX 3090 granitemoe 3B Q5_0 512 pp2048 10048.00 12354.50 1.23
RTX 3090 granitemoe 3B Q5_0 1024 pp2048 11397.51 14355.15 1.26
RTX 3090 granitemoe 3B Q5_0 2048 pp2048 11951.96 15551.36 1.30
RTX 3090 granitemoe 3B Q5_1 16 pp2048 1776.44 2072.85 1.17
RTX 3090 granitemoe 3B Q5_1 32 pp2048 2963.81 3324.60 1.12
RTX 3090 granitemoe 3B Q5_1 64 pp2048 4223.63 4693.99 1.11
RTX 3090 granitemoe 3B Q5_1 128 pp2048 5360.53 5884.15 1.10
RTX 3090 granitemoe 3B Q5_1 256 pp2048 7793.04 8807.18 1.13
RTX 3090 granitemoe 3B Q5_1 512 pp2048 9775.80 12093.06 1.24
RTX 3090 granitemoe 3B Q5_1 1024 pp2048 11076.23 14010.26 1.26
RTX 3090 granitemoe 3B Q5_1 2048 pp2048 11650.25 15215.49 1.31
RTX 3090 granitemoe 3B Q5_K_S 16 pp2048 1873.07 2173.44 1.16
RTX 3090 granitemoe 3B Q5_K_S 32 pp2048 3071.68 3439.85 1.12
RTX 3090 granitemoe 3B Q5_K_S 64 pp2048 4361.56 4816.85 1.10
RTX 3090 granitemoe 3B Q5_K_S 128 pp2048 5504.66 6036.44 1.10
RTX 3090 granitemoe 3B Q5_K_S 256 pp2048 7923.04 9037.97 1.14
RTX 3090 granitemoe 3B Q5_K_S 512 pp2048 9822.26 12297.31 1.25
RTX 3090 granitemoe 3B Q5_K_S 1024 pp2048 11171.43 14288.28 1.28
RTX 3090 granitemoe 3B Q5_K_S 2048 pp2048 11764.47 15675.88 1.33
RTX 3090 granitemoe 3B Q6_K 16 pp2048 1692.68 1934.61 1.14
RTX 3090 granitemoe 3B Q6_K 32 pp2048 2738.66 3150.96 1.15
RTX 3090 granitemoe 3B Q6_K 64 pp2048 4055.55 4459.12 1.10
RTX 3090 granitemoe 3B Q6_K 128 pp2048 5165.04 5642.97 1.09
RTX 3090 granitemoe 3B Q6_K 256 pp2048 7406.79 8410.91 1.14
RTX 3090 granitemoe 3B Q6_K 512 pp2048 9197.48 11560.08 1.26
RTX 3090 granitemoe 3B Q6_K 1024 pp2048 10388.07 13371.87 1.29
RTX 3090 granitemoe 3B Q6_K 2048 pp2048 10968.15 14654.41 1.34
RTX 3090 granitemoe 3B Q8_0 16 pp2048 1658.72 1896.27 1.14
RTX 3090 granitemoe 3B Q8_0 32 pp2048 2922.32 3318.58 1.14
RTX 3090 granitemoe 3B Q8_0 64 pp2048 4408.38 4853.91 1.10
RTX 3090 granitemoe 3B Q8_0 128 pp2048 5805.72 6292.41 1.08
RTX 3090 granitemoe 3B Q8_0 256 pp2048 8338.61 9492.75 1.14
RTX 3090 granitemoe 3B Q8_0 512 pp2048 10459.22 12789.85 1.22
RTX 3090 granitemoe 3B Q8_0 1024 pp2048 11806.51 14780.34 1.25
RTX 3090 granitemoe 3B Q8_0 2048 pp2048 12451.64 15980.50 1.28
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 2693.45 2859.84 1.06
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 4865.08 5219.21 1.07
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 7834.19 8309.44 1.06
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 11069.85 11701.18 1.06
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 16317.05 17266.81 1.06
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 21333.64 28060.87 1.32
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 24428.78 34252.36 1.40
RTX 4090 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 21959.22 30189.49 1.37
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 2662.30 2863.66 1.08
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 4631.92 5042.15 1.09
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 7518.35 8030.84 1.07
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 10548.65 11101.56 1.05
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 15086.26 15969.16 1.06
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 19303.52 25769.32 1.33
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 21362.30 30150.16 1.41
RTX 4090 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 18732.55 25546.14 1.36
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 2666.18 2893.78 1.09
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 4626.31 5066.14 1.10
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 7453.84 7991.15 1.07
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 10391.57 11007.34 1.06
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 15040.75 15962.95 1.06
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 19711.28 26496.25 1.34
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 22518.61 32546.06 1.45
RTX 4090 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 20479.28 28959.90 1.41
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 2755.92 2965.66 1.08
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 4858.57 5277.08 1.09
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 8122.85 8691.34 1.07
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 11656.66 12320.64 1.06
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 17100.49 18105.91 1.06
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 22656.94 29413.55 1.30
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 25738.34 35330.14 1.37
RTX 4090 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 23063.10 30696.56 1.33
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 2582.38 2753.96 1.07
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 4486.94 4914.57 1.10
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 7717.41 8267.29 1.07
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 11423.22 12016.98 1.05
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 16522.98 17537.14 1.06
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 21645.60 28512.29 1.32
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 23564.42 32255.45 1.37
RTX 4090 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 20207.42 26647.88 1.32
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 2580.51 2764.87 1.07
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 4493.01 4921.31 1.10
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 7675.32 8228.47 1.07
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 11214.72 11792.71 1.05
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 16097.30 17088.12 1.06
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 21203.05 27809.93 1.31
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 23306.35 31783.65 1.36
RTX 4090 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 20190.85 26545.83 1.31
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 2626.88 2799.31 1.07
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 4596.28 4965.96 1.08
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 7785.83 8311.21 1.07
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 11381.18 11959.85 1.05
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 16515.33 17441.73 1.06
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 21542.39 28145.16 1.31
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 23532.12 31979.28 1.36
RTX 4090 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 20320.28 26694.28 1.31
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 2651.09 2864.54 1.08
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 4665.15 5087.35 1.09
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 7756.60 8331.18 1.07
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 11262.02 11867.20 1.05
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 16033.47 17045.64 1.06
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 21248.07 27362.39 1.29
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 23578.97 31877.80 1.35
RTX 4090 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 20235.76 26403.85 1.30
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 2565.74 2839.47 1.11
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 4549.94 4985.73 1.10
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 7822.97 8578.74 1.10
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 11502.74 12397.94 1.08
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 17317.77 18516.40 1.07
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 23361.11 30305.99 1.30
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 25915.90 35160.29 1.36
RTX 4090 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 22745.62 30133.45 1.32
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 2720.81 2948.56 1.08
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 4822.63 5217.76 1.08
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 8215.53 8805.02 1.07
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 11869.78 12507.77 1.05
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 17762.72 18618.92 1.05
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 23894.75 30699.04 1.28
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 26721.54 36086.75 1.35
RTX 4090 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 23381.53 30800.25 1.32
RTX 4090 granitemoe 3B Q2_K_M 16 pp2048 2723.06 2952.61 1.08
RTX 4090 granitemoe 3B Q2_K_M 32 pp2048 4629.39 5022.50 1.08
RTX 4090 granitemoe 3B Q2_K_M 64 pp2048 7064.64 7510.29 1.06
RTX 4090 granitemoe 3B Q2_K_M 128 pp2048 8828.52 9225.40 1.04
RTX 4090 granitemoe 3B Q2_K_M 256 pp2048 13309.48 13889.39 1.04
RTX 4090 granitemoe 3B Q2_K_M 512 pp2048 18158.91 24666.23 1.36
RTX 4090 granitemoe 3B Q2_K_M 1024 pp2048 20644.57 30196.15 1.46
RTX 4090 granitemoe 3B Q2_K_M 2048 pp2048 19159.52 27469.90 1.43
RTX 4090 granitemoe 3B Q3_K_S 16 pp2048 2686.01 2917.87 1.09
RTX 4090 granitemoe 3B Q3_K_S 32 pp2048 4677.85 5082.94 1.09
RTX 4090 granitemoe 3B Q3_K_S 64 pp2048 7506.26 7985.98 1.06
RTX 4090 granitemoe 3B Q3_K_S 128 pp2048 10558.64 11038.57 1.05
RTX 4090 granitemoe 3B Q3_K_S 256 pp2048 15349.01 16132.74 1.05
RTX 4090 granitemoe 3B Q3_K_S 512 pp2048 20308.76 27513.88 1.35
RTX 4090 granitemoe 3B Q3_K_S 1024 pp2048 22946.98 33172.80 1.45
RTX 4090 granitemoe 3B Q3_K_S 2048 pp2048 20944.39 29727.38 1.42
RTX 4090 granitemoe 3B Q4_0 16 pp2048 2616.24 2876.10 1.10
RTX 4090 granitemoe 3B Q4_0 32 pp2048 4544.92 5095.97 1.12
RTX 4090 granitemoe 3B Q4_0 64 pp2048 7746.41 8407.11 1.09
RTX 4090 granitemoe 3B Q4_0 128 pp2048 11294.09 12140.49 1.07
RTX 4090 granitemoe 3B Q4_0 256 pp2048 17004.78 18148.37 1.07
RTX 4090 granitemoe 3B Q4_0 512 pp2048 23081.19 30165.26 1.31
RTX 4090 granitemoe 3B Q4_0 1024 pp2048 26145.20 35744.12 1.37
RTX 4090 granitemoe 3B Q4_0 2048 pp2048 23402.65 31575.89 1.35
RTX 4090 granitemoe 3B Q4_1 16 pp2048 2594.17 2821.06 1.09
RTX 4090 granitemoe 3B Q4_1 32 pp2048 4692.49 5184.95 1.10
RTX 4090 granitemoe 3B Q4_1 64 pp2048 7600.07 8297.42 1.09
RTX 4090 granitemoe 3B Q4_1 128 pp2048 10872.11 11685.65 1.07
RTX 4090 granitemoe 3B Q4_1 256 pp2048 16313.75 17454.34 1.07
RTX 4090 granitemoe 3B Q4_1 512 pp2048 22064.34 29192.82 1.32
RTX 4090 granitemoe 3B Q4_1 1024 pp2048 24964.23 34841.51 1.40
RTX 4090 granitemoe 3B Q4_1 2048 pp2048 22690.68 31117.95 1.37
RTX 4090 granitemoe 3B Q4_K_S 16 pp2048 2713.66 2900.08 1.07
RTX 4090 granitemoe 3B Q4_K_S 32 pp2048 4897.21 5249.24 1.07
RTX 4090 granitemoe 3B Q4_K_S 64 pp2048 7843.56 8361.74 1.07
RTX 4090 granitemoe 3B Q4_K_S 128 pp2048 11172.10 11792.47 1.06
RTX 4090 granitemoe 3B Q4_K_S 256 pp2048 16633.15 17631.57 1.06
RTX 4090 granitemoe 3B Q4_K_S 512 pp2048 22403.40 29363.51 1.31
RTX 4090 granitemoe 3B Q4_K_S 1024 pp2048 25353.28 35190.33 1.39
RTX 4090 granitemoe 3B Q4_K_S 2048 pp2048 23010.11 31245.84 1.36
RTX 4090 granitemoe 3B Q5_0 16 pp2048 2394.08 2671.25 1.12
RTX 4090 granitemoe 3B Q5_0 32 pp2048 4425.29 4740.95 1.07
RTX 4090 granitemoe 3B Q5_0 64 pp2048 7349.07 7963.70 1.08
RTX 4090 granitemoe 3B Q5_0 128 pp2048 10875.64 11622.85 1.07
RTX 4090 granitemoe 3B Q5_0 256 pp2048 16345.01 17416.81 1.07
RTX 4090 granitemoe 3B Q5_0 512 pp2048 22000.56 29082.12 1.32
RTX 4090 granitemoe 3B Q5_0 1024 pp2048 24667.61 34278.33 1.39
RTX 4090 granitemoe 3B Q5_0 2048 pp2048 22243.55 30107.13 1.35
RTX 4090 granitemoe 3B Q5_1 16 pp2048 2431.61 2683.74 1.10
RTX 4090 granitemoe 3B Q5_1 32 pp2048 4463.87 4848.59 1.09
RTX 4090 granitemoe 3B Q5_1 64 pp2048 7271.57 7857.28 1.08
RTX 4090 granitemoe 3B Q5_1 128 pp2048 10470.23 11244.05 1.07
RTX 4090 granitemoe 3B Q5_1 256 pp2048 15821.49 16866.11 1.07
RTX 4090 granitemoe 3B Q5_1 512 pp2048 21442.44 28333.67 1.32
RTX 4090 granitemoe 3B Q5_1 1024 pp2048 24030.91 33462.28 1.39
RTX 4090 granitemoe 3B Q5_1 2048 pp2048 21803.50 29710.75 1.36
RTX 4090 granitemoe 3B Q5_K_S 16 pp2048 2606.53 2776.70 1.07
RTX 4090 granitemoe 3B Q5_K_S 32 pp2048 4677.43 4976.88 1.06
RTX 4090 granitemoe 3B Q5_K_S 64 pp2048 7572.09 7991.96 1.06
RTX 4090 granitemoe 3B Q5_K_S 128 pp2048 10888.70 11432.56 1.05
RTX 4090 granitemoe 3B Q5_K_S 256 pp2048 16294.90 17178.21 1.05
RTX 4090 granitemoe 3B Q5_K_S 512 pp2048 21909.54 28907.53 1.32
RTX 4090 granitemoe 3B Q5_K_S 1024 pp2048 24656.27 34528.16 1.40
RTX 4090 granitemoe 3B Q5_K_S 2048 pp2048 22391.31 30809.91 1.38
RTX 4090 granitemoe 3B Q6_K 16 pp2048 2400.52 2584.19 1.08
RTX 4090 granitemoe 3B Q6_K 32 pp2048 4170.36 4503.85 1.08
RTX 4090 granitemoe 3B Q6_K 64 pp2048 7026.03 7447.76 1.06
RTX 4090 granitemoe 3B Q6_K 128 pp2048 10237.10 10696.46 1.04
RTX 4090 granitemoe 3B Q6_K 256 pp2048 15169.63 15925.06 1.05
RTX 4090 granitemoe 3B Q6_K 512 pp2048 20369.38 27178.26 1.33
RTX 4090 granitemoe 3B Q6_K 1024 pp2048 22668.63 31965.50 1.41
RTX 4090 granitemoe 3B Q6_K 2048 pp2048 20614.62 28359.24 1.38
RTX 4090 granitemoe 3B Q8_0 16 pp2048 2224.47 2425.07 1.09
RTX 4090 granitemoe 3B Q8_0 32 pp2048 4053.52 4378.33 1.08
RTX 4090 granitemoe 3B Q8_0 64 pp2048 6843.04 7331.57 1.07
RTX 4090 granitemoe 3B Q8_0 128 pp2048 10528.33 11151.24 1.06
RTX 4090 granitemoe 3B Q8_0 256 pp2048 16357.54 17283.25 1.06
RTX 4090 granitemoe 3B Q8_0 512 pp2048 22697.10 29795.27 1.31
RTX 4090 granitemoe 3B Q8_0 1024 pp2048 25774.77 35185.37 1.37
RTX 4090 granitemoe 3B Q8_0 2048 pp2048 23379.88 31396.93 1.34
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 2354.63 2633.05 1.12
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 3893.59 4431.47 1.14
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 6510.02 7308.92 1.12
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 9812.36 10915.20 1.11
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 14417.90 16635.21 1.15
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 19300.43 25632.41 1.33
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 22401.85 32435.56 1.45
RTX 5090 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 22662.39 33546.27 1.48
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 2311.28 2548.44 1.10
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 3732.28 4293.71 1.15
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 6258.99 7001.50 1.12
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 9186.79 10089.39 1.10
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 13104.57 15399.27 1.18
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 16935.71 22484.83 1.33
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 19068.80 27785.69 1.46
RTX 5090 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 18465.64 27437.24 1.49
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 2312.95 2546.96 1.10
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 3718.95 4283.78 1.15
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 6196.86 6989.50 1.13
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 9060.32 9962.92 1.10
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 13047.02 15188.98 1.16
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 17165.51 22989.23 1.34
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 19820.58 29371.54 1.48
RTX 5090 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 20085.78 31186.05 1.55
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 2387.44 2656.33 1.11
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 3955.51 4495.56 1.14
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 6665.54 7492.76 1.12
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 10270.15 11419.65 1.11
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 15227.76 17502.68 1.15
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 20570.66 26693.95 1.30
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 24088.31 33616.82 1.40
RTX 5090 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 24269.14 34651.40 1.43
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 2281.85 2481.60 1.09
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 3796.53 4342.85 1.14
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 6503.35 7238.95 1.11
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 10039.58 11118.99 1.11
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 14758.79 16937.76 1.15
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 19529.80 25056.04 1.28
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 22018.00 30374.74 1.38
RTX 5090 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 21055.29 29189.05 1.39
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 2284.25 2508.44 1.10
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 3805.72 4343.31 1.14
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 6464.38 7230.32 1.12
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 9996.59 11097.67 1.11
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 14670.17 16779.14 1.14
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 19330.66 24554.89 1.27
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 22032.19 30081.94 1.37
RTX 5090 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 20905.36 28649.13 1.37
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 2311.50 2516.52 1.09
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 3831.04 4375.54 1.14
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 6536.10 7278.72 1.11
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 10127.45 11180.52 1.10
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 14843.91 17098.45 1.15
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 19684.43 24943.07 1.27
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 22319.87 30479.22 1.37
RTX 5090 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 21083.31 29145.01 1.38
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 2337.43 2545.40 1.09
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 3842.28 4403.27 1.15
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 6571.23 7354.14 1.12
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 9996.03 11039.16 1.10
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 14697.68 16977.77 1.16
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 19414.06 24528.28 1.26
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 22036.31 30082.09 1.37
RTX 5090 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 21109.09 29102.97 1.38
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 2304.38 2673.90 1.16
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 3803.35 4483.10 1.18
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 6536.77 7525.99 1.15
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 10052.33 11210.53 1.12
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 15023.86 17499.73 1.16
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 20797.45 26355.11 1.27
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 24007.54 32914.18 1.37
RTX 5090 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 23934.94 33140.45 1.38
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 2394.30 2688.47 1.12
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 3968.18 4493.28 1.13
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 6750.74 7596.47 1.13
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 10276.16 11317.97 1.10
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 15164.42 17551.73 1.16
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 20545.87 26550.81 1.29
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 23714.65 33393.74 1.41
RTX 5090 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 23536.24 33777.71 1.44
RTX 5090 granitemoe 3B Q2_K_M 16 pp2048 2341.49 2553.65 1.09
RTX 5090 granitemoe 3B Q2_K_M 32 pp2048 3712.46 4275.23 1.15
RTX 5090 granitemoe 3B Q2_K_M 64 pp2048 6068.20 6734.68 1.11
RTX 5090 granitemoe 3B Q2_K_M 128 pp2048 8540.86 9593.64 1.12
RTX 5090 granitemoe 3B Q2_K_M 256 pp2048 12112.53 14392.50 1.19
RTX 5090 granitemoe 3B Q2_K_M 512 pp2048 16050.53 21775.14 1.36
RTX 5090 granitemoe 3B Q2_K_M 1024 pp2048 18520.83 28231.83 1.52
RTX 5090 granitemoe 3B Q2_K_M 2048 pp2048 18849.30 30114.90 1.60
RTX 5090 granitemoe 3B Q3_K_S 16 pp2048 2332.07 2522.61 1.08
RTX 5090 granitemoe 3B Q3_K_S 32 pp2048 3757.00 4308.29 1.15
RTX 5090 granitemoe 3B Q3_K_S 64 pp2048 6251.93 6983.10 1.12
RTX 5090 granitemoe 3B Q3_K_S 128 pp2048 9229.20 10058.08 1.09
RTX 5090 granitemoe 3B Q3_K_S 256 pp2048 13346.49 15345.06 1.15
RTX 5090 granitemoe 3B Q3_K_S 512 pp2048 17590.97 23159.10 1.32
RTX 5090 granitemoe 3B Q3_K_S 1024 pp2048 20326.08 29828.68 1.47
RTX 5090 granitemoe 3B Q3_K_S 2048 pp2048 20641.31 32044.70 1.55
RTX 5090 granitemoe 3B Q4_0 16 pp2048 2288.73 2670.44 1.17
RTX 5090 granitemoe 3B Q4_0 32 pp2048 3787.56 4434.22 1.17
RTX 5090 granitemoe 3B Q4_0 64 pp2048 6470.36 7549.64 1.17
RTX 5090 granitemoe 3B Q4_0 128 pp2048 9992.55 11186.50 1.12
RTX 5090 granitemoe 3B Q4_0 256 pp2048 14909.78 17326.30 1.16
RTX 5090 granitemoe 3B Q4_0 512 pp2048 20736.17 26501.18 1.28
RTX 5090 granitemoe 3B Q4_0 1024 pp2048 24417.04 33933.81 1.39
RTX 5090 granitemoe 3B Q4_0 2048 pp2048 24902.02 35466.82 1.42
RTX 5090 granitemoe 3B Q4_1 16 pp2048 2274.78 2668.99 1.17
RTX 5090 granitemoe 3B Q4_1 32 pp2048 3751.59 4408.60 1.18
RTX 5090 granitemoe 3B Q4_1 64 pp2048 6464.94 7307.32 1.13
RTX 5090 granitemoe 3B Q4_1 128 pp2048 9699.22 10870.28 1.12
RTX 5090 granitemoe 3B Q4_1 256 pp2048 14289.76 16624.06 1.16
RTX 5090 granitemoe 3B Q4_1 512 pp2048 19699.91 25491.18 1.29
RTX 5090 granitemoe 3B Q4_1 1024 pp2048 22877.84 32639.58 1.43
RTX 5090 granitemoe 3B Q4_1 2048 pp2048 23236.27 34430.60 1.48
RTX 5090 granitemoe 3B Q4_K_S 16 pp2048 2435.48 2661.32 1.09
RTX 5090 granitemoe 3B Q4_K_S 32 pp2048 3953.00 4490.54 1.14
RTX 5090 granitemoe 3B Q4_K_S 64 pp2048 6604.24 7456.22 1.13
RTX 5090 granitemoe 3B Q4_K_S 128 pp2048 9924.92 11030.32 1.11
RTX 5090 granitemoe 3B Q4_K_S 256 pp2048 14549.97 16835.49 1.16
RTX 5090 granitemoe 3B Q4_K_S 512 pp2048 19802.30 25503.82 1.29
RTX 5090 granitemoe 3B Q4_K_S 1024 pp2048 23218.01 32965.61 1.42
RTX 5090 granitemoe 3B Q4_K_S 2048 pp2048 23451.13 34592.34 1.48
RTX 5090 granitemoe 3B Q5_0 16 pp2048 2194.17 2508.59 1.14
RTX 5090 granitemoe 3B Q5_0 32 pp2048 3649.72 4297.64 1.18
RTX 5090 granitemoe 3B Q5_0 64 pp2048 6302.54 7092.34 1.13
RTX 5090 granitemoe 3B Q5_0 128 pp2048 9712.40 10752.87 1.11
RTX 5090 granitemoe 3B Q5_0 256 pp2048 14305.37 16674.07 1.17
RTX 5090 granitemoe 3B Q5_0 512 pp2048 19854.76 25297.45 1.27
RTX 5090 granitemoe 3B Q5_0 1024 pp2048 23056.84 32026.60 1.39
RTX 5090 granitemoe 3B Q5_0 2048 pp2048 23111.84 33106.22 1.43
RTX 5090 granitemoe 3B Q5_1 16 pp2048 2251.31 2591.93 1.15
RTX 5090 granitemoe 3B Q5_1 32 pp2048 3638.56 4319.73 1.19
RTX 5090 granitemoe 3B Q5_1 64 pp2048 6332.97 7049.58 1.11
RTX 5090 granitemoe 3B Q5_1 128 pp2048 9370.88 10392.25 1.11
RTX 5090 granitemoe 3B Q5_1 256 pp2048 13808.50 15960.61 1.16
RTX 5090 granitemoe 3B Q5_1 512 pp2048 19060.25 24558.48 1.29
RTX 5090 granitemoe 3B Q5_1 1024 pp2048 21951.45 31222.75 1.42
RTX 5090 granitemoe 3B Q5_1 2048 pp2048 22005.38 32071.17 1.46
RTX 5090 granitemoe 3B Q5_K_S 16 pp2048 2355.57 2597.63 1.10
RTX 5090 granitemoe 3B Q5_K_S 32 pp2048 3873.11 4383.62 1.13
RTX 5090 granitemoe 3B Q5_K_S 64 pp2048 6501.59 7209.69 1.11
RTX 5090 granitemoe 3B Q5_K_S 128 pp2048 9673.80 10737.39 1.11
RTX 5090 granitemoe 3B Q5_K_S 256 pp2048 14229.39 16305.59 1.15
RTX 5090 granitemoe 3B Q5_K_S 512 pp2048 19395.52 25041.24 1.29
RTX 5090 granitemoe 3B Q5_K_S 1024 pp2048 22577.41 32198.50 1.43
RTX 5090 granitemoe 3B Q5_K_S 2048 pp2048 22966.96 34103.65 1.48
RTX 5090 granitemoe 3B Q6_K 16 pp2048 2256.83 2473.77 1.10
RTX 5090 granitemoe 3B Q6_K 32 pp2048 3655.61 4175.15 1.14
RTX 5090 granitemoe 3B Q6_K 64 pp2048 6018.54 6685.99 1.11
RTX 5090 granitemoe 3B Q6_K 128 pp2048 8812.40 9545.57 1.08
RTX 5090 granitemoe 3B Q6_K 256 pp2048 12759.39 14595.25 1.14
RTX 5090 granitemoe 3B Q6_K 512 pp2048 17057.81 22354.79 1.31
RTX 5090 granitemoe 3B Q6_K 1024 pp2048 19561.41 28062.20 1.43
RTX 5090 granitemoe 3B Q6_K 2048 pp2048 19688.20 29748.19 1.51
RTX 5090 granitemoe 3B Q8_0 16 pp2048 2146.53 2461.39 1.15
RTX 5090 granitemoe 3B Q8_0 32 pp2048 3511.24 4101.43 1.17
RTX 5090 granitemoe 3B Q8_0 64 pp2048 6166.54 6892.50 1.12
RTX 5090 granitemoe 3B Q8_0 128 pp2048 9522.60 10486.88 1.10
RTX 5090 granitemoe 3B Q8_0 256 pp2048 14483.97 16751.01 1.16
RTX 5090 granitemoe 3B Q8_0 512 pp2048 20356.14 26065.94 1.28
RTX 5090 granitemoe 3B Q8_0 1024 pp2048 23827.66 32764.78 1.38
RTX 5090 granitemoe 3B Q8_0 2048 pp2048 24215.97 34199.38 1.41
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 1015.44 1184.80 1.17
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 1538.31 1823.04 1.19
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 767.05 831.49 1.08
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 1173.34 1269.48 1.08
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 1760.18 1885.23 1.07
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 2558.01 2683.25 1.05
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 3571.88 3672.30 1.03
V100-PCIE-32GB granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 4470.86 4568.92 1.02
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 951.52 1122.63 1.18
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 1477.50 1770.69 1.20
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 754.39 814.94 1.08
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 1183.41 1267.88 1.07
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 1794.35 1906.04 1.06
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 2549.63 2672.25 1.05
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 3546.11 3652.16 1.03
V100-PCIE-32GB granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 4445.39 4526.57 1.02
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 1004.94 1171.51 1.17
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 1528.22 1810.81 1.18
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 745.05 804.02 1.08
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 1175.16 1258.82 1.07
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 1777.58 1890.20 1.06
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 2567.21 2685.61 1.05
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 3594.96 3693.87 1.03
V100-PCIE-32GB granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 4497.93 4600.08 1.02
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 1060.06 1230.41 1.16
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 1608.59 1874.43 1.17
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 755.52 818.53 1.08
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 1178.74 1265.05 1.07
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 1773.88 1889.51 1.07
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 2556.99 2670.16 1.04
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 3583.69 3680.07 1.03
V100-PCIE-32GB granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 4494.32 4590.37 1.02
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 1037.59 1208.14 1.16
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 1617.90 1874.68 1.16
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 741.74 797.36 1.07
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 1171.31 1249.25 1.07
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 1780.88 1881.07 1.06
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 2533.39 2646.81 1.04
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 3527.70 3624.61 1.03
V100-PCIE-32GB granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 4427.53 4494.33 1.02
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 958.70 1135.71 1.18
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 1488.48 1761.02 1.18
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 737.10 800.10 1.09
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 1159.86 1245.81 1.07
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 1767.75 1882.79 1.07
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 2536.63 2655.21 1.05
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 3513.52 3616.69 1.03
V100-PCIE-32GB granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 4408.87 4499.91 1.02
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 968.95 1167.59 1.21
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 1525.03 1808.83 1.19
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 741.38 801.67 1.08
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 1173.13 1256.69 1.07
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 1780.86 1892.45 1.06
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 2544.80 2656.78 1.04
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 3529.45 3636.34 1.03
V100-PCIE-32GB granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 4436.09 4501.27 1.01
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 964.92 1143.97 1.19
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 1525.19 1808.44 1.19
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 756.19 815.07 1.08
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 1190.79 1272.96 1.07
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 1810.05 1916.25 1.06
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 2556.52 2676.52 1.05
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 3536.09 3647.15 1.03
V100-PCIE-32GB granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 4438.54 4521.86 1.02
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 1162.24 1363.09 1.17
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 1744.54 2024.46 1.16
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 788.35 890.60 1.13
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 1256.02 1393.26 1.11
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 1916.76 2080.36 1.09
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 2712.86 2876.19 1.06
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 3739.99 3874.01 1.04
V100-PCIE-32GB granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 4666.24 4755.79 1.02
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 1163.50 1362.58 1.17
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 1776.08 2055.08 1.16
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 812.63 883.82 1.09
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 1295.07 1392.96 1.08
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 1955.81 2084.14 1.07
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 2761.57 2894.00 1.05
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 3811.34 3888.11 1.02
V100-PCIE-32GB granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 4723.32 4773.73 1.01
V100-PCIE-32GB granitemoe 3B Q2_K_M 16 pp2048 971.55 1137.56 1.17
V100-PCIE-32GB granitemoe 3B Q2_K_M 32 pp2048 1393.40 1553.60 1.11
V100-PCIE-32GB granitemoe 3B Q2_K_M 64 pp2048 665.27 711.20 1.07
V100-PCIE-32GB granitemoe 3B Q2_K_M 128 pp2048 1084.26 1154.53 1.06
V100-PCIE-32GB granitemoe 3B Q2_K_M 256 pp2048 1669.03 1766.61 1.06
V100-PCIE-32GB granitemoe 3B Q2_K_M 512 pp2048 2456.20 2563.63 1.04
V100-PCIE-32GB granitemoe 3B Q2_K_M 1024 pp2048 3561.53 3649.87 1.02
V100-PCIE-32GB granitemoe 3B Q2_K_M 2048 pp2048 4577.39 4625.24 1.01
V100-PCIE-32GB granitemoe 3B Q3_K_S 16 pp2048 1107.37 1275.30 1.15
V100-PCIE-32GB granitemoe 3B Q3_K_S 32 pp2048 1596.11 1858.94 1.16
V100-PCIE-32GB granitemoe 3B Q3_K_S 64 pp2048 731.50 788.55 1.08
V100-PCIE-32GB granitemoe 3B Q3_K_S 128 pp2048 1190.66 1274.40 1.07
V100-PCIE-32GB granitemoe 3B Q3_K_S 256 pp2048 1848.77 1965.21 1.06
V100-PCIE-32GB granitemoe 3B Q3_K_S 512 pp2048 2671.45 2790.58 1.04
V100-PCIE-32GB granitemoe 3B Q3_K_S 1024 pp2048 3728.49 3825.40 1.03
V100-PCIE-32GB granitemoe 3B Q3_K_S 2048 pp2048 4676.31 4732.18 1.01
V100-PCIE-32GB granitemoe 3B Q4_0 16 pp2048 1195.72 1427.15 1.19
V100-PCIE-32GB granitemoe 3B Q4_0 32 pp2048 1811.24 2128.00 1.17
V100-PCIE-32GB granitemoe 3B Q4_0 64 pp2048 808.84 911.92 1.13
V100-PCIE-32GB granitemoe 3B Q4_0 128 pp2048 1284.29 1422.25 1.11
V100-PCIE-32GB granitemoe 3B Q4_0 256 pp2048 1950.83 2122.54 1.09
V100-PCIE-32GB granitemoe 3B Q4_0 512 pp2048 2764.83 2930.15 1.06
V100-PCIE-32GB granitemoe 3B Q4_0 1024 pp2048 3780.26 3913.71 1.04
V100-PCIE-32GB granitemoe 3B Q4_0 2048 pp2048 4640.74 4768.25 1.03
V100-PCIE-32GB granitemoe 3B Q4_1 16 pp2048 1197.25 1429.11 1.19
V100-PCIE-32GB granitemoe 3B Q4_1 32 pp2048 1783.85 2099.73 1.18
V100-PCIE-32GB granitemoe 3B Q4_1 64 pp2048 803.23 904.39 1.13
V100-PCIE-32GB granitemoe 3B Q4_1 128 pp2048 1279.41 1413.73 1.10
V100-PCIE-32GB granitemoe 3B Q4_1 256 pp2048 1944.70 2108.39 1.08
V100-PCIE-32GB granitemoe 3B Q4_1 512 pp2048 2756.14 2910.21 1.06
V100-PCIE-32GB granitemoe 3B Q4_1 1024 pp2048 3793.80 3902.94 1.03
V100-PCIE-32GB granitemoe 3B Q4_1 2048 pp2048 4705.05 4739.31 1.01
V100-PCIE-32GB granitemoe 3B Q4_K_S 16 pp2048 1158.65 1366.86 1.18
V100-PCIE-32GB granitemoe 3B Q4_K_S 32 pp2048 1688.76 1882.31 1.11
V100-PCIE-32GB granitemoe 3B Q4_K_S 64 pp2048 731.50 793.03 1.08
V100-PCIE-32GB granitemoe 3B Q4_K_S 128 pp2048 1194.72 1281.89 1.07
V100-PCIE-32GB granitemoe 3B Q4_K_S 256 pp2048 1840.17 1957.15 1.06
V100-PCIE-32GB granitemoe 3B Q4_K_S 512 pp2048 2654.69 2770.72 1.04
V100-PCIE-32GB granitemoe 3B Q4_K_S 1024 pp2048 3705.05 3797.46 1.02
V100-PCIE-32GB granitemoe 3B Q4_K_S 2048 pp2048 4635.80 4717.77 1.02
V100-PCIE-32GB granitemoe 3B Q5_0 16 pp2048 1036.85 1238.49 1.19
V100-PCIE-32GB granitemoe 3B Q5_0 32 pp2048 1606.70 1796.56 1.12
V100-PCIE-32GB granitemoe 3B Q5_0 64 pp2048 738.78 829.78 1.12
V100-PCIE-32GB granitemoe 3B Q5_0 128 pp2048 1208.17 1334.85 1.10
V100-PCIE-32GB granitemoe 3B Q5_0 256 pp2048 1863.06 2026.65 1.09
V100-PCIE-32GB granitemoe 3B Q5_0 512 pp2048 2684.85 2850.02 1.06
V100-PCIE-32GB granitemoe 3B Q5_0 1024 pp2048 3748.91 3891.32 1.04
V100-PCIE-32GB granitemoe 3B Q5_0 2048 pp2048 4689.92 4749.56 1.01
V100-PCIE-32GB granitemoe 3B Q5_1 16 pp2048 1109.50 1303.60 1.17
V100-PCIE-32GB granitemoe 3B Q5_1 32 pp2048 1665.11 1932.62 1.16
V100-PCIE-32GB granitemoe 3B Q5_1 64 pp2048 756.84 851.99 1.13
V100-PCIE-32GB granitemoe 3B Q5_1 128 pp2048 1233.18 1364.40 1.11
V100-PCIE-32GB granitemoe 3B Q5_1 256 pp2048 1896.61 2063.32 1.09
V100-PCIE-32GB granitemoe 3B Q5_1 512 pp2048 2722.58 2885.44 1.06
V100-PCIE-32GB granitemoe 3B Q5_1 1024 pp2048 3768.96 3904.23 1.04
V100-PCIE-32GB granitemoe 3B Q5_1 2048 pp2048 4684.03 4780.53 1.02
V100-PCIE-32GB granitemoe 3B Q5_K_S 16 pp2048 1122.25 1285.03 1.15
V100-PCIE-32GB granitemoe 3B Q5_K_S 32 pp2048 1614.70 1766.97 1.09
V100-PCIE-32GB granitemoe 3B Q5_K_S 64 pp2048 723.56 780.92 1.08
V100-PCIE-32GB granitemoe 3B Q5_K_S 128 pp2048 1193.61 1275.15 1.07
V100-PCIE-32GB granitemoe 3B Q5_K_S 256 pp2048 1853.10 1964.67 1.06
V100-PCIE-32GB granitemoe 3B Q5_K_S 512 pp2048 2676.95 2796.74 1.04
V100-PCIE-32GB granitemoe 3B Q5_K_S 1024 pp2048 3757.93 3844.05 1.02
V100-PCIE-32GB granitemoe 3B Q5_K_S 2048 pp2048 4709.97 4756.66 1.01
V100-PCIE-32GB granitemoe 3B Q6_K 16 pp2048 1053.81 1184.16 1.12
V100-PCIE-32GB granitemoe 3B Q6_K 32 pp2048 1552.82 1685.58 1.09
V100-PCIE-32GB granitemoe 3B Q6_K 64 pp2048 727.37 783.67 1.08
V100-PCIE-32GB granitemoe 3B Q6_K 128 pp2048 1197.61 1275.17 1.06
V100-PCIE-32GB granitemoe 3B Q6_K 256 pp2048 1842.73 1945.83 1.06
V100-PCIE-32GB granitemoe 3B Q6_K 512 pp2048 2652.64 2758.20 1.04
V100-PCIE-32GB granitemoe 3B Q6_K 1024 pp2048 3699.92 3784.69 1.02
V100-PCIE-32GB granitemoe 3B Q6_K 2048 pp2048 4622.69 4683.21 1.01
V100-PCIE-32GB granitemoe 3B Q8_0 16 pp2048 1075.42 1267.19 1.18
V100-PCIE-32GB granitemoe 3B Q8_0 32 pp2048 1673.02 1951.67 1.17
V100-PCIE-32GB granitemoe 3B Q8_0 64 pp2048 801.41 905.25 1.13
V100-PCIE-32GB granitemoe 3B Q8_0 128 pp2048 1299.78 1443.86 1.11
V100-PCIE-32GB granitemoe 3B Q8_0 256 pp2048 1991.62 2177.98 1.09
V100-PCIE-32GB granitemoe 3B Q8_0 512 pp2048 2838.05 3023.08 1.07
V100-PCIE-32GB granitemoe 3B Q8_0 1024 pp2048 3905.78 4034.62 1.03
V100-PCIE-32GB granitemoe 3B Q8_0 2048 pp2048 4827.33 4896.19 1.01

@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 23, 2026

Bigger MoE
GPU Model Microbatch size Test t/s master t/s PR Speedup
MI60 / MI50 gemma4 26B.A4B Q4_0 16 pp2048 234.94 234.93 1.00
MI60 / MI50 gemma4 26B.A4B Q4_0 32 pp2048 317.28 317.16 1.00
MI60 / MI50 gemma4 26B.A4B Q4_0 64 pp2048 469.75 468.35 1.00
MI60 / MI50 gemma4 26B.A4B Q4_0 128 pp2048 740.87 746.68 1.01
MI60 / MI50 gemma4 26B.A4B Q4_0 256 pp2048 1033.45 1040.87 1.01
MI60 / MI50 gemma4 26B.A4B Q4_0 512 pp2048 1356.78 1367.66 1.01
MI60 / MI50 gemma4 26B.A4B Q4_0 1024 pp2048 1623.44 1635.80 1.01
MI60 / MI50 gemma4 26B.A4B Q4_0 2048 pp2048 1758.43 1765.78 1.00
MI60 / MI50 gpt-oss 20B MXFP4 MoE 16 pp2048 115.55 115.65 1.00
MI60 / MI50 gpt-oss 20B MXFP4 MoE 32 pp2048 335.01 333.63 1.00
MI60 / MI50 gpt-oss 20B MXFP4 MoE 64 pp2048 378.22 375.19 0.99
MI60 / MI50 gpt-oss 20B MXFP4 MoE 128 pp2048 580.31 575.30 0.99
MI60 / MI50 gpt-oss 20B MXFP4 MoE 256 pp2048 825.39 820.14 0.99
MI60 / MI50 gpt-oss 20B MXFP4 MoE 512 pp2048 1071.10 1064.40 0.99
MI60 / MI50 gpt-oss 20B MXFP4 MoE 1024 pp2048 1243.51 1229.92 0.99
MI60 / MI50 gpt-oss 20B MXFP4 MoE 2048 pp2048 1300.90 1279.23 0.98
MI60 / MI50 qwen35moe 35B.A3B Q4_0 16 pp2048 205.82 205.46 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 32 pp2048 272.65 272.69 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 64 pp2048 414.34 414.57 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 128 pp2048 632.02 632.10 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 256 pp2048 902.89 902.11 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 512 pp2048 1170.55 1169.87 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 1024 pp2048 1448.21 1449.09 1.00
MI60 / MI50 qwen35moe 35B.A3B Q4_0 2048 pp2048 1555.66 1558.35 1.00
MI100 gemma4 26B.A4B Q4_0 16 pp2048 394.76 420.43 1.07
MI100 gemma4 26B.A4B Q4_0 32 pp2048 644.64 685.87 1.06
MI100 gemma4 26B.A4B Q4_0 64 pp2048 826.15 845.30 1.02
MI100 gemma4 26B.A4B Q4_0 128 pp2048 1214.49 1245.34 1.03
MI100 gemma4 26B.A4B Q4_0 256 pp2048 1713.48 1790.44 1.04
MI100 gemma4 26B.A4B Q4_0 512 pp2048 2150.89 2251.34 1.05
MI100 gemma4 26B.A4B Q4_0 1024 pp2048 2483.36 2618.31 1.05
MI100 gemma4 26B.A4B Q4_0 2048 pp2048 2663.53 2813.87 1.06
MI100 gpt-oss 20B MXFP4 MoE 16 pp2048 540.84 559.31 1.03
MI100 gpt-oss 20B MXFP4 MoE 32 pp2048 782.66 811.71 1.04
MI100 gpt-oss 20B MXFP4 MoE 64 pp2048 1013.89 1039.49 1.03
MI100 gpt-oss 20B MXFP4 MoE 128 pp2048 1539.28 1570.97 1.02
MI100 gpt-oss 20B MXFP4 MoE 256 pp2048 1948.97 2048.26 1.05
MI100 gpt-oss 20B MXFP4 MoE 512 pp2048 2806.82 2916.75 1.04
MI100 gpt-oss 20B MXFP4 MoE 1024 pp2048 3818.24 3899.36 1.02
MI100 gpt-oss 20B MXFP4 MoE 2048 pp2048 4476.20 4489.77 1.00
MI100 qwen35moe 35B.A3B Q4_0 16 pp2048 313.19 342.16 1.09
MI100 qwen35moe 35B.A3B Q4_0 32 pp2048 506.75 548.71 1.08
MI100 qwen35moe 35B.A3B Q4_0 64 pp2048 658.57 692.94 1.05
MI100 qwen35moe 35B.A3B Q4_0 128 pp2048 1068.66 1074.49 1.01
MI100 qwen35moe 35B.A3B Q4_0 256 pp2048 1510.46 1571.89 1.04
MI100 qwen35moe 35B.A3B Q4_0 512 pp2048 1895.12 2041.83 1.08
MI100 qwen35moe 35B.A3B Q4_0 1024 pp2048 2146.93 2248.34 1.05
MI100 qwen35moe 35B.A3B Q4_0 2048 pp2048 2470.52 2636.98 1.07
RTX 3090 gemma4 26B.A4B Q4_K_M 16 pp2048 711.87 883.55 1.24
RTX 3090 gemma4 26B.A4B Q4_K_M 32 pp2048 1075.74 1236.92 1.15
RTX 3090 gemma4 26B.A4B Q4_K_M 64 pp2048 1385.92 1576.37 1.14
RTX 3090 gemma4 26B.A4B Q4_K_M 128 pp2048 1579.74 1785.15 1.13
RTX 3090 gemma4 26B.A4B Q4_K_M 256 pp2048 2418.60 2661.09 1.10
RTX 3090 gemma4 26B.A4B Q4_K_M 512 pp2048 3379.76 3733.71 1.10
RTX 3090 gemma4 26B.A4B Q4_K_M 1024 pp2048 4183.24 4632.17 1.11
RTX 3090 gemma4 26B.A4B Q4_K_M 2048 pp2048 4591.40 5122.40 1.12
RTX 3090 gpt-oss 20B MXFP4 MoE 16 pp2048 844.66 1087.99 1.29
RTX 3090 gpt-oss 20B MXFP4 MoE 32 pp2048 1246.54 1593.55 1.28
RTX 3090 gpt-oss 20B MXFP4 MoE 64 pp2048 1690.38 2147.15 1.27
RTX 3090 gpt-oss 20B MXFP4 MoE 128 pp2048 2054.84 2551.00 1.24
RTX 3090 gpt-oss 20B MXFP4 MoE 256 pp2048 3105.08 3877.13 1.25
RTX 3090 gpt-oss 20B MXFP4 MoE 512 pp2048 4166.91 5250.18 1.26
RTX 3090 gpt-oss 20B MXFP4 MoE 1024 pp2048 5025.92 6387.48 1.27
RTX 3090 gpt-oss 20B MXFP4 MoE 2048 pp2048 5347.97 6871.64 1.28
RTX 3090 qwen35moe 35B.A3B Q4_0 16 pp2048 635.31 802.34 1.26
RTX 3090 qwen35moe 35B.A3B Q4_0 32 pp2048 899.36 1164.98 1.30
RTX 3090 qwen35moe 35B.A3B Q4_0 64 pp2048 1208.30 1435.38 1.19
RTX 3090 qwen35moe 35B.A3B Q4_0 128 pp2048 1385.42 1609.80 1.16
RTX 3090 qwen35moe 35B.A3B Q4_0 256 pp2048 2065.72 2304.19 1.12
RTX 3090 qwen35moe 35B.A3B Q4_0 512 pp2048 2831.29 3092.63 1.09
RTX 3090 qwen35moe 35B.A3B Q4_0 1024 pp2048 3524.06 3725.15 1.06
RTX 3090 qwen35moe 35B.A3B Q4_0 2048 pp2048 4010.48 4196.86 1.05
RTX 4090 gemma4 26B.A4B Q4_K_M 16 pp2048 1055.45 1215.18 1.15
RTX 4090 gemma4 26B.A4B Q4_K_M 32 pp2048 1830.05 2147.74 1.17
RTX 4090 gemma4 26B.A4B Q4_K_M 64 pp2048 2746.56 3310.92 1.21
RTX 4090 gemma4 26B.A4B Q4_K_M 128 pp2048 3586.38 4280.79 1.19
RTX 4090 gemma4 26B.A4B Q4_K_M 256 pp2048 5918.30 6820.73 1.15
RTX 4090 gemma4 26B.A4B Q4_K_M 512 pp2048 8453.75 9755.65 1.15
RTX 4090 gemma4 26B.A4B Q4_K_M 1024 pp2048 9884.98 11454.52 1.16
RTX 4090 gemma4 26B.A4B Q4_K_M 2048 pp2048 9706.31 11190.67 1.15
RTX 4090 gpt-oss 20B MXFP4 MoE 16 pp2048 1372.29 1717.31 1.25
RTX 4090 gpt-oss 20B MXFP4 MoE 32 pp2048 2279.86 2923.49 1.28
RTX 4090 gpt-oss 20B MXFP4 MoE 64 pp2048 3397.16 4449.75 1.31
RTX 4090 gpt-oss 20B MXFP4 MoE 128 pp2048 4430.21 5720.85 1.29
RTX 4090 gpt-oss 20B MXFP4 MoE 256 pp2048 6887.91 9207.09 1.34
RTX 4090 gpt-oss 20B MXFP4 MoE 512 pp2048 9712.72 12957.55 1.33
RTX 4090 gpt-oss 20B MXFP4 MoE 1024 pp2048 10798.17 15001.16 1.39
RTX 4090 gpt-oss 20B MXFP4 MoE 2048 pp2048 10043.71 13571.17 1.35
RTX 4090 qwen35moe 35B.A3B Q4_0 16 pp2048 1005.87 1205.45 1.20
RTX 4090 qwen35moe 35B.A3B Q4_0 32 pp2048 1593.43 1983.13 1.24
RTX 4090 qwen35moe 35B.A3B Q4_0 64 pp2048 2447.28 2930.05 1.20
RTX 4090 qwen35moe 35B.A3B Q4_0 128 pp2048 3108.21 3694.61 1.19
RTX 4090 qwen35moe 35B.A3B Q4_0 256 pp2048 5085.92 5782.42 1.14
RTX 4090 qwen35moe 35B.A3B Q4_0 512 pp2048 7232.82 7999.83 1.11
RTX 4090 qwen35moe 35B.A3B Q4_0 1024 pp2048 8900.00 9559.98 1.07
RTX 4090 qwen35moe 35B.A3B Q4_0 2048 pp2048 9387.76 10026.03 1.07
2x RTX 4090 gemma4 26B.A4B Q4_K_M 16 pp2048 1041.94 1133.13 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 32 pp2048 1625.66 1808.01 1.11
2x RTX 4090 gemma4 26B.A4B Q4_K_M 64 pp2048 2499.41 2732.73 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 128 pp2048 3142.31 3361.39 1.07
2x RTX 4090 gemma4 26B.A4B Q4_K_M 256 pp2048 4265.30 4630.12 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 512 pp2048 5638.13 6126.24 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 1024 pp2048 6373.92 6907.26 1.08
2x RTX 4090 gemma4 26B.A4B Q4_K_M 2048 pp2048 7019.68 7628.68 1.09
2x RTX 4090 gpt-oss 20B MXFP4 MoE 16 pp2048 1511.12 1740.40 1.15
2x RTX 4090 gpt-oss 20B MXFP4 MoE 32 pp2048 2501.65 2855.63 1.14
2x RTX 4090 gpt-oss 20B MXFP4 MoE 64 pp2048 3833.33 4417.05 1.15
2x RTX 4090 gpt-oss 20B MXFP4 MoE 128 pp2048 4990.64 5813.82 1.16
2x RTX 4090 gpt-oss 20B MXFP4 MoE 256 pp2048 6694.52 7814.89 1.17
2x RTX 4090 gpt-oss 20B MXFP4 MoE 512 pp2048 8596.42 10275.85 1.20
2x RTX 4090 gpt-oss 20B MXFP4 MoE 1024 pp2048 9547.54 11639.97 1.22
2x RTX 4090 gpt-oss 20B MXFP4 MoE 2048 pp2048 9698.97 11917.34 1.23
2x RTX 4090 qwen35moe 35B.A3B Q4_0 16 pp2048 884.91 996.14 1.13
2x RTX 4090 qwen35moe 35B.A3B Q4_0 32 pp2048 1339.33 1475.12 1.10
2x RTX 4090 qwen35moe 35B.A3B Q4_0 64 pp2048 2124.02 2260.52 1.06
2x RTX 4090 qwen35moe 35B.A3B Q4_0 128 pp2048 2647.86 2853.61 1.08
2x RTX 4090 qwen35moe 35B.A3B Q4_0 256 pp2048 3846.38 3971.42 1.03
2x RTX 4090 qwen35moe 35B.A3B Q4_0 512 pp2048 5704.48 6015.18 1.05
2x RTX 4090 qwen35moe 35B.A3B Q4_0 1024 pp2048 6941.40 7387.97 1.06
2x RTX 4090 qwen35moe 35B.A3B Q4_0 2048 pp2048 8445.20 8973.19 1.06
4x RTX 4090 gemma4 26B.A4B Q4_K_M 16 pp2048 842.16 869.95 1.03
4x RTX 4090 gemma4 26B.A4B Q4_K_M 32 pp2048 1246.73 1275.54 1.02
4x RTX 4090 gemma4 26B.A4B Q4_K_M 64 pp2048 1877.39 1840.03 0.98
4x RTX 4090 gemma4 26B.A4B Q4_K_M 128 pp2048 2579.43 2612.79 1.01
4x RTX 4090 gemma4 26B.A4B Q4_K_M 256 pp2048 3932.55 3920.69 1.00
4x RTX 4090 gemma4 26B.A4B Q4_K_M 512 pp2048 5468.56 5813.24 1.06
4x RTX 4090 gemma4 26B.A4B Q4_K_M 1024 pp2048 6033.63 6442.98 1.07
4x RTX 4090 gemma4 26B.A4B Q4_K_M 2048 pp2048 6666.57 7092.32 1.06
4x RTX 4090 gpt-oss 20B MXFP4 MoE 16 pp2048 1260.63 1304.75 1.04
4x RTX 4090 gpt-oss 20B MXFP4 MoE 32 pp2048 1954.82 2056.47 1.05
4x RTX 4090 gpt-oss 20B MXFP4 MoE 64 pp2048 3010.47 3082.54 1.02
4x RTX 4090 gpt-oss 20B MXFP4 MoE 128 pp2048 4398.93 4554.38 1.04
4x RTX 4090 gpt-oss 20B MXFP4 MoE 256 pp2048 6817.69 7084.08 1.04
4x RTX 4090 gpt-oss 20B MXFP4 MoE 512 pp2048 8655.12 10203.12 1.18
4x RTX 4090 gpt-oss 20B MXFP4 MoE 1024 pp2048 9669.37 11332.91 1.17
4x RTX 4090 gpt-oss 20B MXFP4 MoE 2048 pp2048 9893.67 11806.42 1.19
4x RTX 4090 qwen35moe 35B.A3B Q4_0 16 pp2048 688.12 745.13 1.08
4x RTX 4090 qwen35moe 35B.A3B Q4_0 32 pp2048 981.36 1061.98 1.08
4x RTX 4090 qwen35moe 35B.A3B Q4_0 64 pp2048 1374.31 1421.22 1.03
4x RTX 4090 qwen35moe 35B.A3B Q4_0 128 pp2048 1959.00 1993.47 1.02
4x RTX 4090 qwen35moe 35B.A3B Q4_0 256 pp2048 3085.36 3147.51 1.02
4x RTX 4090 qwen35moe 35B.A3B Q4_0 512 pp2048 5429.84 5717.49 1.05
4x RTX 4090 qwen35moe 35B.A3B Q4_0 1024 pp2048 6645.22 7088.67 1.07
4x RTX 4090 qwen35moe 35B.A3B Q4_0 2048 pp2048 8338.66 8874.05 1.06
RTX 5090 gemma4 26B.A4B Q4_K_M 16 pp2048 1113.70 1333.64 1.20
RTX 5090 gemma4 26B.A4B Q4_K_M 32 pp2048 1687.47 2049.47 1.21
RTX 5090 gemma4 26B.A4B Q4_K_M 64 pp2048 2536.19 3135.08 1.24
RTX 5090 gemma4 26B.A4B Q4_K_M 128 pp2048 3365.05 4030.24 1.20
RTX 5090 gemma4 26B.A4B Q4_K_M 256 pp2048 5672.90 6687.19 1.18
RTX 5090 gemma4 26B.A4B Q4_K_M 512 pp2048 8271.74 9952.41 1.20
RTX 5090 gemma4 26B.A4B Q4_K_M 1024 pp2048 10302.28 12546.50 1.22
RTX 5090 gemma4 26B.A4B Q4_K_M 2048 pp2048 10470.98 12692.02 1.21
RTX 5090 gpt-oss 20B MXFP4 MoE 16 pp2048 1502.76 1594.91 1.06
RTX 5090 gpt-oss 20B MXFP4 MoE 32 pp2048 2464.17 2607.75 1.06
RTX 5090 gpt-oss 20B MXFP4 MoE 64 pp2048 4007.78 4225.14 1.05
RTX 5090 gpt-oss 20B MXFP4 MoE 128 pp2048 5558.42 5997.07 1.08
RTX 5090 gpt-oss 20B MXFP4 MoE 256 pp2048 8825.70 11745.09 1.33
RTX 5090 gpt-oss 20B MXFP4 MoE 512 pp2048 12227.36 16972.29 1.39
RTX 5090 gpt-oss 20B MXFP4 MoE 1024 pp2048 14129.36 20172.43 1.43
RTX 5090 gpt-oss 20B MXFP4 MoE 2048 pp2048 13606.75 18513.14 1.36
RTX 5090 qwen35moe 35B.A3B Q4_0 16 pp2048 882.66 1075.52 1.22
RTX 5090 qwen35moe 35B.A3B Q4_0 32 pp2048 1375.92 1659.19 1.21
RTX 5090 qwen35moe 35B.A3B Q4_0 64 pp2048 2061.42 2603.04 1.26
RTX 5090 qwen35moe 35B.A3B Q4_0 128 pp2048 2745.27 3412.21 1.24
RTX 5090 qwen35moe 35B.A3B Q4_0 256 pp2048 4679.18 5509.54 1.18
RTX 5090 qwen35moe 35B.A3B Q4_0 512 pp2048 7034.84 8063.38 1.15
RTX 5090 qwen35moe 35B.A3B Q4_0 1024 pp2048 9065.15 10166.74 1.12
RTX 5090 qwen35moe 35B.A3B Q4_0 2048 pp2048 10274.95 11452.43 1.11
RX 6800 gemma4 26B.A4B Q4_0 16 pp2048 249.42 250.41 1.00
RX 6800 gemma4 26B.A4B Q4_0 32 pp2048 353.81 354.52 1.00
RX 6800 gemma4 26B.A4B Q4_0 64 pp2048 409.50 409.87 1.00
RX 6800 gemma4 26B.A4B Q4_0 128 pp2048 611.60 612.82 1.00
RX 6800 gemma4 26B.A4B Q4_0 256 pp2048 863.98 866.57 1.00
RX 6800 gemma4 26B.A4B Q4_0 512 pp2048 1113.88 1117.79 1.00
RX 6800 gemma4 26B.A4B Q4_0 1024 pp2048 1300.84 1306.90 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 16 pp2048 201.16 201.39 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 32 pp2048 147.88 147.71 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 64 pp2048 445.06 445.50 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 128 pp2048 683.91 684.89 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 256 pp2048 969.46 971.01 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 512 pp2048 1226.42 1228.17 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 1024 pp2048 1414.16 1417.63 1.00
RX 6800 gpt-oss 20B MXFP4 MoE 2048 pp2048 1471.51 1475.38 1.00
RX 9060 XT gemma4 26B.A4B Q4_0 16 pp2048 348.98 351.17 1.01
RX 9060 XT gemma4 26B.A4B Q4_0 32 pp2048 516.09 520.49 1.01
RX 9060 XT gemma4 26B.A4B Q4_0 64 pp2048 493.52 495.15 1.00
RX 9060 XT gemma4 26B.A4B Q4_0 128 pp2048 316.27 316.51 1.00
RX 9060 XT gemma4 26B.A4B Q4_0 256 pp2048 479.54 479.94 1.00
RX 9060 XT gemma4 26B.A4B Q4_0 512 pp2048 695.22 697.17 1.00
RX 9060 XT gemma4 26B.A4B Q4_0 1024 pp2048 898.73 900.88 1.00
RX 9060 XT gpt-oss 20B MXFP4 MoE 16 pp2048 528.61 527.00 1.00
RX 9060 XT gpt-oss 20B MXFP4 MoE 32 pp2048 755.22 754.70 1.00
RX 9060 XT gpt-oss 20B MXFP4 MoE 64 pp2048 640.99 635.19 0.99
RX 9060 XT gpt-oss 20B MXFP4 MoE 128 pp2048 355.56 353.20 0.99
RX 9060 XT gpt-oss 20B MXFP4 MoE 256 pp2048 558.02 554.15 0.99
RX 9060 XT gpt-oss 20B MXFP4 MoE 512 pp2048 809.74 805.48 0.99
RX 9060 XT gpt-oss 20B MXFP4 MoE 1024 pp2048 1021.43 1017.06 1.00
RX 9060 XT gpt-oss 20B MXFP4 MoE 2048 pp2048 1156.27 1154.01 1.00

I'll have to re-take some numbers for the AMD sweep. This PR is not rebased on top of the PR that enabled HIP graphs by default and they are not being enabled for the tests.

@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 23, 2026

Tensor parallelism
GPU Model Microbatch size Test t/s master t/s 3d1bbd5 Speedup
2x RTX 4090 gemma4 26B.A4B Q4_K_M 16 pp2048 1041.94 1133.13 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 32 pp2048 1625.66 1808.01 1.11
2x RTX 4090 gemma4 26B.A4B Q4_K_M 64 pp2048 2499.41 2732.73 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 128 pp2048 3142.31 3361.39 1.07
2x RTX 4090 gemma4 26B.A4B Q4_K_M 256 pp2048 4265.30 4630.12 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 512 pp2048 5638.13 6126.24 1.09
2x RTX 4090 gemma4 26B.A4B Q4_K_M 1024 pp2048 6373.92 6907.26 1.08
2x RTX 4090 gemma4 26B.A4B Q4_K_M 2048 pp2048 7019.68 7628.68 1.09
2x RTX 4090 gpt-oss 20B MXFP4 MoE 16 pp2048 1511.12 1740.40 1.15
2x RTX 4090 gpt-oss 20B MXFP4 MoE 32 pp2048 2501.65 2855.63 1.14
2x RTX 4090 gpt-oss 20B MXFP4 MoE 64 pp2048 3833.33 4417.05 1.15
2x RTX 4090 gpt-oss 20B MXFP4 MoE 128 pp2048 4990.64 5813.82 1.16
2x RTX 4090 gpt-oss 20B MXFP4 MoE 256 pp2048 6694.52 7814.89 1.17
2x RTX 4090 gpt-oss 20B MXFP4 MoE 512 pp2048 8596.42 10275.85 1.20
2x RTX 4090 gpt-oss 20B MXFP4 MoE 1024 pp2048 9547.54 11639.97 1.22
2x RTX 4090 gpt-oss 20B MXFP4 MoE 2048 pp2048 9698.97 11917.34 1.23
2x RTX 4090 qwen35moe 35B.A3B Q4_0 16 pp2048 884.91 996.14 1.13
2x RTX 4090 qwen35moe 35B.A3B Q4_0 32 pp2048 1339.33 1475.12 1.10
2x RTX 4090 qwen35moe 35B.A3B Q4_0 64 pp2048 2124.02 2260.52 1.06
2x RTX 4090 qwen35moe 35B.A3B Q4_0 128 pp2048 2647.86 2853.61 1.08
2x RTX 4090 qwen35moe 35B.A3B Q4_0 256 pp2048 3846.38 3971.42 1.03
2x RTX 4090 qwen35moe 35B.A3B Q4_0 512 pp2048 5704.48 6015.18 1.05
2x RTX 4090 qwen35moe 35B.A3B Q4_0 1024 pp2048 6941.40 7387.97 1.06
2x RTX 4090 qwen35moe 35B.A3B Q4_0 2048 pp2048 8445.20 8973.19 1.06
4x RTX 4090 gemma4 26B.A4B Q4_K_M 16 pp2048 842.16 869.95 1.03
4x RTX 4090 gemma4 26B.A4B Q4_K_M 32 pp2048 1246.73 1275.54 1.02
4x RTX 4090 gemma4 26B.A4B Q4_K_M 64 pp2048 1877.39 1840.03 0.98
4x RTX 4090 gemma4 26B.A4B Q4_K_M 128 pp2048 2579.43 2612.79 1.01
4x RTX 4090 gemma4 26B.A4B Q4_K_M 256 pp2048 3932.55 3920.69 1.00
4x RTX 4090 gemma4 26B.A4B Q4_K_M 512 pp2048 5468.56 5813.24 1.06
4x RTX 4090 gemma4 26B.A4B Q4_K_M 1024 pp2048 6033.63 6442.98 1.07
4x RTX 4090 gemma4 26B.A4B Q4_K_M 2048 pp2048 6666.57 7092.32 1.06
4x RTX 4090 gpt-oss 20B MXFP4 MoE 16 pp2048 1260.63 1304.75 1.04
4x RTX 4090 gpt-oss 20B MXFP4 MoE 32 pp2048 1954.82 2056.47 1.05
4x RTX 4090 gpt-oss 20B MXFP4 MoE 64 pp2048 3010.47 3082.54 1.02
4x RTX 4090 gpt-oss 20B MXFP4 MoE 128 pp2048 4398.93 4554.38 1.04
4x RTX 4090 gpt-oss 20B MXFP4 MoE 256 pp2048 6817.69 7084.08 1.04
4x RTX 4090 gpt-oss 20B MXFP4 MoE 512 pp2048 8655.12 10203.12 1.18
4x RTX 4090 gpt-oss 20B MXFP4 MoE 1024 pp2048 9669.37 11332.91 1.17
4x RTX 4090 gpt-oss 20B MXFP4 MoE 2048 pp2048 9893.67 11806.42 1.19
4x RTX 4090 qwen35moe 35B.A3B Q4_0 16 pp2048 688.12 745.13 1.08
4x RTX 4090 qwen35moe 35B.A3B Q4_0 32 pp2048 981.36 1061.98 1.08
4x RTX 4090 qwen35moe 35B.A3B Q4_0 64 pp2048 1374.31 1421.22 1.03
4x RTX 4090 qwen35moe 35B.A3B Q4_0 128 pp2048 1959.00 1993.47 1.02
4x RTX 4090 qwen35moe 35B.A3B Q4_0 256 pp2048 3085.36 3147.51 1.02
4x RTX 4090 qwen35moe 35B.A3B Q4_0 512 pp2048 5429.84 5717.49 1.05
4x RTX 4090 qwen35moe 35B.A3B Q4_0 1024 pp2048 6645.22 7088.67 1.07
4x RTX 4090 qwen35moe 35B.A3B Q4_0 2048 pp2048 8338.66 8874.05 1.06

This was referenced Apr 23, 2026
Closed
Closed
@nisparks
Copy link
Copy Markdown
Contributor

nisparks commented Apr 23, 2026

Nice, excited to see this merged. There is still more juice to squeeze out of this, so working on the next performance bump now.

@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 23, 2026 •
edited
Loading

Quantization sweep AMD dense
GPU Model Microbatch size Test t/s 82209ef t/s 1b2cf95 Speedup
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 238.68 238.85 1.00
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 354.05 353.93 1.00
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 456.79 453.98 0.99
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 496.65 496.51 1.00
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 597.13 597.28 1.00
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 666.83 667.51 1.00
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 703.00 704.09 1.00
MI60 / MI50 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 699.07 704.45 1.01
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 16 pp2048 181.90 181.10 1.00
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 32 pp2048 318.98 318.59 1.00
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 64 pp2048 433.68 432.12 1.00
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 128 pp2048 475.31 475.38 1.00
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 256 pp2048 570.53 571.01 1.00
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 512 pp2048 630.87 634.40 1.01
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 656.50 666.66 1.02
MI60 / MI50 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 654.86 665.45 1.02
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 180.92 180.14 1.00
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 315.52 315.98 1.00
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 409.16 407.09 0.99
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 443.34 443.08 1.00
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 531.61 532.21 1.00
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 591.13 593.02 1.00
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 614.27 621.97 1.01
MI60 / MI50 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 612.86 621.70 1.01
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 217.02 217.39 1.00
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 308.46 307.51 1.00
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 436.42 436.17 1.00
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 475.50 475.36 1.00
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 571.34 571.43 1.00
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 635.86 637.71 1.00
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 660.15 667.66 1.01
MI60 / MI50 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 659.17 667.13 1.01
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 200.56 200.49 1.00
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 320.25 317.90 0.99
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 448.03 447.24 1.00
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 491.63 492.52 1.00
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 590.73 592.24 1.00
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 650.10 658.68 1.01
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 677.27 689.20 1.02
MI60 / MI50 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 676.59 687.47 1.02
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 203.16 203.16 1.00
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 316.24 314.71 1.00
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 453.00 452.97 1.00
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 513.26 514.91 1.00
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 615.10 617.18 1.00
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 671.73 683.26 1.02
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 702.03 714.96 1.02
MI60 / MI50 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 700.24 713.65 1.02
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 192.75 193.32 1.00
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 308.18 306.87 1.00
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 447.35 447.34 1.00
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 493.17 493.92 1.00
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 593.10 593.82 1.00
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 655.63 663.56 1.01
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 684.16 695.83 1.02
MI60 / MI50 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 683.25 695.01 1.02
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 182.50 183.46 1.01
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 300.67 299.42 1.00
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 444.97 443.88 1.00
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 492.40 492.23 1.00
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 592.36 592.46 1.00
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 655.31 661.69 1.01
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 683.90 694.85 1.02
MI60 / MI50 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 682.88 693.94 1.02
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 205.11 203.85 0.99
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 402.32 399.46 0.99
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 384.46 385.50 1.00
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 423.32 424.11 1.00
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 512.91 513.53 1.00
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 574.19 574.86 1.00
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 609.61 611.07 1.00
MI60 / MI50 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 610.52 611.41 1.00
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 223.05 223.14 1.00
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 403.28 403.11 1.00
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 488.18 492.01 1.01
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 546.87 549.27 1.00
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 670.78 672.94 1.00
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 726.10 744.39 1.03
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 763.79 786.43 1.03
MI60 / MI50 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 764.78 784.89 1.03
MI60 / MI50 llama 8B Q2_K_M 16 pp2048 171.82 171.85 1.00
MI60 / MI50 llama 8B Q2_K_M 32 pp2048 250.67 252.39 1.01
MI60 / MI50 llama 8B Q2_K_M 64 pp2048 206.90 210.61 1.02
MI60 / MI50 llama 8B Q2_K_M 128 pp2048 219.99 224.38 1.02
MI60 / MI50 llama 8B Q2_K_M 256 pp2048 247.97 252.42 1.02
MI60 / MI50 llama 8B Q2_K_M 512 pp2048 260.69 266.45 1.02
MI60 / MI50 llama 8B Q2_K_M 1024 pp2048 267.86 274.09 1.02
MI60 / MI50 llama 8B Q2_K_M 2048 pp2048 268.03 274.30 1.02
MI60 / MI50 llama 8B Q3_K_S 16 pp2048 128.02 128.13 1.00
MI60 / MI50 llama 8B Q3_K_S 32 pp2048 289.19 289.92 1.00
MI60 / MI50 llama 8B Q3_K_S 64 pp2048 386.57 386.23 1.00
MI60 / MI50 llama 8B Q3_K_S 128 pp2048 422.17 422.23 1.00
MI60 / MI50 llama 8B Q3_K_S 256 pp2048 506.93 507.00 1.00
MI60 / MI50 llama 8B Q3_K_S 512 pp2048 563.00 562.99 1.00
MI60 / MI50 llama 8B Q3_K_S 1024 pp2048 591.63 594.69 1.01
MI60 / MI50 llama 8B Q3_K_S 2048 pp2048 591.50 595.46 1.01
MI60 / MI50 llama 8B Q4_0 16 pp2048 251.41 249.73 0.99
MI60 / MI50 llama 8B Q4_0 32 pp2048 369.58 368.75 1.00
MI60 / MI50 llama 8B Q4_0 64 pp2048 633.66 631.79 1.00
MI60 / MI50 llama 8B Q4_0 128 pp2048 856.65 856.82 1.00
MI60 / MI50 llama 8B Q4_0 256 pp2048 989.62 1011.77 1.02
MI60 / MI50 llama 8B Q4_0 512 pp2048 1106.09 1142.78 1.03
MI60 / MI50 llama 8B Q4_0 1024 pp2048 1166.05 1200.94 1.03
MI60 / MI50 llama 8B Q4_0 2048 pp2048 1173.44 1205.62 1.03
MI60 / MI50 llama 8B Q4_1 16 pp2048 240.60 239.37 0.99
MI60 / MI50 llama 8B Q4_1 32 pp2048 365.54 365.34 1.00
MI60 / MI50 llama 8B Q4_1 64 pp2048 615.79 614.37 1.00
MI60 / MI50 llama 8B Q4_1 128 pp2048 842.44 841.54 1.00
MI60 / MI50 llama 8B Q4_1 256 pp2048 963.85 981.15 1.02
MI60 / MI50 llama 8B Q4_1 512 pp2048 1084.33 1100.66 1.02
MI60 / MI50 llama 8B Q4_1 1024 pp2048 1145.02 1162.54 1.02
MI60 / MI50 llama 8B Q4_1 2048 pp2048 1155.54 1171.25 1.01
MI60 / MI50 llama 8B Q4_K_S 16 pp2048 240.45 237.72 0.99
MI60 / MI50 llama 8B Q4_K_S 32 pp2048 354.96 354.97 1.00
MI60 / MI50 llama 8B Q4_K_S 64 pp2048 539.15 536.89 1.00
MI60 / MI50 llama 8B Q4_K_S 128 pp2048 686.19 679.81 0.99
MI60 / MI50 llama 8B Q4_K_S 256 pp2048 801.91 808.97 1.01
MI60 / MI50 llama 8B Q4_K_S 512 pp2048 891.72 903.79 1.01
MI60 / MI50 llama 8B Q4_K_S 1024 pp2048 944.52 958.65 1.01
MI60 / MI50 llama 8B Q4_K_S 2048 pp2048 949.31 965.07 1.02
MI60 / MI50 llama 8B Q5_0 16 pp2048 181.01 180.61 1.00
MI60 / MI50 llama 8B Q5_0 32 pp2048 343.79 344.17 1.00
MI60 / MI50 llama 8B Q5_0 64 pp2048 364.19 362.93 1.00
MI60 / MI50 llama 8B Q5_0 128 pp2048 398.57 397.30 1.00
MI60 / MI50 llama 8B Q5_0 256 pp2048 483.67 482.20 1.00
MI60 / MI50 llama 8B Q5_0 512 pp2048 540.95 539.43 1.00
MI60 / MI50 llama 8B Q5_0 1024 pp2048 576.75 574.88 1.00
MI60 / MI50 llama 8B Q5_0 2048 pp2048 578.79 575.96 1.00
MI60 / MI50 llama 8B Q5_1 16 pp2048 167.38 167.49 1.00
MI60 / MI50 llama 8B Q5_1 32 pp2048 368.57 368.19 1.00
MI60 / MI50 llama 8B Q5_1 64 pp2048 363.69 365.61 1.01
MI60 / MI50 llama 8B Q5_1 128 pp2048 397.90 400.12 1.01
MI60 / MI50 llama 8B Q5_1 256 pp2048 485.46 488.13 1.01
MI60 / MI50 llama 8B Q5_1 512 pp2048 542.29 545.71 1.01
MI60 / MI50 llama 8B Q5_1 1024 pp2048 576.54 580.38 1.01
MI60 / MI50 llama 8B Q5_1 2048 pp2048 577.13 580.61 1.01
MI60 / MI50 llama 8B Q5_K_S 16 pp2048 133.95 136.75 1.02
MI60 / MI50 llama 8B Q5_K_S 32 pp2048 390.39 391.29 1.00
MI60 / MI50 llama 8B Q5_K_S 64 pp2048 354.57 354.33 1.00
MI60 / MI50 llama 8B Q5_K_S 128 pp2048 388.94 389.80 1.00
MI60 / MI50 llama 8B Q5_K_S 256 pp2048 474.20 475.13 1.00
MI60 / MI50 llama 8B Q5_K_S 512 pp2048 530.22 531.18 1.00
MI60 / MI50 llama 8B Q5_K_S 1024 pp2048 564.80 564.52 1.00
MI60 / MI50 llama 8B Q5_K_S 2048 pp2048 567.26 566.04 1.00
MI60 / MI50 llama 8B Q6_K 16 pp2048 117.32 119.48 1.02
MI60 / MI50 llama 8B Q6_K 32 pp2048 264.24 268.40 1.02
MI60 / MI50 llama 8B Q6_K 64 pp2048 412.70 389.92 0.94
MI60 / MI50 llama 8B Q6_K 128 pp2048 452.85 422.28 0.93
MI60 / MI50 llama 8B Q6_K 256 pp2048 548.32 509.55 0.93
MI60 / MI50 llama 8B Q6_K 512 pp2048 590.73 558.92 0.95
MI60 / MI50 llama 8B Q6_K 1024 pp2048 620.65 586.47 0.94
MI60 / MI50 llama 8B Q6_K 2048 pp2048 620.46 585.25 0.94
MI60 / MI50 llama 8B Q8_0 16 pp2048 116.22 115.90 1.00
MI60 / MI50 llama 8B Q8_0 32 pp2048 388.84 389.27 1.00
MI60 / MI50 llama 8B Q8_0 64 pp2048 338.64 336.41 0.99
MI60 / MI50 llama 8B Q8_0 128 pp2048 367.64 365.25 0.99
MI60 / MI50 llama 8B Q8_0 256 pp2048 451.15 448.46 0.99
MI60 / MI50 llama 8B Q8_0 512 pp2048 503.95 501.35 0.99
MI60 / MI50 llama 8B Q8_0 1024 pp2048 536.53 533.77 0.99
MI60 / MI50 llama 8B Q8_0 2048 pp2048 537.07 534.46 1.00
MI100 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 704.71 772.39 1.10
MI100 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 1180.90 1270.08 1.08
MI100 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 1660.16 1784.76 1.08
MI100 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 2088.36 2164.29 1.04
MI100 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 2271.68 2297.21 1.01
MI100 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 2940.63 3015.74 1.03
MI100 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 3496.85 3579.53 1.02
MI100 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 3889.78 3956.61 1.02
MI100 llama 8B IQ2_S - 2.5 bpw 16 pp2048 522.40 574.12 1.10
MI100 llama 8B IQ2_S - 2.5 bpw 32 pp2048 909.13 922.24 1.01
MI100 llama 8B IQ2_S - 2.5 bpw 64 pp2048 1208.39 1318.92 1.09
MI100 llama 8B IQ2_S - 2.5 bpw 128 pp2048 1489.55 1563.94 1.05
MI100 llama 8B IQ2_S - 2.5 bpw 256 pp2048 2147.09 2177.08 1.01
MI100 llama 8B IQ2_S - 2.5 bpw 512 pp2048 2887.09 2908.14 1.01
MI100 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 3412.38 3464.45 1.02
MI100 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 3780.96 3802.74 1.01
MI100 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 539.26 587.83 1.09
MI100 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 929.89 929.29 1.00
MI100 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 1196.87 1305.46 1.09
MI100 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 1470.42 1555.19 1.06
MI100 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 2173.07 2159.00 0.99
MI100 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 2867.78 2924.75 1.02
MI100 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 3476.84 3504.34 1.01
MI100 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 3877.65 3899.86 1.01
MI100 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 608.62 652.01 1.07
MI100 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 1013.88 1106.84 1.09
MI100 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 1490.41 1600.53 1.07
MI100 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 1868.54 1914.31 1.02
MI100 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 2216.21 2253.27 1.02
MI100 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 2912.40 2953.33 1.01
MI100 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 3499.44 3524.91 1.01
MI100 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 3842.39 3922.40 1.02
MI100 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 552.47 594.63 1.08
MI100 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 933.56 1023.38 1.10
MI100 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 1374.90 1467.73 1.07
MI100 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 1658.91 1721.53 1.04
MI100 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 2127.23 2171.08 1.02
MI100 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 2861.76 2905.00 1.02
MI100 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 3402.35 3470.82 1.02
MI100 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 3795.91 3791.18 1.00
MI100 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 563.05 604.51 1.07
MI100 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 939.50 1039.97 1.11
MI100 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 1370.81 1468.87 1.07
MI100 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 1679.74 1746.73 1.04
MI100 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 2214.18 2258.15 1.02
MI100 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 2784.17 2869.46 1.03
MI100 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 3411.17 3461.63 1.01
MI100 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 3770.92 3823.89 1.01
MI100 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 537.32 589.99 1.10
MI100 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 903.64 999.66 1.11
MI100 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 1337.55 1433.18 1.07
MI100 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 1646.20 1707.50 1.04
MI100 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 2122.36 2201.89 1.04
MI100 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 2868.57 2915.34 1.02
MI100 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 3447.39 3486.34 1.01
MI100 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 3746.63 3817.35 1.02
MI100 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 532.21 575.12 1.08
MI100 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 898.87 965.51 1.07
MI100 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 1306.70 1367.81 1.05
MI100 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 1624.65 1654.76 1.02
MI100 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 2155.38 2207.87 1.02
MI100 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 2872.43 2910.08 1.01
MI100 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 3407.97 3484.26 1.02
MI100 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 3798.36 3806.41 1.00
MI100 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 720.85 683.09 0.95
MI100 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 1220.15 1292.78 1.06
MI100 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 1673.74 1755.43 1.05
MI100 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 2114.22 2127.55 1.01
MI100 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 2144.77 2207.17 1.03
MI100 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 2830.40 2921.00 1.03
MI100 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 3395.20 3530.24 1.04
MI100 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 3885.59 3900.45 1.00
MI100 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 754.64 823.04 1.09
MI100 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 1280.75 1357.47 1.06
MI100 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 1776.53 1819.11 1.02
MI100 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 2277.42 2188.13 0.96
MI100 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 2141.07 2223.82 1.04
MI100 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 2856.41 2947.01 1.03
MI100 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 3450.17 3544.64 1.03
MI100 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 3856.63 3924.40 1.02
MI100 llama 8B Q2_K_M 16 pp2048 635.90 686.76 1.08
MI100 llama 8B Q2_K_M 32 pp2048 936.15 1100.23 1.18
MI100 llama 8B Q2_K_M 64 pp2048 1109.19 1211.03 1.09
MI100 llama 8B Q2_K_M 128 pp2048 1377.82 1422.57 1.03
MI100 llama 8B Q2_K_M 256 pp2048 2259.22 2227.04 0.99
MI100 llama 8B Q2_K_M 512 pp2048 2976.21 2957.42 0.99
MI100 llama 8B Q2_K_M 1024 pp2048 3534.66 3557.97 1.01
MI100 llama 8B Q2_K_M 2048 pp2048 3965.87 3959.71 1.00
MI100 llama 8B Q3_K_S 16 pp2048 682.14 732.09 1.07
MI100 llama 8B Q3_K_S 32 pp2048 1100.06 1110.81 1.01
MI100 llama 8B Q3_K_S 64 pp2048 1162.36 1276.31 1.10
MI100 llama 8B Q3_K_S 128 pp2048 1327.40 1495.32 1.13
MI100 llama 8B Q3_K_S 256 pp2048 2109.85 2125.74 1.01
MI100 llama 8B Q3_K_S 512 pp2048 2836.83 2891.20 1.02
MI100 llama 8B Q3_K_S 1024 pp2048 3469.52 3503.03 1.01
MI100 llama 8B Q3_K_S 2048 pp2048 3791.74 3901.83 1.03
MI100 llama 8B Q4_0 16 pp2048 739.87 800.47 1.08
MI100 llama 8B Q4_0 32 pp2048 1306.43 1386.29 1.06
MI100 llama 8B Q4_0 64 pp2048 1810.05 1931.72 1.07
MI100 llama 8B Q4_0 128 pp2048 2282.81 2349.66 1.03
MI100 llama 8B Q4_0 256 pp2048 2515.34 2514.64 1.00
MI100 llama 8B Q4_0 512 pp2048 2669.94 2627.01 0.98
MI100 llama 8B Q4_0 1024 pp2048 2723.84 2703.64 0.99
MI100 llama 8B Q4_0 2048 pp2048 2711.82 2654.27 0.98
MI100 llama 8B Q4_1 16 pp2048 749.35 824.55 1.10
MI100 llama 8B Q4_1 32 pp2048 1301.38 1367.69 1.05
MI100 llama 8B Q4_1 64 pp2048 1812.27 1927.11 1.06
MI100 llama 8B Q4_1 128 pp2048 2285.39 2352.44 1.03
MI100 llama 8B Q4_1 256 pp2048 2515.44 2539.75 1.01
MI100 llama 8B Q4_1 512 pp2048 2657.27 2646.68 1.00
MI100 llama 8B Q4_1 1024 pp2048 2697.60 2742.78 1.02
MI100 llama 8B Q4_1 2048 pp2048 2697.68 2709.88 1.00
MI100 llama 8B Q4_K_S 16 pp2048 759.72 831.10 1.09
MI100 llama 8B Q4_K_S 32 pp2048 1162.67 1418.79 1.22
MI100 llama 8B Q4_K_S 64 pp2048 1621.11 1788.10 1.10
MI100 llama 8B Q4_K_S 128 pp2048 2022.57 2175.40 1.08
MI100 llama 8B Q4_K_S 256 pp2048 2217.64 2355.95 1.06
MI100 llama 8B Q4_K_S 512 pp2048 2896.11 2984.55 1.03
MI100 llama 8B Q4_K_S 1024 pp2048 3547.26 3598.55 1.01
MI100 llama 8B Q4_K_S 2048 pp2048 3916.76 3979.71 1.02
MI100 llama 8B Q5_0 16 pp2048 599.69 630.95 1.05
MI100 llama 8B Q5_0 32 pp2048 1052.47 1115.50 1.06
MI100 llama 8B Q5_0 64 pp2048 1553.43 1669.62 1.07
MI100 llama 8B Q5_0 128 pp2048 1889.60 1984.19 1.05
MI100 llama 8B Q5_0 256 pp2048 2039.47 2116.73 1.04
MI100 llama 8B Q5_0 512 pp2048 2161.80 2196.77 1.02
MI100 llama 8B Q5_0 1024 pp2048 2204.01 2275.94 1.03
MI100 llama 8B Q5_0 2048 pp2048 2193.41 2239.71 1.02
MI100 llama 8B Q5_1 16 pp2048 649.81 711.48 1.09
MI100 llama 8B Q5_1 32 pp2048 1186.73 1153.24 0.97
MI100 llama 8B Q5_1 64 pp2048 1636.78 1750.57 1.07
MI100 llama 8B Q5_1 128 pp2048 2061.51 2141.52 1.04
MI100 llama 8B Q5_1 256 pp2048 2268.56 2312.16 1.02
MI100 llama 8B Q5_1 512 pp2048 2383.41 2389.39 1.00
MI100 llama 8B Q5_1 1024 pp2048 2410.54 2466.51 1.02
MI100 llama 8B Q5_1 2048 pp2048 2397.70 2406.44 1.00
MI100 llama 8B Q5_K_S 16 pp2048 630.34 665.26 1.06
MI100 llama 8B Q5_K_S 32 pp2048 1254.23 1148.39 0.92
MI100 llama 8B Q5_K_S 64 pp2048 1575.21 1656.80 1.05
MI100 llama 8B Q5_K_S 128 pp2048 1948.91 2003.82 1.03
MI100 llama 8B Q5_K_S 256 pp2048 2128.68 2168.23 1.02
MI100 llama 8B Q5_K_S 512 pp2048 2836.26 2928.82 1.03
MI100 llama 8B Q5_K_S 1024 pp2048 3537.03 3556.15 1.01
MI100 llama 8B Q5_K_S 2048 pp2048 3838.65 3956.20 1.03
MI100 llama 8B Q6_K 16 pp2048 583.49 609.32 1.04
MI100 llama 8B Q6_K 32 pp2048 909.20 964.14 1.06
MI100 llama 8B Q6_K 64 pp2048 853.10 1046.89 1.23
MI100 llama 8B Q6_K 128 pp2048 970.57 1205.24 1.24
MI100 llama 8B Q6_K 256 pp2048 2201.15 2068.63 0.94
MI100 llama 8B Q6_K 512 pp2048 2910.77 2924.20 1.00
MI100 llama 8B Q6_K 1024 pp2048 3536.69 3504.74 0.99
MI100 llama 8B Q6_K 2048 pp2048 3808.64 3905.01 1.03
MI100 llama 8B Q8_0 16 pp2048 649.16 710.52 1.09
MI100 llama 8B Q8_0 32 pp2048 1155.92 1171.21 1.01
MI100 llama 8B Q8_0 64 pp2048 1649.05 1730.59 1.05
MI100 llama 8B Q8_0 128 pp2048 2040.99 2126.92 1.04
MI100 llama 8B Q8_0 256 pp2048 2498.22 2447.96 0.98
MI100 llama 8B Q8_0 512 pp2048 3346.95 3314.42 0.99
MI100 llama 8B Q8_0 1024 pp2048 3879.62 3878.61 1.00
MI100 llama 8B Q8_0 2048 pp2048 4183.15 4204.60 1.01
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 16 pp2048 427.79 428.40 1.00
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 32 pp2048 454.13 455.79 1.00
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 64 pp2048 315.73 303.39 0.96
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 128 pp2048 838.09 837.65 1.00
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 256 pp2048 877.55 879.02 1.00
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 512 pp2048 902.47 903.79 1.00
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 953.75 945.97 0.99
Radeon 8060S Graphics llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 955.69 958.18 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 16 pp2048 298.45 298.26 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 32 pp2048 410.37 409.99 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 64 pp2048 302.07 301.47 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 128 pp2048 718.08 719.61 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 256 pp2048 745.67 745.48 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 512 pp2048 754.36 750.77 1.00
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 1024 pp2048 782.75 787.50 1.01
Radeon 8060S Graphics llama 8B IQ2_S - 2.5 bpw 2048 pp2048 791.04 792.84 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 294.42 294.59 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 403.62 403.70 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 290.38 288.30 0.99
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 807.34 807.28 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 838.70 839.34 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 855.21 855.76 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 898.38 899.66 1.00
Radeon 8060S Graphics llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 908.57 904.94 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 350.55 350.84 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 382.60 387.65 1.01
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 645.84 645.69 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 406.92 408.04 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 427.57 428.49 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 437.40 438.94 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 446.29 447.04 1.00
Radeon 8060S Graphics llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 443.92 443.66 1.00
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 16 pp2048 316.66 317.17 1.00
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 32 pp2048 451.90 452.08 1.00
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 64 pp2048 593.81 593.87 1.00
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 128 pp2048 384.75 390.12 1.01
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 256 pp2048 400.69 403.07 1.01
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 512 pp2048 409.69 407.08 0.99
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 420.50 418.41 1.00
Radeon 8060S Graphics llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 415.07 414.01 1.00
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 326.20 326.37 1.00
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 456.36 456.48 1.00
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 598.93 599.81 1.00
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 415.83 418.69 1.01
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 430.13 429.73 1.00
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 437.45 434.47 0.99
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 446.78 448.22 1.00
Radeon 8060S Graphics llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 446.07 445.83 1.00
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 16 pp2048 327.33 326.56 1.00
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 32 pp2048 477.41 477.25 1.00
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 64 pp2048 607.92 610.21 1.00
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 128 pp2048 392.57 395.47 1.01
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 256 pp2048 405.96 406.57 1.00
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 512 pp2048 408.58 414.04 1.01
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 418.77 423.02 1.01
Radeon 8060S Graphics llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 417.87 418.92 1.00
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 330.42 330.66 1.00
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 476.39 473.77 0.99
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 531.11 532.20 1.00
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 424.36 421.54 0.99
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 435.44 431.91 0.99
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 435.06 438.93 1.01
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 447.35 446.90 1.00
Radeon 8060S Graphics llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 441.40 440.98 1.00
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 16 pp2048 435.06 434.16 1.00
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 32 pp2048 489.51 488.17 1.00
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 64 pp2048 558.33 556.67 1.00
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 128 pp2048 370.29 367.90 0.99
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 256 pp2048 385.23 382.74 0.99
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 512 pp2048 391.17 389.02 0.99
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 399.55 398.67 1.00
Radeon 8060S Graphics llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 396.56 393.30 0.99
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 16 pp2048 459.67 459.50 1.00
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 32 pp2048 501.86 501.91 1.00
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 64 pp2048 554.26 553.05 1.00
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 128 pp2048 375.48 369.19 0.98
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 256 pp2048 389.41 384.16 0.99
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 512 pp2048 396.52 391.94 0.99
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 403.90 403.98 1.00
Radeon 8060S Graphics llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 400.25 399.02 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 16 pp2048 241.81 238.26 0.99
Radeon 8060S Graphics llama 8B Q2_K_M 32 pp2048 371.56 372.76 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 64 pp2048 375.97 375.87 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 128 pp2048 635.22 637.68 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 256 pp2048 688.48 690.34 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 512 pp2048 864.71 864.64 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 1024 pp2048 995.59 998.89 1.00
Radeon 8060S Graphics llama 8B Q2_K_M 2048 pp2048 1053.72 1057.80 1.00
Radeon 8060S Graphics llama 8B Q3_K_S 16 pp2048 297.33 291.18 0.98
Radeon 8060S Graphics llama 8B Q3_K_S 32 pp2048 393.30 393.67 1.00
Radeon 8060S Graphics llama 8B Q3_K_S 64 pp2048 277.61 278.05 1.00
Radeon 8060S Graphics llama 8B Q3_K_S 128 pp2048 914.69 921.58 1.01
Radeon 8060S Graphics llama 8B Q3_K_S 256 pp2048 958.77 965.30 1.01
Radeon 8060S Graphics llama 8B Q3_K_S 512 pp2048 988.76 988.80 1.00
Radeon 8060S Graphics llama 8B Q3_K_S 1024 pp2048 1022.65 1029.73 1.01
Radeon 8060S Graphics llama 8B Q3_K_S 2048 pp2048 1021.84 1032.21 1.01
Radeon 8060S Graphics llama 8B Q4_0 16 pp2048 402.12 404.78 1.01
Radeon 8060S Graphics llama 8B Q4_0 32 pp2048 388.74 388.38 1.00
Radeon 8060S Graphics llama 8B Q4_0 64 pp2048 550.59 541.34 0.98
Radeon 8060S Graphics llama 8B Q4_0 128 pp2048 364.56 356.66 0.98
Radeon 8060S Graphics llama 8B Q4_0 256 pp2048 385.96 379.78 0.98
Radeon 8060S Graphics llama 8B Q4_0 512 pp2048 402.02 393.89 0.98
Radeon 8060S Graphics llama 8B Q4_0 1024 pp2048 408.04 402.16 0.99
Radeon 8060S Graphics llama 8B Q4_0 2048 pp2048 405.10 395.59 0.98
Radeon 8060S Graphics llama 8B Q4_1 16 pp2048 403.98 405.74 1.00
Radeon 8060S Graphics llama 8B Q4_1 32 pp2048 379.28 380.35 1.00
Radeon 8060S Graphics llama 8B Q4_1 64 pp2048 243.92 245.73 1.01
Radeon 8060S Graphics llama 8B Q4_1 128 pp2048 904.40 916.05 1.01
Radeon 8060S Graphics llama 8B Q4_1 256 pp2048 955.14 970.36 1.02
Radeon 8060S Graphics llama 8B Q4_1 512 pp2048 977.70 993.28 1.02
Radeon 8060S Graphics llama 8B Q4_1 1024 pp2048 1034.16 1044.88 1.01
Radeon 8060S Graphics llama 8B Q4_1 2048 pp2048 1040.56 1058.44 1.02
Radeon 8060S Graphics llama 8B Q4_K_S 16 pp2048 407.03 408.20 1.00
Radeon 8060S Graphics llama 8B Q4_K_S 32 pp2048 496.45 496.48 1.00
Radeon 8060S Graphics llama 8B Q4_K_S 64 pp2048 618.71 614.80 0.99
Radeon 8060S Graphics llama 8B Q4_K_S 128 pp2048 948.57 945.58 1.00
Radeon 8060S Graphics llama 8B Q4_K_S 256 pp2048 998.68 995.37 1.00
Radeon 8060S Graphics llama 8B Q4_K_S 512 pp2048 1018.01 1010.93 0.99
Radeon 8060S Graphics llama 8B Q4_K_S 1024 pp2048 1069.02 1071.43 1.00
Radeon 8060S Graphics llama 8B Q4_K_S 2048 pp2048 1081.52 1079.77 1.00
Radeon 8060S Graphics llama 8B Q5_0 16 pp2048 350.43 353.52 1.01
Radeon 8060S Graphics llama 8B Q5_0 32 pp2048 310.57 310.68 1.00
Radeon 8060S Graphics llama 8B Q5_0 64 pp2048 498.18 503.50 1.01
Radeon 8060S Graphics llama 8B Q5_0 128 pp2048 322.31 327.30 1.02
Radeon 8060S Graphics llama 8B Q5_0 256 pp2048 340.14 344.55 1.01
Radeon 8060S Graphics llama 8B Q5_0 512 pp2048 351.42 356.90 1.02
Radeon 8060S Graphics llama 8B Q5_0 1024 pp2048 363.09 364.56 1.00
Radeon 8060S Graphics llama 8B Q5_0 2048 pp2048 357.60 363.39 1.02
Radeon 8060S Graphics llama 8B Q5_1 16 pp2048 289.28 290.09 1.00
Radeon 8060S Graphics llama 8B Q5_1 32 pp2048 254.82 253.56 1.00
Radeon 8060S Graphics llama 8B Q5_1 64 pp2048 210.08 206.25 0.98
Radeon 8060S Graphics llama 8B Q5_1 128 pp2048 869.65 860.15 0.99
Radeon 8060S Graphics llama 8B Q5_1 256 pp2048 920.53 918.50 1.00
Radeon 8060S Graphics llama 8B Q5_1 512 pp2048 958.29 952.41 0.99
Radeon 8060S Graphics llama 8B Q5_1 1024 pp2048 1008.60 1000.27 0.99
Radeon 8060S Graphics llama 8B Q5_1 2048 pp2048 1020.44 1017.87 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 16 pp2048 407.79 407.84 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 32 pp2048 357.81 358.59 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 64 pp2048 233.91 231.82 0.99
Radeon 8060S Graphics llama 8B Q5_K_S 128 pp2048 915.77 915.18 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 256 pp2048 965.12 965.28 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 512 pp2048 982.25 981.74 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 1024 pp2048 1042.67 1042.50 1.00
Radeon 8060S Graphics llama 8B Q5_K_S 2048 pp2048 1048.39 1046.70 1.00
Radeon 8060S Graphics llama 8B Q6_K 16 pp2048 316.84 316.92 1.00
Radeon 8060S Graphics llama 8B Q6_K 32 pp2048 129.28 128.94 1.00
Radeon 8060S Graphics llama 8B Q6_K 64 pp2048 600.82 598.04 1.00
Radeon 8060S Graphics llama 8B Q6_K 128 pp2048 693.34 693.94 1.00
Radeon 8060S Graphics llama 8B Q6_K 256 pp2048 722.49 722.69 1.00
Radeon 8060S Graphics llama 8B Q6_K 512 pp2048 770.08 771.99 1.00
Radeon 8060S Graphics llama 8B Q6_K 1024 pp2048 903.26 903.25 1.00
Radeon 8060S Graphics llama 8B Q6_K 2048 pp2048 987.37 991.50 1.00
Radeon 8060S Graphics llama 8B Q8_0 16 pp2048 305.11 307.36 1.01
Radeon 8060S Graphics llama 8B Q8_0 32 pp2048 408.96 408.26 1.00
Radeon 8060S Graphics llama 8B Q8_0 64 pp2048 531.62 531.23 1.00
Radeon 8060S Graphics llama 8B Q8_0 128 pp2048 343.67 342.92 1.00
Radeon 8060S Graphics llama 8B Q8_0 256 pp2048 352.69 352.29 1.00
Radeon 8060S Graphics llama 8B Q8_0 512 pp2048 367.68 365.82 0.99
Radeon 8060S Graphics llama 8B Q8_0 1024 pp2048 377.32 377.50 1.00
Radeon 8060S Graphics llama 8B Q8_0 2048 pp2048 376.85 376.47 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 16 pp2048 246.87 244.74 0.99
RX 6800 llama 8B IQ1_S - 1.5625 bpw 32 pp2048 385.08 384.23 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 64 pp2048 514.35 514.73 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 128 pp2048 653.47 654.96 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 256 pp2048 769.67 771.77 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 512 pp2048 825.80 827.80 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 852.95 854.56 1.00
RX 6800 llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 872.66 874.55 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 16 pp2048 169.10 169.93 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 32 pp2048 317.31 319.04 1.01
RX 6800 llama 8B IQ2_S - 2.5 bpw 64 pp2048 454.19 454.64 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 128 pp2048 577.63 577.70 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 256 pp2048 671.84 672.33 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 512 pp2048 717.03 717.62 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 1024 pp2048 734.61 734.97 1.00
RX 6800 llama 8B IQ2_S - 2.5 bpw 2048 pp2048 746.14 746.35 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 166.74 167.55 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 342.02 342.51 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 441.97 442.42 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 558.30 559.36 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 653.23 654.10 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 700.28 701.28 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 719.93 720.24 1.00
RX 6800 llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 734.39 735.10 1.00
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 222.57 180.62 0.81
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 220.12 215.55 0.98
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 477.50 479.46 1.00
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 605.89 609.35 1.01
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 712.70 716.10 1.00
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 764.50 768.35 1.01
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 787.98 791.63 1.00
RX 6800 llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 803.53 807.49 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 16 pp2048 213.87 213.96 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 32 pp2048 209.29 212.78 1.02
RX 6800 llama 8B IQ3_S - 3.4375 bpw 64 pp2048 485.61 486.25 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 128 pp2048 616.86 617.55 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 256 pp2048 723.17 723.61 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 512 pp2048 773.21 773.74 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 793.75 794.23 1.00
RX 6800 llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 804.05 804.63 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 218.65 218.86 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 221.26 224.41 1.01
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 468.59 469.06 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 588.86 589.39 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 696.35 697.05 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 749.08 749.74 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 770.06 770.84 1.00
RX 6800 llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 781.13 781.76 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 16 pp2048 215.93 215.58 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 32 pp2048 202.35 205.81 1.02
RX 6800 llama 8B IQ3_XS - 3.3 bpw 64 pp2048 483.80 484.10 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 128 pp2048 616.33 616.34 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 256 pp2048 724.14 724.18 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 512 pp2048 774.72 774.85 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 795.42 795.65 1.00
RX 6800 llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 806.28 806.96 1.00
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 208.96 207.33 0.99
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 215.32 218.33 1.01
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 479.27 480.12 1.00
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 612.63 612.94 1.00
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 718.51 719.15 1.00
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 769.70 770.38 1.00
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 790.29 791.35 1.00
RX 6800 llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 802.93 803.34 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 16 pp2048 233.87 234.32 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 32 pp2048 179.22 180.28 1.01
RX 6800 llama 8B IQ4_NL - 4.5 bpw 64 pp2048 547.82 547.24 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 128 pp2048 699.95 700.74 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 256 pp2048 822.25 823.15 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 512 pp2048 879.73 880.64 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 906.41 907.40 1.00
RX 6800 llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 921.80 922.70 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 16 pp2048 221.96 222.24 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 32 pp2048 182.34 180.96 0.99
RX 6800 llama 8B IQ4_XS - 4.25 bpw 64 pp2048 546.26 547.61 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 128 pp2048 701.38 702.27 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 256 pp2048 821.82 823.65 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 512 pp2048 877.94 879.63 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 903.62 904.64 1.00
RX 6800 llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 920.81 921.72 1.00
RX 6800 llama 8B Q2_K_M 16 pp2048 217.66 217.78 1.00
RX 6800 llama 8B Q2_K_M 32 pp2048 301.75 302.25 1.00
RX 6800 llama 8B Q2_K_M 64 pp2048 334.13 335.76 1.00
RX 6800 llama 8B Q2_K_M 128 pp2048 416.30 419.11 1.01
RX 6800 llama 8B Q2_K_M 256 pp2048 494.38 500.58 1.01
RX 6800 llama 8B Q2_K_M 512 pp2048 531.78 537.54 1.01
RX 6800 llama 8B Q2_K_M 1024 pp2048 550.41 555.75 1.01
RX 6800 llama 8B Q2_K_M 2048 pp2048 563.09 567.42 1.01
RX 6800 llama 8B Q3_K_S 16 pp2048 181.13 181.32 1.00
RX 6800 llama 8B Q3_K_S 32 pp2048 334.70 335.43 1.00
RX 6800 llama 8B Q3_K_S 64 pp2048 399.55 399.54 1.00
RX 6800 llama 8B Q3_K_S 128 pp2048 501.67 501.85 1.00
RX 6800 llama 8B Q3_K_S 256 pp2048 582.33 582.57 1.00
RX 6800 llama 8B Q3_K_S 512 pp2048 625.99 626.17 1.00
RX 6800 llama 8B Q3_K_S 1024 pp2048 641.53 641.74 1.00
RX 6800 llama 8B Q3_K_S 2048 pp2048 653.52 654.00 1.00
RX 6800 llama 8B Q4_0 16 pp2048 296.92 296.56 1.00
RX 6800 llama 8B Q4_0 32 pp2048 458.34 457.11 1.00
RX 6800 llama 8B Q4_0 64 pp2048 585.20 585.12 1.00
RX 6800 llama 8B Q4_0 128 pp2048 740.27 740.17 1.00
RX 6800 llama 8B Q4_0 256 pp2048 859.36 859.28 1.00
RX 6800 llama 8B Q4_0 512 pp2048 920.62 920.67 1.00
RX 6800 llama 8B Q4_0 1024 pp2048 946.76 947.35 1.00
RX 6800 llama 8B Q4_0 2048 pp2048 963.43 963.65 1.00
RX 6800 llama 8B Q4_1 16 pp2048 303.59 303.91 1.00
RX 6800 llama 8B Q4_1 32 pp2048 429.21 428.98 1.00
RX 6800 llama 8B Q4_1 64 pp2048 552.97 552.03 1.00
RX 6800 llama 8B Q4_1 128 pp2048 701.89 702.02 1.00
RX 6800 llama 8B Q4_1 256 pp2048 818.17 818.15 1.00
RX 6800 llama 8B Q4_1 512 pp2048 879.73 880.36 1.00
RX 6800 llama 8B Q4_1 1024 pp2048 907.25 907.48 1.00
RX 6800 llama 8B Q4_1 2048 pp2048 923.08 923.58 1.00
RX 6800 llama 8B Q4_K_S 16 pp2048 255.02 254.80 1.00
RX 6800 llama 8B Q4_K_S 32 pp2048 359.63 360.32 1.00
RX 6800 llama 8B Q4_K_S 64 pp2048 383.54 383.64 1.00
RX 6800 llama 8B Q4_K_S 128 pp2048 467.63 467.89 1.00
RX 6800 llama 8B Q4_K_S 256 pp2048 567.23 567.44 1.00
RX 6800 llama 8B Q4_K_S 512 pp2048 615.90 616.18 1.00
RX 6800 llama 8B Q4_K_S 1024 pp2048 637.52 637.78 1.00
RX 6800 llama 8B Q4_K_S 2048 pp2048 652.12 652.37 1.00
RX 6800 llama 8B Q5_0 16 pp2048 215.46 215.68 1.00
RX 6800 llama 8B Q5_0 32 pp2048 168.58 168.54 1.00
RX 6800 llama 8B Q5_0 64 pp2048 525.61 525.18 1.00
RX 6800 llama 8B Q5_0 128 pp2048 662.39 662.38 1.00
RX 6800 llama 8B Q5_0 256 pp2048 765.23 764.81 1.00
RX 6800 llama 8B Q5_0 512 pp2048 818.92 818.87 1.00
RX 6800 llama 8B Q5_0 1024 pp2048 839.25 839.37 1.00
RX 6800 llama 8B Q5_0 2048 pp2048 852.44 852.69 1.00
RX 6800 llama 8B Q5_1 16 pp2048 197.31 197.51 1.00
RX 6800 llama 8B Q5_1 32 pp2048 392.62 392.01 1.00
RX 6800 llama 8B Q5_1 64 pp2048 524.40 525.12 1.00
RX 6800 llama 8B Q5_1 128 pp2048 671.80 672.63 1.00
RX 6800 llama 8B Q5_1 256 pp2048 783.18 783.86 1.00
RX 6800 llama 8B Q5_1 512 pp2048 841.29 841.96 1.00
RX 6800 llama 8B Q5_1 1024 pp2048 864.39 865.33 1.00
RX 6800 llama 8B Q5_1 2048 pp2048 880.91 881.88 1.00
RX 6800 llama 8B Q5_K_S 16 pp2048 232.43 232.85 1.00
RX 6800 llama 8B Q5_K_S 32 pp2048 360.95 360.55 1.00
RX 6800 llama 8B Q5_K_S 64 pp2048 383.91 384.45 1.00
RX 6800 llama 8B Q5_K_S 128 pp2048 469.79 470.03 1.00
RX 6800 llama 8B Q5_K_S 256 pp2048 569.56 569.94 1.00
RX 6800 llama 8B Q5_K_S 512 pp2048 618.62 618.94 1.00
RX 6800 llama 8B Q5_K_S 1024 pp2048 640.50 640.91 1.00
RX 6800 llama 8B Q5_K_S 2048 pp2048 655.01 655.27 1.00
RX 6800 llama 8B Q6_K 16 pp2048 155.63 155.70 1.00
RX 6800 llama 8B Q6_K 32 pp2048 327.58 327.80 1.00
RX 6800 llama 8B Q6_K 64 pp2048 337.93 338.35 1.00
RX 6800 llama 8B Q6_K 128 pp2048 416.39 416.85 1.00
RX 6800 llama 8B Q6_K 256 pp2048 500.43 500.92 1.00
RX 6800 llama 8B Q6_K 512 pp2048 542.02 542.50 1.00
RX 6800 llama 8B Q6_K 1024 pp2048 558.55 558.94 1.00
RX 6800 llama 8B Q6_K 2048 pp2048 569.92 570.22 1.00
RX 6800 llama 8B Q8_0 16 pp2048 264.47 264.61 1.00
RX 6800 llama 8B Q8_0 32 pp2048 260.27 264.50 1.02
RX 6800 llama 8B Q8_0 64 pp2048 549.22 549.38 1.00
RX 6800 llama 8B Q8_0 128 pp2048 698.78 698.19 1.00
RX 6800 llama 8B Q8_0 256 pp2048 820.77 820.10 1.00
RX 6800 llama 8B Q8_0 512 pp2048 884.36 884.15 1.00
RX 6800 llama 8B Q8_0 1024 pp2048 911.15 911.88 1.00
RX 6800 llama 8B Q8_0 2048 pp2048 930.58 931.11 1.00
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 16 pp2048 632.83 632.58 1.00
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 32 pp2048 859.28 862.84 1.00
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 64 pp2048 447.41 454.59 1.02
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 128 pp2048 1704.34 1692.45 0.99
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 256 pp2048 1824.88 1812.74 0.99
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 512 pp2048 1854.85 1842.25 0.99
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 1024 pp2048 1833.28 1819.91 0.99
RX 9060 XT llama 8B IQ1_S - 1.5625 bpw 2048 pp2048 1739.63 1731.95 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 16 pp2048 410.76 410.37 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 32 pp2048 543.18 542.99 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 64 pp2048 483.16 479.17 0.99
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 128 pp2048 1455.56 1452.68 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 256 pp2048 1546.48 1543.05 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 512 pp2048 1584.44 1580.47 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 1024 pp2048 1560.08 1557.55 1.00
RX 9060 XT llama 8B IQ2_S - 2.5 bpw 2048 pp2048 1482.69 1480.84 1.00
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 16 pp2048 399.25 397.55 1.00
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 32 pp2048 531.95 530.92 1.00
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 64 pp2048 460.32 457.40 0.99
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 128 pp2048 1630.84 1620.10 0.99
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 256 pp2048 1744.62 1733.18 0.99
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 512 pp2048 1779.85 1769.11 0.99
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 1024 pp2048 1771.61 1760.43 0.99
RX 9060 XT llama 8B IQ2_XS - 2.3125 bpw 2048 pp2048 1725.29 1713.38 0.99
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 16 pp2048 522.37 521.90 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 32 pp2048 707.92 710.11 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 64 pp2048 1022.32 1020.67 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 128 pp2048 726.86 726.69 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 256 pp2048 818.86 817.64 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 512 pp2048 833.62 832.35 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 1024 pp2048 828.21 826.47 1.00
RX 9060 XT llama 8B IQ2_XXS - 2.0625 bpw 2048 pp2048 816.50 815.16 1.00
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 16 pp2048 507.70 507.73 1.00
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 32 pp2048 688.42 688.72 1.00
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 64 pp2048 955.98 955.19 1.00
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 128 pp2048 690.07 693.32 1.00
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 256 pp2048 770.38 773.15 1.00
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 512 pp2048 786.85 791.22 1.01
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 1024 pp2048 781.25 786.51 1.01
RX 9060 XT llama 8B IQ3_S - 3.4375 bpw 2048 pp2048 768.44 773.79 1.01
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 16 pp2048 516.94 516.09 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 32 pp2048 701.31 703.00 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 64 pp2048 968.23 966.38 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 128 pp2048 741.30 742.50 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 256 pp2048 831.22 832.80 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 512 pp2048 848.87 851.44 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 1024 pp2048 844.14 846.90 1.00
RX 9060 XT llama 8B IQ3_S mix - 3.66 bpw 2048 pp2048 830.09 832.50 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 16 pp2048 508.67 509.50 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 32 pp2048 685.47 686.11 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 64 pp2048 972.34 971.18 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 128 pp2048 700.73 700.99 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 256 pp2048 780.90 782.18 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 512 pp2048 798.87 801.45 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 1024 pp2048 794.80 796.31 1.00
RX 9060 XT llama 8B IQ3_XS - 3.3 bpw 2048 pp2048 781.88 783.23 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 16 pp2048 494.71 494.46 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 32 pp2048 656.76 656.20 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 64 pp2048 852.08 848.89 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 128 pp2048 768.39 766.50 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 256 pp2048 848.98 848.04 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 512 pp2048 857.97 857.25 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 1024 pp2048 849.17 849.37 1.00
RX 9060 XT llama 8B IQ3_XXS - 3.0625 bpw 2048 pp2048 828.37 828.86 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 16 pp2048 651.46 651.87 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 32 pp2048 891.52 894.57 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 64 pp2048 850.88 852.97 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 128 pp2048 683.13 682.11 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 256 pp2048 757.73 755.46 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 512 pp2048 772.46 770.62 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 1024 pp2048 766.61 765.43 1.00
RX 9060 XT llama 8B IQ4_NL - 4.5 bpw 2048 pp2048 755.00 753.59 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 16 pp2048 676.93 679.51 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 32 pp2048 902.36 907.47 1.01
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 64 pp2048 820.16 820.84 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 128 pp2048 687.88 686.65 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 256 pp2048 769.82 767.90 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 512 pp2048 787.47 786.00 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 1024 pp2048 782.85 781.70 1.00
RX 9060 XT llama 8B IQ4_XS - 4.25 bpw 2048 pp2048 771.52 770.84 1.00
RX 9060 XT llama 8B Q2_K_M 16 pp2048 439.58 431.53 0.98
RX 9060 XT llama 8B Q2_K_M 32 pp2048 523.79 524.01 1.00
RX 9060 XT llama 8B Q2_K_M 64 pp2048 632.32 634.06 1.00
RX 9060 XT llama 8B Q2_K_M 128 pp2048 1137.47 1141.24 1.00
RX 9060 XT llama 8B Q2_K_M 256 pp2048 1218.11 1223.29 1.00
RX 9060 XT llama 8B Q2_K_M 512 pp2048 1220.73 1228.39 1.01
RX 9060 XT llama 8B Q2_K_M 1024 pp2048 1263.28 1273.34 1.01
RX 9060 XT llama 8B Q2_K_M 2048 pp2048 1244.26 1252.30 1.01
RX 9060 XT llama 8B Q3_K_S 16 pp2048 508.73 516.09 1.01
RX 9060 XT llama 8B Q3_K_S 32 pp2048 528.81 528.26 1.00
RX 9060 XT llama 8B Q3_K_S 64 pp2048 421.80 422.72 1.00
RX 9060 XT llama 8B Q3_K_S 128 pp2048 1819.69 1824.27 1.00
RX 9060 XT llama 8B Q3_K_S 256 pp2048 1943.24 1942.31 1.00
RX 9060 XT llama 8B Q3_K_S 512 pp2048 1991.42 1994.64 1.00
RX 9060 XT llama 8B Q3_K_S 1024 pp2048 1976.18 1979.74 1.00
RX 9060 XT llama 8B Q3_K_S 2048 pp2048 1920.92 1923.83 1.00
RX 9060 XT llama 8B Q4_0 16 pp2048 629.47 630.89 1.00
RX 9060 XT llama 8B Q4_0 32 pp2048 898.29 900.44 1.00
RX 9060 XT llama 8B Q4_0 64 pp2048 869.30 866.14 1.00
RX 9060 XT llama 8B Q4_0 128 pp2048 645.39 641.36 0.99
RX 9060 XT llama 8B Q4_0 256 pp2048 734.34 730.80 1.00
RX 9060 XT llama 8B Q4_0 512 pp2048 754.37 751.52 1.00
RX 9060 XT llama 8B Q4_0 1024 pp2048 749.11 746.49 1.00
RX 9060 XT llama 8B Q4_0 2048 pp2048 739.55 737.22 1.00
RX 9060 XT llama 8B Q4_1 16 pp2048 618.72 619.03 1.00
RX 9060 XT llama 8B Q4_1 32 pp2048 883.72 886.59 1.00
RX 9060 XT llama 8B Q4_1 64 pp2048 407.52 395.15 0.97
RX 9060 XT llama 8B Q4_1 128 pp2048 1945.45 1955.37 1.01
RX 9060 XT llama 8B Q4_1 256 pp2048 2085.00 2097.52 1.01
RX 9060 XT llama 8B Q4_1 512 pp2048 2146.91 2157.90 1.01
RX 9060 XT llama 8B Q4_1 1024 pp2048 2124.91 2134.75 1.00
RX 9060 XT llama 8B Q4_1 2048 pp2048 2056.87 2068.27 1.01
RX 9060 XT llama 8B Q4_K_S 16 pp2048 605.57 606.84 1.00
RX 9060 XT llama 8B Q4_K_S 32 pp2048 847.22 851.61 1.01
RX 9060 XT llama 8B Q4_K_S 64 pp2048 1052.45 1053.45 1.00
RX 9060 XT llama 8B Q4_K_S 128 pp2048 1972.77 1971.46 1.00
RX 9060 XT llama 8B Q4_K_S 256 pp2048 2115.51 2111.85 1.00
RX 9060 XT llama 8B Q4_K_S 512 pp2048 2167.79 2159.82 1.00
RX 9060 XT llama 8B Q4_K_S 1024 pp2048 2145.83 2143.40 1.00
RX 9060 XT llama 8B Q4_K_S 2048 pp2048 2081.04 2075.15 1.00
RX 9060 XT llama 8B Q5_0 16 pp2048 545.88 547.12 1.00
RX 9060 XT llama 8B Q5_0 32 pp2048 796.78 799.34 1.00
RX 9060 XT llama 8B Q5_0 64 pp2048 805.53 804.29 1.00
RX 9060 XT llama 8B Q5_0 128 pp2048 593.27 593.80 1.00
RX 9060 XT llama 8B Q5_0 256 pp2048 665.70 667.17 1.00
RX 9060 XT llama 8B Q5_0 512 pp2048 684.15 685.57 1.00
RX 9060 XT llama 8B Q5_0 1024 pp2048 680.33 681.87 1.00
RX 9060 XT llama 8B Q5_0 2048 pp2048 671.60 673.02 1.00
RX 9060 XT llama 8B Q5_1 16 pp2048 456.29 457.46 1.00
RX 9060 XT llama 8B Q5_1 32 pp2048 717.35 719.07 1.00
RX 9060 XT llama 8B Q5_1 64 pp2048 334.16 336.26 1.01
RX 9060 XT llama 8B Q5_1 128 pp2048 1786.20 1792.72 1.00
RX 9060 XT llama 8B Q5_1 256 pp2048 1928.66 1936.24 1.00
RX 9060 XT llama 8B Q5_1 512 pp2048 1992.35 2001.91 1.00
RX 9060 XT llama 8B Q5_1 1024 pp2048 1977.35 1990.62 1.01
RX 9060 XT llama 8B Q5_1 2048 pp2048 1916.80 1927.42 1.01
RX 9060 XT llama 8B Q5_K_S 16 pp2048 585.25 586.99 1.00
RX 9060 XT llama 8B Q5_K_S 32 pp2048 549.77 548.89 1.00
RX 9060 XT llama 8B Q5_K_S 64 pp2048 390.37 406.30 1.04
RX 9060 XT llama 8B Q5_K_S 128 pp2048 1907.57 1895.92 0.99
RX 9060 XT llama 8B Q5_K_S 256 pp2048 2040.98 2025.24 0.99
RX 9060 XT llama 8B Q5_K_S 512 pp2048 2101.98 2087.18 0.99
RX 9060 XT llama 8B Q5_K_S 1024 pp2048 2081.26 2065.57 0.99
RX 9060 XT llama 8B Q5_K_S 2048 pp2048 2014.91 2000.89 0.99
RX 9060 XT llama 8B Q6_K 16 pp2048 457.45 457.47 1.00
RX 9060 XT llama 8B Q6_K 32 pp2048 292.72 290.46 0.99
RX 9060 XT llama 8B Q6_K 64 pp2048 955.31 951.86 1.00
RX 9060 XT llama 8B Q6_K 128 pp2048 1218.68 1211.78 0.99
RX 9060 XT llama 8B Q6_K 256 pp2048 1310.95 1300.21 0.99
RX 9060 XT llama 8B Q6_K 512 pp2048 1331.76 1321.87 0.99
RX 9060 XT llama 8B Q6_K 1024 pp2048 1326.31 1316.07 0.99
RX 9060 XT llama 8B Q6_K 2048 pp2048 1299.07 1291.23 0.99
RX 9060 XT llama 8B Q8_0 16 pp2048 467.43 467.75 1.00
RX 9060 XT llama 8B Q8_0 32 pp2048 757.82 759.25 1.00
RX 9060 XT llama 8B Q8_0 64 pp2048 787.15 784.44 1.00
RX 9060 XT llama 8B Q8_0 128 pp2048 619.85 620.62 1.00
RX 9060 XT llama 8B Q8_0 256 pp2048 689.15 690.61 1.00
RX 9060 XT llama 8B Q8_0 512 pp2048 707.45 709.39 1.00
RX 9060 XT llama 8B Q8_0 1024 pp2048 705.35 706.97 1.00
RX 9060 XT llama 8B Q8_0 2048 pp2048 697.37 699.20 1.00
Quantization sweep AMD MoE
GPU Model Microbatch size Test t/s 82209ef t/s 1b2cf95 Speedup
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 506.30 507.14 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 743.66 744.35 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 966.93 966.94 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 1704.24 1705.10 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 2589.94 2592.36 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 3341.86 3347.06 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 3701.48 3711.01 1.00
MI60 / MI50 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 3897.71 3915.89 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 416.99 415.19 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 707.07 706.19 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 945.96 940.28 0.99
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 1656.80 1648.67 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 2477.27 2475.22 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 3229.00 3231.95 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 3571.60 3580.38 1.00
MI60 / MI50 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 3744.08 3761.79 1.00
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 414.03 413.77 1.00
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 698.97 699.60 1.00
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 898.40 890.54 0.99
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 1563.92 1550.23 0.99
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 2358.63 2347.66 1.00
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 3069.46 3069.21 1.00
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 3409.79 3412.57 1.00
MI60 / MI50 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 3612.92 3623.57 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 482.17 481.92 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 679.98 676.37 0.99
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 943.99 943.11 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 1644.08 1644.14 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 2497.99 2502.30 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 3234.38 3241.36 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 3583.93 3594.62 1.00
MI60 / MI50 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 3779.45 3797.72 1.00
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 467.47 466.99 1.00
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 726.82 721.86 0.99
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 973.29 967.93 0.99
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 1699.67 1693.80 1.00
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 2590.33 2582.04 1.00
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 3330.46 3325.69 1.00
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 3646.07 3646.29 1.00
MI60 / MI50 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 3827.01 3829.89 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 466.68 468.48 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 703.13 702.23 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 970.47 969.05 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 1699.40 1696.37 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 2598.95 2597.63 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 3404.54 3410.57 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 3779.54 3792.80 1.00
MI60 / MI50 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 3961.66 3981.92 1.01
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 440.10 442.05 1.00
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 687.68 683.00 0.99
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 959.72 957.64 1.00
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 1684.47 1684.20 1.00
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 2564.68 2565.56 1.00
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 3313.42 3321.76 1.00
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 3646.13 3654.85 1.00
MI60 / MI50 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 3836.93 3845.37 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 416.76 419.11 1.01
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 680.68 678.89 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 960.42 959.76 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 1683.37 1682.93 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 2563.08 2564.27 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 3317.27 3321.97 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 3635.35 3642.94 1.00
MI60 / MI50 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 3826.60 3844.23 1.00
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 493.97 491.03 0.99
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 860.82 854.06 0.99
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 886.21 886.96 1.00
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 1550.22 1552.03 1.00
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 2392.07 2397.76 1.00
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 3085.37 3090.70 1.00
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 3391.84 3395.35 1.00
MI60 / MI50 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 3600.93 3605.11 1.00
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 527.38 527.74 1.00
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 854.30 854.43 1.00
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 1082.41 1097.14 1.01
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 1919.10 1940.42 1.01
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 2862.44 2888.91 1.01
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 3666.43 3698.85 1.01
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 4014.38 4043.52 1.01
MI60 / MI50 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 4223.93 4249.78 1.01
MI60 / MI50 granitemoe 3B Q2_K_M 16 pp2048 389.64 389.24 1.00
MI60 / MI50 granitemoe 3B Q2_K_M 32 pp2048 545.19 548.56 1.01
MI60 / MI50 granitemoe 3B Q2_K_M 64 pp2048 475.94 481.70 1.01
MI60 / MI50 granitemoe 3B Q2_K_M 128 pp2048 817.11 828.76 1.01
MI60 / MI50 granitemoe 3B Q2_K_M 256 pp2048 1217.29 1240.77 1.02
MI60 / MI50 granitemoe 3B Q2_K_M 512 pp2048 1571.07 1604.37 1.02
MI60 / MI50 granitemoe 3B Q2_K_M 1024 pp2048 1736.96 1778.26 1.02
MI60 / MI50 granitemoe 3B Q2_K_M 2048 pp2048 1862.22 1910.52 1.03
MI60 / MI50 granitemoe 3B Q3_K_S 16 pp2048 324.76 325.50 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 32 pp2048 665.93 666.58 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 64 pp2048 874.08 873.43 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 128 pp2048 1538.27 1537.34 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 256 pp2048 2343.44 2344.89 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 512 pp2048 3052.69 3052.93 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 1024 pp2048 3351.71 3354.58 1.00
MI60 / MI50 granitemoe 3B Q3_K_S 2048 pp2048 3563.00 3574.16 1.00
MI60 / MI50 granitemoe 3B Q4_0 16 pp2048 589.74 585.16 0.99
MI60 / MI50 granitemoe 3B Q4_0 32 pp2048 857.46 855.92 1.00
MI60 / MI50 granitemoe 3B Q4_0 64 pp2048 1382.95 1381.41 1.00
MI60 / MI50 granitemoe 3B Q4_0 128 pp2048 2429.18 2429.89 1.00
MI60 / MI50 granitemoe 3B Q4_0 256 pp2048 3644.46 3650.28 1.00
MI60 / MI50 granitemoe 3B Q4_0 512 pp2048 4847.76 4877.85 1.01
MI60 / MI50 granitemoe 3B Q4_0 1024 pp2048 5341.22 5420.09 1.01
MI60 / MI50 granitemoe 3B Q4_0 2048 pp2048 5477.67 5595.27 1.02
MI60 / MI50 granitemoe 3B Q4_1 16 pp2048 582.82 579.95 1.00
MI60 / MI50 granitemoe 3B Q4_1 32 pp2048 864.52 863.13 1.00
MI60 / MI50 granitemoe 3B Q4_1 64 pp2048 1363.11 1359.93 1.00
MI60 / MI50 granitemoe 3B Q4_1 128 pp2048 2393.70 2393.53 1.00
MI60 / MI50 granitemoe 3B Q4_1 256 pp2048 3614.81 3615.34 1.00
MI60 / MI50 granitemoe 3B Q4_1 512 pp2048 4837.85 4845.73 1.00
MI60 / MI50 granitemoe 3B Q4_1 1024 pp2048 5355.92 5408.21 1.01
MI60 / MI50 granitemoe 3B Q4_1 2048 pp2048 5495.97 5573.34 1.01
MI60 / MI50 granitemoe 3B Q4_K_S 16 pp2048 590.66 579.84 0.98
MI60 / MI50 granitemoe 3B Q4_K_S 32 pp2048 820.62 821.29 1.00
MI60 / MI50 granitemoe 3B Q4_K_S 64 pp2048 1201.04 1190.37 0.99
MI60 / MI50 granitemoe 3B Q4_K_S 128 pp2048 2117.35 2097.94 0.99
MI60 / MI50 granitemoe 3B Q4_K_S 256 pp2048 3202.20 3184.35 0.99
MI60 / MI50 granitemoe 3B Q4_K_S 512 pp2048 4279.86 4276.96 1.00
MI60 / MI50 granitemoe 3B Q4_K_S 1024 pp2048 4752.30 4781.27 1.01
MI60 / MI50 granitemoe 3B Q4_K_S 2048 pp2048 4934.96 4999.81 1.01
MI60 / MI50 granitemoe 3B Q5_0 16 pp2048 426.53 427.20 1.00
MI60 / MI50 granitemoe 3B Q5_0 32 pp2048 750.64 749.01 1.00
MI60 / MI50 granitemoe 3B Q5_0 64 pp2048 831.48 828.60 1.00
MI60 / MI50 granitemoe 3B Q5_0 128 pp2048 1460.73 1457.49 1.00
MI60 / MI50 granitemoe 3B Q5_0 256 pp2048 2260.62 2258.43 1.00
MI60 / MI50 granitemoe 3B Q5_0 512 pp2048 2919.67 2918.71 1.00
MI60 / MI50 granitemoe 3B Q5_0 1024 pp2048 3209.09 3212.41 1.00
MI60 / MI50 granitemoe 3B Q5_0 2048 pp2048 3410.72 3416.18 1.00
MI60 / MI50 granitemoe 3B Q5_1 16 pp2048 416.60 415.41 1.00
MI60 / MI50 granitemoe 3B Q5_1 32 pp2048 825.39 824.70 1.00
MI60 / MI50 granitemoe 3B Q5_1 64 pp2048 844.26 847.86 1.00
MI60 / MI50 granitemoe 3B Q5_1 128 pp2048 1492.92 1500.10 1.00
MI60 / MI50 granitemoe 3B Q5_1 256 pp2048 2299.00 2312.55 1.01
MI60 / MI50 granitemoe 3B Q5_1 512 pp2048 2966.70 2983.52 1.01
MI60 / MI50 granitemoe 3B Q5_1 1024 pp2048 3257.30 3278.58 1.01
MI60 / MI50 granitemoe 3B Q5_1 2048 pp2048 3457.10 3483.86 1.01
MI60 / MI50 granitemoe 3B Q5_K_S 16 pp2048 339.60 342.20 1.01
MI60 / MI50 granitemoe 3B Q5_K_S 32 pp2048 850.75 854.40 1.00
MI60 / MI50 granitemoe 3B Q5_K_S 64 pp2048 791.86 793.38 1.00
MI60 / MI50 granitemoe 3B Q5_K_S 128 pp2048 1415.28 1415.78 1.00
MI60 / MI50 granitemoe 3B Q5_K_S 256 pp2048 2188.91 2192.74 1.00
MI60 / MI50 granitemoe 3B Q5_K_S 512 pp2048 2840.31 2841.87 1.00
MI60 / MI50 granitemoe 3B Q5_K_S 1024 pp2048 3134.03 3136.88 1.00
MI60 / MI50 granitemoe 3B Q5_K_S 2048 pp2048 3352.65 3358.35 1.00
MI60 / MI50 granitemoe 3B Q6_K 16 pp2048 300.70 304.44 1.01
MI60 / MI50 granitemoe 3B Q6_K 32 pp2048 597.76 605.68 1.01
MI60 / MI50 granitemoe 3B Q6_K 64 pp2048 882.74 813.57 0.92
MI60 / MI50 granitemoe 3B Q6_K 128 pp2048 1563.05 1459.03 0.93
MI60 / MI50 granitemoe 3B Q6_K 256 pp2048 2367.03 2220.86 0.94
MI60 / MI50 granitemoe 3B Q6_K 512 pp2048 3082.23 2898.16 0.94
MI60 / MI50 granitemoe 3B Q6_K 1024 pp2048 3419.40 3217.62 0.94
MI60 / MI50 granitemoe 3B Q6_K 2048 pp2048 3631.62 3436.57 0.95
MI60 / MI50 granitemoe 3B Q8_0 16 pp2048 298.08 297.26 1.00
MI60 / MI50 granitemoe 3B Q8_0 32 pp2048 834.99 835.77 1.00
MI60 / MI50 granitemoe 3B Q8_0 64 pp2048 766.24 761.93 0.99
MI60 / MI50 granitemoe 3B Q8_0 128 pp2048 1393.07 1384.91 0.99
MI60 / MI50 granitemoe 3B Q8_0 256 pp2048 2126.43 2118.27 1.00
MI60 / MI50 granitemoe 3B Q8_0 512 pp2048 2758.15 2746.74 1.00
MI60 / MI50 granitemoe 3B Q8_0 1024 pp2048 3045.58 3036.93 1.00
MI60 / MI50 granitemoe 3B Q8_0 2048 pp2048 3260.71 3256.82 1.00
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 929.53 1010.82 1.09
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 1582.93 1718.20 1.09
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 2293.33 2416.42 1.05
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 3521.24 3664.52 1.04
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 1810.85 1975.12 1.09
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 2907.47 3114.94 1.07
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 4493.12 4758.59 1.06
MI100 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 6114.25 6365.25 1.04
MI100 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 779.02 839.25 1.08
MI100 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 1367.46 1419.37 1.04
MI100 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 1906.60 2002.29 1.05
MI100 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 2835.40 2982.80 1.05
MI100 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 1614.37 1756.10 1.09
MI100 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 2603.83 2804.34 1.08
MI100 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 4103.01 4382.90 1.07
MI100 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 5710.13 5948.29 1.04
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 814.94 867.77 1.06
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 1433.76 1461.78 1.02
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 1925.85 2047.05 1.06
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 2826.35 3000.23 1.06
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 1598.69 1732.98 1.08
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 2601.57 2789.15 1.07
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 4112.90 4399.60 1.07
MI100 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 5767.43 6031.53 1.05
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 872.99 927.26 1.06
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 1506.30 1615.96 1.07
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 2209.13 2351.90 1.06
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 3304.22 3464.41 1.05
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 1691.66 1830.46 1.08
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 2744.22 2935.17 1.07
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 4306.97 4553.32 1.06
MI100 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 5940.49 6180.54 1.04
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 855.23 904.35 1.06
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 1493.30 1579.64 1.06
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 2193.31 2284.20 1.04
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 3226.38 3298.76 1.02
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 1688.55 1818.56 1.08
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 2712.67 2902.29 1.07
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 4235.89 4492.90 1.06
MI100 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 5790.22 6049.60 1.04
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 829.21 888.50 1.07
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 1423.77 1542.15 1.08
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 2079.25 2206.73 1.06
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 3113.69 3225.67 1.04
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 1698.62 1846.88 1.09
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 2703.52 2894.73 1.07
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 4226.97 4510.66 1.07
MI100 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 5773.82 6021.13 1.04
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 804.38 870.28 1.08
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 1401.86 1503.31 1.07
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 2065.89 2192.00 1.06
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 3057.19 3194.90 1.05
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 1696.62 1836.45 1.08
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 2729.56 2914.31 1.07
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 4242.31 4530.40 1.07
MI100 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 5790.69 6045.65 1.04
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 780.08 846.99 1.09
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 1368.43 1478.01 1.08
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 2032.78 2139.26 1.05
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 3001.17 3110.86 1.04
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 1732.26 1867.09 1.08
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 2762.23 2948.98 1.07
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 4269.92 4547.92 1.07
MI100 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 5822.37 6084.51 1.05
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 1029.19 992.95 0.96
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 1726.57 1864.45 1.08
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 2474.89 2580.48 1.04
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 3707.95 3835.70 1.03
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 1872.63 2084.41 1.11
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 2946.54 3230.68 1.10
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 4521.31 4837.70 1.07
MI100 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 6030.92 6325.84 1.05
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 1057.81 1139.98 1.08
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 1807.62 1913.71 1.06
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 2588.54 2630.94 1.02
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 3941.63 3899.36 0.99
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 1960.42 2127.83 1.09
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 3060.06 3279.48 1.07
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 4620.24 4885.39 1.06
MI100 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 6130.18 6384.05 1.04
MI100 granitemoe 3B Q2_K_M 16 pp2048 867.33 955.37 1.10
MI100 granitemoe 3B Q2_K_M 32 pp2048 1382.15 1492.31 1.08
MI100 granitemoe 3B Q2_K_M 64 pp2048 1785.36 1847.74 1.03
MI100 granitemoe 3B Q2_K_M 128 pp2048 2545.51 2666.68 1.05
MI100 granitemoe 3B Q2_K_M 256 pp2048 1624.21 1756.89 1.08
MI100 granitemoe 3B Q2_K_M 512 pp2048 2610.63 2805.88 1.07
MI100 granitemoe 3B Q2_K_M 1024 pp2048 4125.80 4387.52 1.06
MI100 granitemoe 3B Q2_K_M 2048 pp2048 5812.49 6041.53 1.04
MI100 granitemoe 3B Q3_K_S 16 pp2048 964.15 1024.55 1.06
MI100 granitemoe 3B Q3_K_S 32 pp2048 1605.06 1629.49 1.02
MI100 granitemoe 3B Q3_K_S 64 pp2048 1968.70 2033.18 1.03
MI100 granitemoe 3B Q3_K_S 128 pp2048 2918.12 3002.02 1.03
MI100 granitemoe 3B Q3_K_S 256 pp2048 1683.72 1832.51 1.09
MI100 granitemoe 3B Q3_K_S 512 pp2048 2710.16 2910.55 1.07
MI100 granitemoe 3B Q3_K_S 1024 pp2048 4259.60 4518.49 1.06
MI100 granitemoe 3B Q3_K_S 2048 pp2048 5875.86 6088.69 1.04
MI100 granitemoe 3B Q4_0 16 pp2048 977.06 1074.94 1.10
MI100 granitemoe 3B Q4_0 32 pp2048 1705.21 1833.04 1.07
MI100 granitemoe 3B Q4_0 64 pp2048 2425.82 2573.98 1.06
MI100 granitemoe 3B Q4_0 128 pp2048 3679.87 3878.01 1.05
MI100 granitemoe 3B Q4_0 256 pp2048 5118.00 5363.49 1.05
MI100 granitemoe 3B Q4_0 512 pp2048 5999.36 6218.22 1.04
MI100 granitemoe 3B Q4_0 1024 pp2048 6627.21 6838.37 1.03
MI100 granitemoe 3B Q4_0 2048 pp2048 6756.41 6926.72 1.03
MI100 granitemoe 3B Q4_1 16 pp2048 1010.80 1108.21 1.10
MI100 granitemoe 3B Q4_1 32 pp2048 1731.88 1856.29 1.07
MI100 granitemoe 3B Q4_1 64 pp2048 2467.32 2641.58 1.07
MI100 granitemoe 3B Q4_1 128 pp2048 3800.74 3979.89 1.05
MI100 granitemoe 3B Q4_1 256 pp2048 5283.84 5537.56 1.05
MI100 granitemoe 3B Q4_1 512 pp2048 6208.59 6432.58 1.04
MI100 granitemoe 3B Q4_1 1024 pp2048 6856.34 7090.92 1.03
MI100 granitemoe 3B Q4_1 2048 pp2048 6969.97 7206.93 1.03
MI100 granitemoe 3B Q4_K_S 16 pp2048 1039.70 1110.49 1.07
MI100 granitemoe 3B Q4_K_S 32 pp2048 1647.77 1921.91 1.17
MI100 granitemoe 3B Q4_K_S 64 pp2048 2381.63 2542.07 1.07
MI100 granitemoe 3B Q4_K_S 128 pp2048 3565.63 3816.55 1.07
MI100 granitemoe 3B Q4_K_S 256 pp2048 4931.84 5281.33 1.07
MI100 granitemoe 3B Q4_K_S 512 pp2048 2851.59 3106.89 1.09
MI100 granitemoe 3B Q4_K_S 1024 pp2048 4383.96 4733.29 1.08
MI100 granitemoe 3B Q4_K_S 2048 pp2048 5925.06 6232.38 1.05
MI100 granitemoe 3B Q5_0 16 pp2048 863.03 924.30 1.07
MI100 granitemoe 3B Q5_0 32 pp2048 1526.90 1641.71 1.08
MI100 granitemoe 3B Q5_0 64 pp2048 2291.47 2432.49 1.06
MI100 granitemoe 3B Q5_0 128 pp2048 3415.90 3594.82 1.05
MI100 granitemoe 3B Q5_0 256 pp2048 4693.57 4915.27 1.05
MI100 granitemoe 3B Q5_0 512 pp2048 5440.97 5690.69 1.05
MI100 granitemoe 3B Q5_0 1024 pp2048 5959.69 6234.62 1.05
MI100 granitemoe 3B Q5_0 2048 pp2048 6054.29 6328.64 1.05
MI100 granitemoe 3B Q5_1 16 pp2048 911.99 1008.15 1.11
MI100 granitemoe 3B Q5_1 32 pp2048 1665.23 1663.33 1.00
MI100 granitemoe 3B Q5_1 64 pp2048 2360.51 2495.90 1.06
MI100 granitemoe 3B Q5_1 128 pp2048 3615.76 3763.02 1.04
MI100 granitemoe 3B Q5_1 256 pp2048 5014.48 5175.52 1.03
MI100 granitemoe 3B Q5_1 512 pp2048 5857.63 6033.13 1.03
MI100 granitemoe 3B Q5_1 1024 pp2048 6458.87 6641.70 1.03
MI100 granitemoe 3B Q5_1 2048 pp2048 6556.15 6747.08 1.03
MI100 granitemoe 3B Q5_K_S 16 pp2048 905.74 963.94 1.06
MI100 granitemoe 3B Q5_K_S 32 pp2048 1781.27 1679.60 0.94
MI100 granitemoe 3B Q5_K_S 64 pp2048 2369.98 2452.06 1.03
MI100 granitemoe 3B Q5_K_S 128 pp2048 3522.78 3649.54 1.04
MI100 granitemoe 3B Q5_K_S 256 pp2048 4847.43 4996.80 1.03
MI100 granitemoe 3B Q5_K_S 512 pp2048 2812.16 3042.48 1.08
MI100 granitemoe 3B Q5_K_S 1024 pp2048 4353.22 4666.95 1.07
MI100 granitemoe 3B Q5_K_S 2048 pp2048 5897.25 6194.22 1.05
MI100 granitemoe 3B Q6_K 16 pp2048 894.38 922.36 1.03
MI100 granitemoe 3B Q6_K 32 pp2048 1431.17 1517.78 1.06
MI100 granitemoe 3B Q6_K 64 pp2048 1659.07 1808.50 1.09
MI100 granitemoe 3B Q6_K 128 pp2048 2434.44 2614.02 1.07
MI100 granitemoe 3B Q6_K 256 pp2048 1651.87 1805.19 1.09
MI100 granitemoe 3B Q6_K 512 pp2048 2673.74 2870.10 1.07
MI100 granitemoe 3B Q6_K 1024 pp2048 4204.28 4478.42 1.07
MI100 granitemoe 3B Q6_K 2048 pp2048 5822.41 6055.93 1.04
MI100 granitemoe 3B Q8_0 16 pp2048 929.35 1012.33 1.09
MI100 granitemoe 3B Q8_0 32 pp2048 1643.97 1712.89 1.04
MI100 granitemoe 3B Q8_0 64 pp2048 2397.40 2499.83 1.04
MI100 granitemoe 3B Q8_0 128 pp2048 3664.35 3757.27 1.03
MI100 granitemoe 3B Q8_0 256 pp2048 1845.18 2083.85 1.13
MI100 granitemoe 3B Q8_0 512 pp2048 2935.19 3235.38 1.10
MI100 granitemoe 3B Q8_0 1024 pp2048 4542.14 4902.30 1.08
MI100 granitemoe 3B Q8_0 2048 pp2048 6094.28 6417.40 1.05
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 766.51 763.99 1.00
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 883.32 885.86 1.00
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 643.50 617.92 0.96
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 1673.17 1684.99 1.01
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 2369.32 2374.82 1.00
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 2965.21 2973.50 1.00
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 3236.09 3266.47 1.01
Radeon 8060S Graphics granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 3306.89 3309.77 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 598.83 602.53 1.01
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 857.06 859.79 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 653.07 652.86 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 1476.63 1479.62 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 2182.01 2186.66 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 2719.89 2705.79 0.99
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 2991.85 2979.32 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 3065.61 3065.02 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 592.75 591.99 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 840.69 838.48 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 627.07 624.64 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 1591.79 1583.51 0.99
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 2253.50 2251.99 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 2847.92 2836.05 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 3159.56 3160.08 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 3248.88 3244.93 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 691.78 688.35 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 808.08 808.24 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 1221.24 1228.50 1.01
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 895.05 898.96 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 1346.48 1349.47 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 1736.51 1731.63 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 1949.09 1955.55 1.00
Radeon 8060S Graphics granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 2047.62 2065.34 1.01
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 632.73 631.64 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 916.99 914.93 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 1143.44 1145.45 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 859.59 852.61 0.99
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 1320.85 1325.46 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 1684.40 1694.95 1.01
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 1935.79 1901.86 0.98
Radeon 8060S Graphics granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 1983.93 1994.82 1.01
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 642.08 643.06 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 920.91 923.80 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 1157.29 1157.92 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 917.24 919.88 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 1396.90 1388.87 0.99
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 1768.02 1797.82 1.02
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 2026.43 2017.23 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 2107.52 2101.11 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 646.31 646.01 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 947.05 947.03 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 1164.96 1160.12 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 852.21 865.55 1.02
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 1321.28 1331.68 1.01
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 1678.09 1691.44 1.01
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 1938.85 1937.53 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 1999.53 2001.47 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 658.74 660.75 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 956.18 955.48 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 1077.38 1075.55 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 906.02 913.08 1.01
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 1379.24 1384.94 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 1762.42 1770.13 1.00
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 1973.51 2008.87 1.02
Radeon 8060S Graphics granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 2064.76 2083.11 1.01
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 752.31 754.90 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 950.92 950.37 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 1120.68 1120.10 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 832.46 821.63 0.99
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 1291.86 1281.29 0.99
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 1673.56 1654.36 0.99
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 1926.49 1939.13 1.01
Radeon 8060S Graphics granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 2036.64 2017.95 0.99
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 751.39 753.15 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 941.36 941.95 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 1097.09 1099.36 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 830.95 832.43 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 1288.93 1269.83 0.99
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 1675.99 1676.47 1.00
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 1925.44 1914.39 0.99
Radeon 8060S Graphics granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 2029.61 2039.10 1.00
Radeon 8060S Graphics granitemoe 3B Q2_K_M 16 pp2048 485.50 479.66 0.99
Radeon 8060S Graphics granitemoe 3B Q2_K_M 32 pp2048 755.56 754.70 1.00
Radeon 8060S Graphics granitemoe 3B Q2_K_M 64 pp2048 778.73 778.98 1.00
Radeon 8060S Graphics granitemoe 3B Q2_K_M 128 pp2048 1576.21 1584.30 1.01
Radeon 8060S Graphics granitemoe 3B Q2_K_M 256 pp2048 1156.05 1163.06 1.01
Radeon 8060S Graphics granitemoe 3B Q2_K_M 512 pp2048 1601.30 1601.01 1.00
Radeon 8060S Graphics granitemoe 3B Q2_K_M 1024 pp2048 2069.96 2073.21 1.00
Radeon 8060S Graphics granitemoe 3B Q2_K_M 2048 pp2048 2394.88 2389.16 1.00
Radeon 8060S Graphics granitemoe 3B Q3_K_S 16 pp2048 576.40 564.33 0.98
Radeon 8060S Graphics granitemoe 3B Q3_K_S 32 pp2048 818.15 819.72 1.00
Radeon 8060S Graphics granitemoe 3B Q3_K_S 64 pp2048 616.49 617.96 1.00
Radeon 8060S Graphics granitemoe 3B Q3_K_S 128 pp2048 1753.25 1762.75 1.01
Radeon 8060S Graphics granitemoe 3B Q3_K_S 256 pp2048 2584.41 2610.44 1.01
Radeon 8060S Graphics granitemoe 3B Q3_K_S 512 pp2048 3174.15 3232.09 1.02
Radeon 8060S Graphics granitemoe 3B Q3_K_S 1024 pp2048 3463.74 3477.08 1.00
Radeon 8060S Graphics granitemoe 3B Q3_K_S 2048 pp2048 3514.83 3547.04 1.01
Radeon 8060S Graphics granitemoe 3B Q4_0 16 pp2048 713.63 716.07 1.00
Radeon 8060S Graphics granitemoe 3B Q4_0 32 pp2048 770.60 771.98 1.00
Radeon 8060S Graphics granitemoe 3B Q4_0 64 pp2048 1018.73 1007.45 0.99
Radeon 8060S Graphics granitemoe 3B Q4_0 128 pp2048 856.58 848.83 0.99
Radeon 8060S Graphics granitemoe 3B Q4_0 256 pp2048 1318.02 1308.91 0.99
Radeon 8060S Graphics granitemoe 3B Q4_0 512 pp2048 1728.04 1703.82 0.99
Radeon 8060S Graphics granitemoe 3B Q4_0 1024 pp2048 1975.56 1929.72 0.98
Radeon 8060S Graphics granitemoe 3B Q4_0 2048 pp2048 2064.59 2058.01 1.00
Radeon 8060S Graphics granitemoe 3B Q4_1 16 pp2048 734.33 734.95 1.00
Radeon 8060S Graphics granitemoe 3B Q4_1 32 pp2048 820.44 826.94 1.01
Radeon 8060S Graphics granitemoe 3B Q4_1 64 pp2048 583.90 586.14 1.00
Radeon 8060S Graphics granitemoe 3B Q4_1 128 pp2048 1782.40 1793.19 1.01
Radeon 8060S Graphics granitemoe 3B Q4_1 256 pp2048 2652.01 2667.92 1.01
Radeon 8060S Graphics granitemoe 3B Q4_1 512 pp2048 3233.04 3270.15 1.01
Radeon 8060S Graphics granitemoe 3B Q4_1 1024 pp2048 3533.43 3544.06 1.00
Radeon 8060S Graphics granitemoe 3B Q4_1 2048 pp2048 3573.49 3594.76 1.01
Radeon 8060S Graphics granitemoe 3B Q4_K_S 16 pp2048 712.74 711.54 1.00
Radeon 8060S Graphics granitemoe 3B Q4_K_S 32 pp2048 935.30 936.94 1.00
Radeon 8060S Graphics granitemoe 3B Q4_K_S 64 pp2048 1196.97 1188.42 0.99
Radeon 8060S Graphics granitemoe 3B Q4_K_S 128 pp2048 1816.56 1810.99 1.00
Radeon 8060S Graphics granitemoe 3B Q4_K_S 256 pp2048 2683.00 2671.11 1.00
Radeon 8060S Graphics granitemoe 3B Q4_K_S 512 pp2048 3269.47 3251.75 0.99
Radeon 8060S Graphics granitemoe 3B Q4_K_S 1024 pp2048 3575.96 3558.85 1.00
Radeon 8060S Graphics granitemoe 3B Q4_K_S 2048 pp2048 3601.70 3610.41 1.00
Radeon 8060S Graphics granitemoe 3B Q5_0 16 pp2048 651.96 652.99 1.00
Radeon 8060S Graphics granitemoe 3B Q5_0 32 pp2048 651.34 652.87 1.00
Radeon 8060S Graphics granitemoe 3B Q5_0 64 pp2048 947.38 949.76 1.00
Radeon 8060S Graphics granitemoe 3B Q5_0 128 pp2048 783.74 785.56 1.00
Radeon 8060S Graphics granitemoe 3B Q5_0 256 pp2048 1215.44 1224.41 1.01
Radeon 8060S Graphics granitemoe 3B Q5_0 512 pp2048 1561.60 1581.91 1.01
Radeon 8060S Graphics granitemoe 3B Q5_0 1024 pp2048 1830.53 1849.35 1.01
Radeon 8060S Graphics granitemoe 3B Q5_0 2048 pp2048 1968.48 1972.63 1.00
Radeon 8060S Graphics granitemoe 3B Q5_1 16 pp2048 680.68 680.41 1.00
Radeon 8060S Graphics granitemoe 3B Q5_1 32 pp2048 622.03 619.45 1.00
Radeon 8060S Graphics granitemoe 3B Q5_1 64 pp2048 523.00 514.76 0.98
Radeon 8060S Graphics granitemoe 3B Q5_1 128 pp2048 1774.70 1765.35 0.99
Radeon 8060S Graphics granitemoe 3B Q5_1 256 pp2048 2632.49 2621.83 1.00
Radeon 8060S Graphics granitemoe 3B Q5_1 512 pp2048 3240.92 3213.73 0.99
Radeon 8060S Graphics granitemoe 3B Q5_1 1024 pp2048 3502.80 3498.26 1.00
Radeon 8060S Graphics granitemoe 3B Q5_1 2048 pp2048 3561.91 3554.38 1.00
Radeon 8060S Graphics granitemoe 3B Q5_K_S 16 pp2048 693.10 695.91 1.00
Radeon 8060S Graphics granitemoe 3B Q5_K_S 32 pp2048 745.37 746.86 1.00
Radeon 8060S Graphics granitemoe 3B Q5_K_S 64 pp2048 567.06 562.96 0.99
Radeon 8060S Graphics granitemoe 3B Q5_K_S 128 pp2048 1764.74 1760.26 1.00
Radeon 8060S Graphics granitemoe 3B Q5_K_S 256 pp2048 2624.24 2621.41 1.00
Radeon 8060S Graphics granitemoe 3B Q5_K_S 512 pp2048 3204.79 3211.89 1.00
Radeon 8060S Graphics granitemoe 3B Q5_K_S 1024 pp2048 3520.40 3500.34 0.99
Radeon 8060S Graphics granitemoe 3B Q5_K_S 2048 pp2048 3558.21 3576.20 1.01
Radeon 8060S Graphics granitemoe 3B Q6_K 16 pp2048 609.46 609.13 1.00
Radeon 8060S Graphics granitemoe 3B Q6_K 32 pp2048 343.10 346.71 1.01
Radeon 8060S Graphics granitemoe 3B Q6_K 64 pp2048 1146.25 1145.95 1.00
Radeon 8060S Graphics granitemoe 3B Q6_K 128 pp2048 1399.78 1407.30 1.01
Radeon 8060S Graphics granitemoe 3B Q6_K 256 pp2048 2109.99 2110.44 1.00
Radeon 8060S Graphics granitemoe 3B Q6_K 512 pp2048 1415.02 1417.29 1.00
Radeon 8060S Graphics granitemoe 3B Q6_K 1024 pp2048 1819.19 1819.35 1.00
Radeon 8060S Graphics granitemoe 3B Q6_K 2048 pp2048 2096.15 2090.98 1.00
Radeon 8060S Graphics granitemoe 3B Q8_0 16 pp2048 610.42 609.50 1.00
Radeon 8060S Graphics granitemoe 3B Q8_0 32 pp2048 741.32 737.79 1.00
Radeon 8060S Graphics granitemoe 3B Q8_0 64 pp2048 962.54 968.20 1.01
Radeon 8060S Graphics granitemoe 3B Q8_0 128 pp2048 798.09 804.17 1.01
Radeon 8060S Graphics granitemoe 3B Q8_0 256 pp2048 1229.83 1241.02 1.01
Radeon 8060S Graphics granitemoe 3B Q8_0 512 pp2048 1626.22 1618.17 1.00
Radeon 8060S Graphics granitemoe 3B Q8_0 1024 pp2048 1898.30 1875.74 0.99
Radeon 8060S Graphics granitemoe 3B Q8_0 2048 pp2048 1998.42 2000.84 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 471.58 466.98 0.99
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 791.73 791.19 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 1059.23 1061.28 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 1765.28 1768.13 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 2540.39 2545.32 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 3047.93 3053.62 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 3371.30 3381.01 1.00
RX 6800 granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 3596.65 3600.03 1.00
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 378.89 380.81 1.01
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 698.26 701.77 1.01
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 990.68 992.62 1.00
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 1625.10 1629.49 1.00
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 2314.57 2321.58 1.00
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 2788.89 2797.34 1.00
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 3096.11 3101.89 1.00
RX 6800 granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 3275.51 3284.50 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 371.30 372.81 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 739.41 740.63 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 975.04 976.11 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 1595.87 1597.76 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 2279.59 2285.45 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 2757.33 2763.64 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 3073.61 3081.97 1.00
RX 6800 granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 3279.32 3289.69 1.00
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 458.42 384.05 0.84
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 497.73 488.46 0.98
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 1021.55 1025.36 1.00
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 1682.19 1689.34 1.00
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 2417.29 2429.09 1.00
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 2919.88 2932.05 1.00
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 3242.74 3258.05 1.00
RX 6800 granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 3454.78 3466.91 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 448.41 450.63 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 472.80 478.35 1.01
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 1070.15 1070.69 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 1750.45 1751.91 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 2474.22 2481.46 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 2938.66 2944.91 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 3263.20 3270.74 1.00
RX 6800 granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 3437.59 3442.84 1.00
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 454.59 456.91 1.01
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 505.19 510.85 1.01
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 999.99 1001.79 1.00
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 1646.66 1651.27 1.00
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 2380.63 2386.05 1.00
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 2860.75 2866.14 1.00
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 3172.08 3177.58 1.00
RX 6800 granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 3353.33 3361.99 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 447.19 446.37 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 456.50 462.99 1.01
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 1059.15 1058.57 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 1732.42 1732.03 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 2461.74 2464.65 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 2939.37 2941.12 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 3262.18 3268.29 1.00
RX 6800 granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 3437.05 3446.97 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 430.81 429.16 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 480.82 487.67 1.01
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 1056.22 1059.16 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 1730.58 1734.55 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 2453.35 2459.64 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 2923.10 2928.22 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 3241.62 3249.81 1.00
RX 6800 granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 3420.03 3427.54 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 454.30 455.49 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 407.63 409.86 1.01
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 1174.54 1177.68 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 1909.25 1914.31 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 2696.02 2701.47 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 3184.75 3196.02 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 3527.41 3543.09 1.00
RX 6800 granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 3725.59 3741.81 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 439.48 440.56 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 416.43 413.84 0.99
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 1172.43 1177.48 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 1910.80 1917.40 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 2692.69 2701.89 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 3181.54 3191.52 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 3523.48 3537.80 1.00
RX 6800 granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 3720.43 3736.73 1.00
RX 6800 granitemoe 3B Q2_K_M 16 pp2048 441.03 443.02 1.00
RX 6800 granitemoe 3B Q2_K_M 32 pp2048 631.81 633.77 1.00
RX 6800 granitemoe 3B Q2_K_M 64 pp2048 723.14 730.37 1.01
RX 6800 granitemoe 3B Q2_K_M 128 pp2048 1215.16 1226.80 1.01
RX 6800 granitemoe 3B Q2_K_M 256 pp2048 1826.00 1839.98 1.01
RX 6800 granitemoe 3B Q2_K_M 512 pp2048 2276.17 2288.59 1.01
RX 6800 granitemoe 3B Q2_K_M 1024 pp2048 2571.44 2592.29 1.01
RX 6800 granitemoe 3B Q2_K_M 2048 pp2048 2789.14 2804.64 1.01
RX 6800 granitemoe 3B Q3_K_S 16 pp2048 396.09 396.57 1.00
RX 6800 granitemoe 3B Q3_K_S 32 pp2048 734.45 736.87 1.00
RX 6800 granitemoe 3B Q3_K_S 64 pp2048 903.49 903.88 1.00
RX 6800 granitemoe 3B Q3_K_S 128 pp2048 1490.99 1492.21 1.00
RX 6800 granitemoe 3B Q3_K_S 256 pp2048 2131.14 2135.14 1.00
RX 6800 granitemoe 3B Q3_K_S 512 pp2048 2562.12 2566.22 1.00
RX 6800 granitemoe 3B Q3_K_S 1024 pp2048 2887.43 2893.69 1.00
RX 6800 granitemoe 3B Q3_K_S 2048 pp2048 3091.26 3100.49 1.00
RX 6800 granitemoe 3B Q4_0 16 pp2048 546.85 549.50 1.00
RX 6800 granitemoe 3B Q4_0 32 pp2048 886.39 888.33 1.00
RX 6800 granitemoe 3B Q4_0 64 pp2048 1185.93 1188.63 1.00
RX 6800 granitemoe 3B Q4_0 128 pp2048 1929.91 1936.24 1.00
RX 6800 granitemoe 3B Q4_0 256 pp2048 2718.04 2728.95 1.00
RX 6800 granitemoe 3B Q4_0 512 pp2048 3228.14 3241.52 1.00
RX 6800 granitemoe 3B Q4_0 1024 pp2048 3587.16 3602.12 1.00
RX 6800 granitemoe 3B Q4_0 2048 pp2048 3810.17 3819.92 1.00
RX 6800 granitemoe 3B Q4_1 16 pp2048 561.65 563.02 1.00
RX 6800 granitemoe 3B Q4_1 32 pp2048 876.84 879.64 1.00
RX 6800 granitemoe 3B Q4_1 64 pp2048 1152.72 1155.53 1.00
RX 6800 granitemoe 3B Q4_1 128 pp2048 1878.38 1884.10 1.00
RX 6800 granitemoe 3B Q4_1 256 pp2048 2653.12 2659.49 1.00
RX 6800 granitemoe 3B Q4_1 512 pp2048 3155.62 3165.43 1.00
RX 6800 granitemoe 3B Q4_1 1024 pp2048 3512.79 3527.30 1.00
RX 6800 granitemoe 3B Q4_1 2048 pp2048 3729.29 3744.64 1.00
RX 6800 granitemoe 3B Q4_K_S 16 pp2048 515.06 515.91 1.00
RX 6800 granitemoe 3B Q4_K_S 32 pp2048 774.80 776.49 1.00
RX 6800 granitemoe 3B Q4_K_S 64 pp2048 853.10 853.29 1.00
RX 6800 granitemoe 3B Q4_K_S 128 pp2048 1420.04 1420.79 1.00
RX 6800 granitemoe 3B Q4_K_S 256 pp2048 2086.01 2090.25 1.00
RX 6800 granitemoe 3B Q4_K_S 512 pp2048 2551.87 2554.10 1.00
RX 6800 granitemoe 3B Q4_K_S 1024 pp2048 2864.56 2868.15 1.00
RX 6800 granitemoe 3B Q4_K_S 2048 pp2048 3083.85 3090.97 1.00
RX 6800 granitemoe 3B Q5_0 16 pp2048 438.58 439.18 1.00
RX 6800 granitemoe 3B Q5_0 32 pp2048 403.24 403.01 1.00
RX 6800 granitemoe 3B Q5_0 64 pp2048 1105.52 1105.70 1.00
RX 6800 granitemoe 3B Q5_0 128 pp2048 1802.74 1804.12 1.00
RX 6800 granitemoe 3B Q5_0 256 pp2048 2551.14 2555.98 1.00
RX 6800 granitemoe 3B Q5_0 512 pp2048 3020.83 3028.94 1.00
RX 6800 granitemoe 3B Q5_0 1024 pp2048 3366.41 3375.38 1.00
RX 6800 granitemoe 3B Q5_0 2048 pp2048 3567.09 3575.05 1.00
RX 6800 granitemoe 3B Q5_1 16 pp2048 420.44 420.93 1.00
RX 6800 granitemoe 3B Q5_1 32 pp2048 848.45 847.53 1.00
RX 6800 granitemoe 3B Q5_1 64 pp2048 1129.22 1131.80 1.00
RX 6800 granitemoe 3B Q5_1 128 pp2048 1841.18 1845.37 1.00
RX 6800 granitemoe 3B Q5_1 256 pp2048 2603.80 2609.76 1.00
RX 6800 granitemoe 3B Q5_1 512 pp2048 3084.52 3084.67 1.00
RX 6800 granitemoe 3B Q5_1 1024 pp2048 3428.71 3440.81 1.00
RX 6800 granitemoe 3B Q5_1 2048 pp2048 3634.75 3637.65 1.00
RX 6800 granitemoe 3B Q5_K_S 16 pp2048 479.07 479.41 1.00
RX 6800 granitemoe 3B Q5_K_S 32 pp2048 772.22 772.65 1.00
RX 6800 granitemoe 3B Q5_K_S 64 pp2048 853.33 853.23 1.00
RX 6800 granitemoe 3B Q5_K_S 128 pp2048 1423.77 1422.81 1.00
RX 6800 granitemoe 3B Q5_K_S 256 pp2048 2090.84 2092.98 1.00
RX 6800 granitemoe 3B Q5_K_S 512 pp2048 2545.53 2551.20 1.00
RX 6800 granitemoe 3B Q5_K_S 1024 pp2048 2862.90 2866.97 1.00
RX 6800 granitemoe 3B Q5_K_S 2048 pp2048 3078.61 3087.76 1.00
RX 6800 granitemoe 3B Q6_K 16 pp2048 348.46 348.79 1.00
RX 6800 granitemoe 3B Q6_K 32 pp2048 718.85 719.93 1.00
RX 6800 granitemoe 3B Q6_K 64 pp2048 774.20 774.61 1.00
RX 6800 granitemoe 3B Q6_K 128 pp2048 1292.16 1293.52 1.00
RX 6800 granitemoe 3B Q6_K 256 pp2048 1893.68 1896.30 1.00
RX 6800 granitemoe 3B Q6_K 512 pp2048 2309.84 2312.64 1.00
RX 6800 granitemoe 3B Q6_K 1024 pp2048 2619.83 2623.07 1.00
RX 6800 granitemoe 3B Q6_K 2048 pp2048 2817.52 2819.87 1.00
RX 6800 granitemoe 3B Q8_0 16 pp2048 501.85 503.37 1.00
RX 6800 granitemoe 3B Q8_0 32 pp2048 559.85 571.31 1.02
RX 6800 granitemoe 3B Q8_0 64 pp2048 1134.60 1135.22 1.00
RX 6800 granitemoe 3B Q8_0 128 pp2048 1863.44 1865.50 1.00
RX 6800 granitemoe 3B Q8_0 256 pp2048 2655.07 2655.39 1.00
RX 6800 granitemoe 3B Q8_0 512 pp2048 3157.58 3163.68 1.00
RX 6800 granitemoe 3B Q8_0 1024 pp2048 3539.68 3550.41 1.00
RX 6800 granitemoe 3B Q8_0 2048 pp2048 3764.93 3771.18 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 16 pp2048 852.63 854.89 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 32 pp2048 1475.36 1479.26 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 64 pp2048 899.71 898.67 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 128 pp2048 2925.08 2912.11 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 256 pp2048 4678.15 4655.30 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 512 pp2048 6016.82 5996.22 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 1024 pp2048 6494.02 6480.33 1.00
RX 9060 XT granitemoe 3B IQ1_S - 1.5625 bpw 2048 pp2048 6283.86 6269.56 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 16 pp2048 700.44 701.38 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 32 pp2048 1076.53 1079.99 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 64 pp2048 987.99 984.62 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 128 pp2048 2614.31 2614.84 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 256 pp2048 4162.85 4147.66 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 512 pp2048 5363.83 5374.88 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 1024 pp2048 5870.02 5868.02 1.00
RX 9060 XT granitemoe 3B IQ2_S - 2.5 bpw 2048 pp2048 5733.67 5731.58 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 16 pp2048 671.71 674.37 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 32 pp2048 1043.62 1049.20 1.01
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 64 pp2048 948.39 952.09 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 128 pp2048 2857.58 2857.11 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 256 pp2048 4426.66 4431.46 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 512 pp2048 5855.64 5866.68 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 1024 pp2048 6327.91 6340.49 1.00
RX 9060 XT granitemoe 3B IQ2_XS - 2.3125 bpw 2048 pp2048 6204.06 6236.55 1.01
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 16 pp2048 793.78 795.22 1.00
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 32 pp2048 1318.47 1322.04 1.00
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 64 pp2048 1675.01 1680.51 1.00
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 128 pp2048 1448.56 1465.75 1.01
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 256 pp2048 2363.57 2389.00 1.01
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 512 pp2048 3205.99 3230.99 1.01
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 1024 pp2048 3761.69 3781.93 1.01
RX 9060 XT granitemoe 3B IQ2_XXS - 2.0625 bpw 2048 pp2048 3899.06 3916.21 1.00
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 16 pp2048 789.84 792.57 1.00
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 32 pp2048 1278.48 1284.54 1.00
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 64 pp2048 1645.24 1644.63 1.00
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 128 pp2048 1344.27 1360.25 1.01
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 256 pp2048 2180.15 2209.43 1.01
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 512 pp2048 2964.23 3004.69 1.01
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 1024 pp2048 3525.79 3570.03 1.01
RX 9060 XT granitemoe 3B IQ3_S - 3.4375 bpw 2048 pp2048 3658.22 3699.96 1.01
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 16 pp2048 803.56 806.84 1.00
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 32 pp2048 1306.92 1316.29 1.01
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 64 pp2048 1656.88 1662.81 1.00
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 128 pp2048 1449.48 1472.13 1.02
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 256 pp2048 2328.87 2359.74 1.01
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 512 pp2048 3179.59 3229.90 1.02
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 1024 pp2048 3734.17 3784.30 1.01
RX 9060 XT granitemoe 3B IQ3_S mix - 3.66 bpw 2048 pp2048 3889.34 3928.06 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 16 pp2048 783.93 789.06 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 32 pp2048 1279.24 1285.74 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 64 pp2048 1672.60 1672.06 1.00
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 128 pp2048 1361.13 1378.97 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 256 pp2048 2205.59 2232.85 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 512 pp2048 2992.58 3031.68 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 1024 pp2048 3553.99 3596.10 1.01
RX 9060 XT granitemoe 3B IQ3_XS - 3.3 bpw 2048 pp2048 3690.94 3720.54 1.01
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 16 pp2048 772.35 773.20 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 32 pp2048 1253.32 1253.83 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 64 pp2048 1553.61 1547.18 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 128 pp2048 1481.73 1480.44 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 256 pp2048 2364.68 2358.48 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 512 pp2048 3224.86 3227.30 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 1024 pp2048 3758.58 3761.42 1.00
RX 9060 XT granitemoe 3B IQ3_XXS - 3.0625 bpw 2048 pp2048 3884.58 3886.98 1.00
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 16 pp2048 893.06 896.74 1.00
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 32 pp2048 1582.12 1590.45 1.01
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 64 pp2048 1586.64 1591.13 1.00
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 128 pp2048 1323.49 1335.92 1.01
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 256 pp2048 2163.82 2178.31 1.01
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 512 pp2048 2974.91 2994.01 1.01
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 1024 pp2048 3586.76 3598.57 1.00
RX 9060 XT granitemoe 3B IQ4_NL - 4.5 bpw 2048 pp2048 3766.10 3780.39 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 16 pp2048 900.32 904.25 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 32 pp2048 1588.18 1599.50 1.01
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 64 pp2048 1615.42 1617.03 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 128 pp2048 1366.62 1370.16 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 256 pp2048 2221.34 2220.30 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 512 pp2048 3035.28 3038.60 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 1024 pp2048 3631.68 3634.82 1.00
RX 9060 XT granitemoe 3B IQ4_XS - 4.25 bpw 2048 pp2048 3811.70 3806.98 1.00
RX 9060 XT granitemoe 3B Q2_K_M 16 pp2048 710.91 701.61 0.99
RX 9060 XT granitemoe 3B Q2_K_M 32 pp2048 1021.42 1014.84 0.99
RX 9060 XT granitemoe 3B Q2_K_M 64 pp2048 1160.09 1159.91 1.00
RX 9060 XT granitemoe 3B Q2_K_M 128 pp2048 2605.87 2605.21 1.00
RX 9060 XT granitemoe 3B Q2_K_M 256 pp2048 3922.53 3925.19 1.00
RX 9060 XT granitemoe 3B Q2_K_M 512 pp2048 4984.24 4973.56 1.00
RX 9060 XT granitemoe 3B Q2_K_M 1024 pp2048 5398.56 5387.89 1.00
RX 9060 XT granitemoe 3B Q2_K_M 2048 pp2048 5297.30 5280.48 1.00
RX 9060 XT granitemoe 3B Q3_K_S 16 pp2048 782.42 791.38 1.01
RX 9060 XT granitemoe 3B Q3_K_S 32 pp2048 1076.63 1076.01 1.00
RX 9060 XT granitemoe 3B Q3_K_S 64 pp2048 901.53 898.65 1.00
RX 9060 XT granitemoe 3B Q3_K_S 128 pp2048 3029.52 3019.78 1.00
RX 9060 XT granitemoe 3B Q3_K_S 256 pp2048 4765.94 4743.21 1.00
RX 9060 XT granitemoe 3B Q3_K_S 512 pp2048 6250.78 6230.26 1.00
RX 9060 XT granitemoe 3B Q3_K_S 1024 pp2048 6741.61 6721.14 1.00
RX 9060 XT granitemoe 3B Q3_K_S 2048 pp2048 6572.51 6540.01 1.00
RX 9060 XT granitemoe 3B Q4_0 16 pp2048 874.71 876.16 1.00
RX 9060 XT granitemoe 3B Q4_0 32 pp2048 1535.87 1543.40 1.00
RX 9060 XT granitemoe 3B Q4_0 64 pp2048 1520.31 1516.82 1.00
RX 9060 XT granitemoe 3B Q4_0 128 pp2048 1350.99 1341.95 0.99
RX 9060 XT granitemoe 3B Q4_0 256 pp2048 2197.00 2182.45 0.99
RX 9060 XT granitemoe 3B Q4_0 512 pp2048 3016.73 3003.33 1.00
RX 9060 XT granitemoe 3B Q4_0 1024 pp2048 3639.03 3635.17 1.00
RX 9060 XT granitemoe 3B Q4_0 2048 pp2048 3822.36 3818.86 1.00
RX 9060 XT granitemoe 3B Q4_1 16 pp2048 879.44 881.12 1.00
RX 9060 XT granitemoe 3B Q4_1 32 pp2048 1557.50 1563.38 1.00
RX 9060 XT granitemoe 3B Q4_1 64 pp2048 865.73 840.36 0.97
RX 9060 XT granitemoe 3B Q4_1 128 pp2048 3171.16 3170.38 1.00
RX 9060 XT granitemoe 3B Q4_1 256 pp2048 4929.99 4922.77 1.00
RX 9060 XT granitemoe 3B Q4_1 512 pp2048 6403.15 6396.97 1.00
RX 9060 XT granitemoe 3B Q4_1 1024 pp2048 6935.28 6933.58 1.00
RX 9060 XT granitemoe 3B Q4_1 2048 pp2048 6738.48 6723.30 1.00
RX 9060 XT granitemoe 3B Q4_K_S 16 pp2048 871.27 876.07 1.01
RX 9060 XT granitemoe 3B Q4_K_S 32 pp2048 1491.01 1505.89 1.01
RX 9060 XT granitemoe 3B Q4_K_S 64 pp2048 1772.51 1778.02 1.00
RX 9060 XT granitemoe 3B Q4_K_S 128 pp2048 3225.71 3236.68 1.00
RX 9060 XT granitemoe 3B Q4_K_S 256 pp2048 5025.77 5029.11 1.00
RX 9060 XT granitemoe 3B Q4_K_S 512 pp2048 6536.56 6551.24 1.00
RX 9060 XT granitemoe 3B Q4_K_S 1024 pp2048 7034.09 7053.94 1.00
RX 9060 XT granitemoe 3B Q4_K_S 2048 pp2048 6821.24 6827.91 1.00
RX 9060 XT granitemoe 3B Q5_0 16 pp2048 816.37 819.51 1.00
RX 9060 XT granitemoe 3B Q5_0 32 pp2048 1421.13 1431.73 1.01
RX 9060 XT granitemoe 3B Q5_0 64 pp2048 1431.54 1437.35 1.00
RX 9060 XT granitemoe 3B Q5_0 128 pp2048 1246.34 1251.79 1.00
RX 9060 XT granitemoe 3B Q5_0 256 pp2048 2029.77 2045.07 1.01
RX 9060 XT granitemoe 3B Q5_0 512 pp2048 2812.00 2827.11 1.01
RX 9060 XT granitemoe 3B Q5_0 1024 pp2048 3433.81 3444.99 1.00
RX 9060 XT granitemoe 3B Q5_0 2048 pp2048 3628.42 3642.47 1.00
RX 9060 XT granitemoe 3B Q5_1 16 pp2048 839.82 842.91 1.00
RX 9060 XT granitemoe 3B Q5_1 32 pp2048 1482.11 1484.40 1.00
RX 9060 XT granitemoe 3B Q5_1 64 pp2048 757.20 757.64 1.00
RX 9060 XT granitemoe 3B Q5_1 128 pp2048 3075.83 3083.53 1.00
RX 9060 XT granitemoe 3B Q5_1 256 pp2048 4778.97 4790.05 1.00
RX 9060 XT granitemoe 3B Q5_1 512 pp2048 6219.64 6226.34 1.00
RX 9060 XT granitemoe 3B Q5_1 1024 pp2048 6748.57 6760.99 1.00
RX 9060 XT granitemoe 3B Q5_1 2048 pp2048 6557.97 6569.34 1.00
RX 9060 XT granitemoe 3B Q5_K_S 16 pp2048 848.92 853.54 1.01
RX 9060 XT granitemoe 3B Q5_K_S 32 pp2048 1053.86 1058.35 1.00
RX 9060 XT granitemoe 3B Q5_K_S 64 pp2048 821.58 855.37 1.04
RX 9060 XT granitemoe 3B Q5_K_S 128 pp2048 3114.10 3120.32 1.00
RX 9060 XT granitemoe 3B Q5_K_S 256 pp2048 4853.58 4846.80 1.00
RX 9060 XT granitemoe 3B Q5_K_S 512 pp2048 6314.42 6330.61 1.00
RX 9060 XT granitemoe 3B Q5_K_S 1024 pp2048 6880.47 6876.25 1.00
RX 9060 XT granitemoe 3B Q5_K_S 2048 pp2048 6695.82 6689.43 1.00
RX 9060 XT granitemoe 3B Q6_K 16 pp2048 741.40 743.36 1.00
RX 9060 XT granitemoe 3B Q6_K 32 pp2048 680.38 680.92 1.00
RX 9060 XT granitemoe 3B Q6_K 64 pp2048 1663.13 1653.41 0.99
RX 9060 XT granitemoe 3B Q6_K 128 pp2048 2290.26 2292.35 1.00
RX 9060 XT granitemoe 3B Q6_K 256 pp2048 3579.20 3580.82 1.00
RX 9060 XT granitemoe 3B Q6_K 512 pp2048 4811.52 4810.91 1.00
RX 9060 XT granitemoe 3B Q6_K 1024 pp2048 5356.37 5353.89 1.00
RX 9060 XT granitemoe 3B Q6_K 2048 pp2048 5374.06 5371.13 1.00
RX 9060 XT granitemoe 3B Q8_0 16 pp2048 774.25 775.22 1.00
RX 9060 XT granitemoe 3B Q8_0 32 pp2048 1368.20 1368.47 1.00
RX 9060 XT granitemoe 3B Q8_0 64 pp2048 1483.93 1480.17 1.00
RX 9060 XT granitemoe 3B Q8_0 128 pp2048 1270.43 1267.72 1.00
RX 9060 XT granitemoe 3B Q8_0 256 pp2048 2088.16 2077.09 0.99
RX 9060 XT granitemoe 3B Q8_0 512 pp2048 2887.19 2884.05 1.00
RX 9060 XT granitemoe 3B Q8_0 1024 pp2048 3532.12 3525.19 1.00
RX 9060 XT granitemoe 3B Q8_0 2048 pp2048 3755.49 3747.30 1.00

@IMbackK
Copy link
Copy Markdown
Collaborator

IMbackK commented Apr 23, 2026

Besides MI100 this path is disabled on all of those anyhow. The question would be if with these improvements its now also faster outside of cdna

@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 23, 2026

Besides MI100 this path is disabled on all of those anyhow. The question would be if with these improvements its now also faster outside of cdna

Though the impact should be negligible, there is one tiny part that is affected by the fastdiv changes so I figured I should run the benchmarks just in case. I re-tested stream-k on the GPUs where it is currently disabled and found that it performs poorly. For the MI100 large grids also performed worse for MoE models.

@IMbackK
Copy link
Copy Markdown
Collaborator

IMbackK commented Apr 23, 2026

good, thanks for checking!

@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Apr 23, 2026
ORippler
ORippler reviewed Apr 24, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh
kbc -= (kbc % blocks_per_ne00) % blocks_per_iter;
kbc_stop -= (kbc_stop % blocks_per_ne00) % blocks_per_iter;
kbc -= fastmodulo(kbc, blocks_per_ne00) % blocks_per_iter;
kbc_stop -= fastmodulo(kbc_stop, blocks_per_ne00) % blocks_per_iter;
Copy link
Copy Markdown
Collaborator

@ORippler ORippler Apr 24, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Afaik fastmodulo/fastdiv are currently only implemented for uint32_t data-types. Does this mean kbc & kbc_stop as the output of their respecitve 64-bit signed int calculations can actually be fully represented in this without wrap-around? If so, let's cast them to uint32_t to make this clear.
Sadly, if the respective expressions for computing kbc and kbc_stop require 64_bit precision, we will still have register spills on NVGPUs due to how 64 bit int division is handled here currently (can elaborate if needed, I can also add 64-bit uint fastdiv/fastmodulo for uint64_t, that was fairly easy compared to 64-bit int)

Copy link
Copy Markdown
Contributor

@am17an am17an Apr 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can pass a template parameter when we know we can use 32 bit fastmodulo vs when we can't?

ORippler
ORippler reviewed Apr 24, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ORippler ORippler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would love some more context so I can gain clarity for the required precision, types & conversions regarding kbc & kbc_stop, as I hit the register spill caused by 64-bit kbc & kbc_stop and this code touches upon it (it's still present, and removing the spill will bring a bit of perf):

Image
Also, some more numbers on NVGPUS

build: 5f1074e (8871)

  • ./scripts/compare-llama-bench.py -b 82209ef -c cuda-mmq-fastdiv-8 --tool llama-bench -i llama-bench.sqlite
Model Microbatch size Test t/s 82209ef t/s cuda-mmq-fastdiv-8 Speedup
gemma4 26B.A4B Q4_K_M 16 pp2048 914.79 1044.69 1.14
gemma4 26B.A4B Q4_K_M 16 pp2048@d32768 878.58 992.74 1.13
gemma4 26B.A4B Q4_K_M 32 pp2048 1401.23 1567.44 1.12
gemma4 26B.A4B Q4_K_M 32 pp2048@d32768 1323.25 1472.42 1.11
gemma4 26B.A4B Q4_K_M 64 pp2048 1952.74 2202.41 1.13
gemma4 26B.A4B Q4_K_M 64 pp2048@d32768 1791.50 2000.43 1.12
gemma4 26B.A4B Q4_K_M 128 pp2048 2332.70 2597.84 1.11
gemma4 26B.A4B Q4_K_M 128 pp2048@d32768 1974.79 2174.40 1.10
gemma4 26B.A4B Q4_K_M 256 pp2048 3721.95 4051.54 1.09
gemma4 26B.A4B Q4_K_M 256 pp2048@d32768 2730.86 2930.24 1.07
gemma4 26B.A4B Q4_K_M 512 pp2048 5274.42 5724.11 1.09
gemma4 26B.A4B Q4_K_M 512 pp2048@d32768 3442.34 3666.19 1.07
gemma4 26B.A4B Q4_K_M 1024 pp2048 6510.57 7088.02 1.09
gemma4 26B.A4B Q4_K_M 1024 pp2048@d32768 4060.76 4310.83 1.06
gemma4 26B.A4B Q4_K_M 2048 pp2048 7040.79 7629.48 1.08
gemma4 26B.A4B Q4_K_M 2048 pp2048@d32768 4170.70 4394.05 1.05
gemma4 31B Q4_K_M 16 pp2048 477.72 480.22 1.01
gemma4 31B Q4_K_M 16 pp2048@d32768 409.16 411.12 1.00
gemma4 31B Q4_K_M 32 pp2048 791.78 795.84 1.01
gemma4 31B Q4_K_M 32 pp2048@d32768 633.03 634.10 1.00
gemma4 31B Q4_K_M 64 pp2048 1156.28 1165.77 1.01
gemma4 31B Q4_K_M 64 pp2048@d32768 774.43 775.35 1.00
gemma4 31B Q4_K_M 128 pp2048 1398.67 1386.47 0.99
gemma4 31B Q4_K_M 128 pp2048@d32768 872.87 870.31 1.00
gemma4 31B Q4_K_M 256 pp2048 1537.61 1531.11 1.00
gemma4 31B Q4_K_M 256 pp2048@d32768 932.54 929.32 1.00
gemma4 31B Q4_K_M 512 pp2048 1612.27 1599.30 0.99
gemma4 31B Q4_K_M 512 pp2048@d32768 974.72 969.81 0.99
gemma4 31B Q4_K_M 1024 pp2048 1632.93 1630.52 1.00
gemma4 31B Q4_K_M 1024 pp2048@d32768 992.45 991.43 1.00
gemma4 31B Q4_K_M 2048 pp2048 1585.85 1621.86 1.02
gemma4 31B Q4_K_M 2048 pp2048@d32768 981.80 994.26 1.01
gemma4 E4B Q4_K_M 16 pp2048 1985.43 2080.39 1.05
gemma4 E4B Q4_K_M 16 pp2048@d32768 1703.79 1774.21 1.04
gemma4 E4B Q4_K_M 32 pp2048 3154.90 3468.88 1.10
gemma4 E4B Q4_K_M 32 pp2048@d32768 2754.09 2795.73 1.02
gemma4 E4B Q4_K_M 64 pp2048 4687.11 5024.77 1.07
gemma4 E4B Q4_K_M 64 pp2048@d32768 3789.76 3894.56 1.03
gemma4 E4B Q4_K_M 128 pp2048 6135.13 6489.89 1.06
gemma4 E4B Q4_K_M 128 pp2048@d32768 4581.40 4675.19 1.02
gemma4 E4B Q4_K_M 256 pp2048 8239.29 8470.00 1.03
gemma4 E4B Q4_K_M 256 pp2048@d32768 5023.51 5121.48 1.02
gemma4 E4B Q4_K_M 512 pp2048 9361.88 9703.76 1.04
gemma4 E4B Q4_K_M 512 pp2048@d32768 5424.37 5513.46 1.02
gemma4 E4B Q4_K_M 1024 pp2048 9788.44 9893.58 1.01
gemma4 E4B Q4_K_M 1024 pp2048@d32768 5907.11 5935.78 1.00
gemma4 E4B Q4_K_M 2048 pp2048 9257.66 9284.09 1.00
gemma4 E4B Q4_K_M 2048 pp2048@d32768 5538.49 5538.96 1.00
gpt-oss 20B MXFP4 MoE 16 pp2048 1266.06 1521.89 1.20
gpt-oss 20B MXFP4 MoE 16 pp2048@d32768 1153.43 1353.75 1.17
gpt-oss 20B MXFP4 MoE 32 pp2048 2029.82 2448.66 1.21
gpt-oss 20B MXFP4 MoE 32 pp2048@d32768 1852.03 2091.67 1.13
gpt-oss 20B MXFP4 MoE 64 pp2048 3208.61 3706.54 1.16
gpt-oss 20B MXFP4 MoE 64 pp2048@d32768 2539.52 2883.99 1.14
gpt-oss 20B MXFP4 MoE 128 pp2048 4327.98 5168.11 1.19
gpt-oss 20B MXFP4 MoE 128 pp2048@d32768 3313.67 3778.38 1.14
gpt-oss 20B MXFP4 MoE 256 pp2048 6651.97 7946.76 1.19
gpt-oss 20B MXFP4 MoE 256 pp2048@d32768 4444.43 4982.64 1.12
gpt-oss 20B MXFP4 MoE 512 pp2048 8905.91 10501.62 1.18
gpt-oss 20B MXFP4 MoE 512 pp2048@d32768 5507.18 6091.97 1.11
gpt-oss 20B MXFP4 MoE 1024 pp2048 10225.28 12091.25 1.18
gpt-oss 20B MXFP4 MoE 1024 pp2048@d32768 5945.34 6543.82 1.10
gpt-oss 20B MXFP4 MoE 2048 pp2048 9961.50 11529.04 1.16
gpt-oss 20B MXFP4 MoE 2048 pp2048@d32768 5682.62 6187.17 1.09
nemotron_h_moe 31B.A3.5B NVFP4 16 pp2048 791.58 923.37 1.17
nemotron_h_moe 31B.A3.5B NVFP4 16 pp2048@d32768 774.97 905.00 1.17
nemotron_h_moe 31B.A3.5B NVFP4 32 pp2048 1048.88 1201.36 1.15
nemotron_h_moe 31B.A3.5B NVFP4 32 pp2048@d32768 1030.68 1175.21 1.14
nemotron_h_moe 31B.A3.5B NVFP4 64 pp2048 1322.33 1473.22 1.11
nemotron_h_moe 31B.A3.5B NVFP4 64 pp2048@d32768 1301.95 1453.62 1.12
nemotron_h_moe 31B.A3.5B NVFP4 128 pp2048 1497.22 1628.06 1.09
nemotron_h_moe 31B.A3.5B NVFP4 128 pp2048@d32768 1478.76 1608.97 1.09
nemotron_h_moe 31B.A3.5B NVFP4 256 pp2048 2462.64 2613.81 1.06
nemotron_h_moe 31B.A3.5B NVFP4 256 pp2048@d32768 2343.09 2494.88 1.06
nemotron_h_moe 31B.A3.5B NVFP4 512 pp2048 3765.66 3986.95 1.06
nemotron_h_moe 31B.A3.5B NVFP4 512 pp2048@d32768 3437.19 3623.00 1.05
nemotron_h_moe 31B.A3.5B NVFP4 1024 pp2048 5083.19 5357.53 1.05
nemotron_h_moe 31B.A3.5B NVFP4 1024 pp2048@d32768 4373.03 4588.62 1.05
nemotron_h_moe 31B.A3.5B NVFP4 2048 pp2048 5732.08 6068.46 1.06
nemotron_h_moe 31B.A3.5B NVFP4 2048 pp2048@d32768 4688.64 4946.34 1.05
nemotron_h_moe 31B.A3.5B Q4_K_M 16 pp2048 766.28 891.92 1.16
nemotron_h_moe 31B.A3.5B Q4_K_M 16 pp2048@d32768 737.00 849.94 1.15
nemotron_h_moe 31B.A3.5B Q4_K_M 32 pp2048 1112.57 1236.99 1.11
nemotron_h_moe 31B.A3.5B Q4_K_M 32 pp2048@d32768 1062.44 1177.25 1.11
nemotron_h_moe 31B.A3.5B Q4_K_M 64 pp2048 1491.11 1634.99 1.10
nemotron_h_moe 31B.A3.5B Q4_K_M 64 pp2048@d32768 1414.13 1534.08 1.08
nemotron_h_moe 31B.A3.5B Q4_K_M 128 pp2048 1810.81 1934.42 1.07
nemotron_h_moe 31B.A3.5B Q4_K_M 128 pp2048@d32768 1702.64 1807.84 1.06
nemotron_h_moe 31B.A3.5B Q4_K_M 256 pp2048 2956.06 3098.15 1.05
nemotron_h_moe 31B.A3.5B Q4_K_M 256 pp2048@d32768 2719.20 2833.47 1.04
nemotron_h_moe 31B.A3.5B Q4_K_M 512 pp2048 4481.30 4659.46 1.04
nemotron_h_moe 31B.A3.5B Q4_K_M 512 pp2048@d32768 3981.66 4105.71 1.03
nemotron_h_moe 31B.A3.5B Q4_K_M 1024 pp2048 5971.13 6208.51 1.04
nemotron_h_moe 31B.A3.5B Q4_K_M 1024 pp2048@d32768 5032.85 5188.73 1.03
nemotron_h_moe 31B.A3.5B Q4_K_M 2048 pp2048 6783.78 7013.52 1.03
nemotron_h_moe 31B.A3.5B Q4_K_M 2048 pp2048@d32768 5402.65 5550.75 1.03
qwen35 27B Q4_K_M 16 pp2048 512.54 515.14 1.01
qwen35 27B Q4_K_M 16 pp2048@d32768 461.18 466.58 1.01
qwen35 27B Q4_K_M 32 pp2048 836.91 850.53 1.02
qwen35 27B Q4_K_M 32 pp2048@d32768 734.27 742.34 1.01
qwen35 27B Q4_K_M 64 pp2048 1214.65 1213.17 1.00
qwen35 27B Q4_K_M 64 pp2048@d32768 1006.92 1015.35 1.01
qwen35 27B Q4_K_M 128 pp2048 1471.08 1472.33 1.00
qwen35 27B Q4_K_M 128 pp2048@d32768 1022.19 1025.24 1.00
qwen35 27B Q4_K_M 256 pp2048 1633.20 1646.32 1.01
qwen35 27B Q4_K_M 256 pp2048@d32768 1124.95 1134.36 1.01
qwen35 27B Q4_K_M 512 pp2048 1712.19 1730.45 1.01
qwen35 27B Q4_K_M 512 pp2048@d32768 1223.96 1234.40 1.01
qwen35 27B Q4_K_M 1024 pp2048 1724.11 1735.87 1.01
qwen35 27B Q4_K_M 1024 pp2048@d32768 1287.23 1298.57 1.01
qwen35 27B Q4_K_M 2048 pp2048 1694.41 1742.45 1.03
qwen35 27B Q4_K_M 2048 pp2048@d32768 1266.60 1300.48 1.03
qwen35moe 35B.A3B Q4_K_M 16 pp2048 751.25 883.56 1.18
qwen35moe 35B.A3B Q4_K_M 16 pp2048@d32768 704.92 826.94 1.17
qwen35moe 35B.A3B Q4_K_M 32 pp2048 1164.01 1382.06 1.19
qwen35moe 35B.A3B Q4_K_M 32 pp2048@d32768 1103.32 1282.73 1.16
qwen35moe 35B.A3B Q4_K_M 64 pp2048 1624.36 1898.11 1.17
qwen35moe 35B.A3B Q4_K_M 64 pp2048@d32768 1483.76 1743.78 1.18
qwen35moe 35B.A3B Q4_K_M 128 pp2048 1923.22 2262.29 1.18
qwen35moe 35B.A3B Q4_K_M 128 pp2048@d32768 1726.89 1994.00 1.15
qwen35moe 35B.A3B Q4_K_M 256 pp2048 2912.44 3262.79 1.12
qwen35moe 35B.A3B Q4_K_M 256 pp2048@d32768 2501.22 2758.12 1.10
qwen35moe 35B.A3B Q4_K_M 512 pp2048 4109.02 4521.77 1.10
qwen35moe 35B.A3B Q4_K_M 512 pp2048@d32768 3352.94 3625.34 1.08
qwen35moe 35B.A3B Q4_K_M 1024 pp2048 5001.70 5532.55 1.11
qwen35moe 35B.A3B Q4_K_M 1024 pp2048@d32768 3928.51 4255.15 1.08
qwen35moe 35B.A3B Q4_K_M 2048 pp2048 5495.72 6036.81 1.10
qwen35moe 35B.A3B Q4_K_M 2048 pp2048@d32768 4163.90 4455.37 1.07

Comment thread ggml/src/ggml-cuda/mmq.cuh

kb0_start = 0;
kb0_stop = min(blocks_per_ne00, kbc_stop - kbc);
kb0_stop = min(blocks_per_ne00.z, uint32_t(kbc_stop - kbc));
Copy link
Copy Markdown
Collaborator

@ORippler ORippler Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cf. above w.r.t wrap-around during int -> uint32_t conversions

@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 24, 2026

@ORippler during development I had experimented with a version where kbc and kbc_stop were being calculated using 32 bit arithmetic. This was a bit faster but problematic in terms of the numerical range which is why I reverted it again. However, I think I did not do this consistently.

@ikawrakow ikawrakow mentioned this pull request Apr 24, 2026
Merged
@nisparks
Copy link
Copy Markdown
Contributor

nisparks commented Apr 24, 2026

Don't have a lot to add, but if you wanted for reference a fastdiv64, had agent write one up:
nisparks@f973e33

@JohannesGaessler
Copy link
Copy Markdown
Contributor Author

JohannesGaessler commented Apr 25, 2026

The maximum value of a 32 bit signed integer is ~2 billion, the minimum ggml block size we have is 32. So in terms of the continuous k space that we are trying to represent we can go up to at least 68.7 billion. I think that for typical tensor sizes we should be well below this limit. So I've just changed the data type of kbc to a 32 bit signed integer with an assert in host code to make sure there aren't any unexpected overflows. For the calculation of kbc it is still necessary to do a 64 bit multiplication and division.

Performance changes
GPU Model Microbatch size Test t/s 5f1074e t/s 14293fb Speedup
RTX 3090 gemma4 26B.A4B Q4_K_M 16 pp2048 911.67 889.45 0.98
RTX 3090 gemma4 26B.A4B Q4_K_M 32 pp2048 1263.76 1245.73 0.99
RTX 3090 gemma4 26B.A4B Q4_K_M 64 pp2048 1606.39 1564.59 0.97
RTX 3090 gemma4 26B.A4B Q4_K_M 128 pp2048 1768.79 1727.16 0.98
RTX 3090 gemma4 26B.A4B Q4_K_M 256 pp2048 2552.34 2496.96 0.98
RTX 3090 gemma4 26B.A4B Q4_K_M 512 pp2048 3487.80 3488.13 1.00
RTX 3090 gemma4 26B.A4B Q4_K_M 1024 pp2048 4438.32 4387.83 0.99
RTX 3090 gemma4 26B.A4B Q4_K_M 2048 pp2048 5212.41 5172.07 0.99
RTX 3090 gpt-oss 20B MXFP4 MoE 16 pp2048 1095.24 1098.97 1.00
RTX 3090 gpt-oss 20B MXFP4 MoE 32 pp2048 1576.91 1582.25 1.00
RTX 3090 gpt-oss 20B MXFP4 MoE 64 pp2048 2089.84 2088.30 1.00
RTX 3090 gpt-oss 20B MXFP4 MoE 128 pp2048 2412.66 2445.75 1.01
RTX 3090 gpt-oss 20B MXFP4 MoE 256 pp2048 3623.96 3635.62 1.00
RTX 3090 gpt-oss 20B MXFP4 MoE 512 pp2048 4946.12 5045.98 1.02
RTX 3090 gpt-oss 20B MXFP4 MoE 1024 pp2048 6082.48 6134.86 1.01
RTX 3090 gpt-oss 20B MXFP4 MoE 2048 pp2048 6695.18 6835.51 1.02
RTX 3090 llama 8B Q4_0 16 pp2048 1346.29 1359.88 1.01
RTX 3090 llama 8B Q4_0 32 pp2048 2093.43 2106.39 1.01
RTX 3090 llama 8B Q4_0 64 pp2048 3020.09 3030.42 1.00
RTX 3090 llama 8B Q4_0 128 pp2048 3560.07 3614.67 1.02
RTX 3090 llama 8B Q4_0 256 pp2048 3966.92 4018.91 1.01
RTX 3090 llama 8B Q4_0 512 pp2048 4317.66 4376.15 1.01
RTX 3090 llama 8B Q4_0 1024 pp2048 4561.14 4563.40 1.00
RTX 3090 llama 8B Q4_0 2048 pp2048 4485.57 4555.84 1.02
RTX 3090 qwen35moe 35B.A3B Q4_0 16 pp2048 801.80 809.11 1.01
RTX 3090 qwen35moe 35B.A3B Q4_0 32 pp2048 1141.07 1138.92 1.00
RTX 3090 qwen35moe 35B.A3B Q4_0 64 pp2048 1392.01 1368.42 0.98
RTX 3090 qwen35moe 35B.A3B Q4_0 128 pp2048 1533.77 1520.60 0.99
RTX 3090 qwen35moe 35B.A3B Q4_0 256 pp2048 2100.16 2091.68 1.00
RTX 3090 qwen35moe 35B.A3B Q4_0 512 pp2048 2888.45 2757.84 0.95
RTX 3090 qwen35moe 35B.A3B Q4_0 1024 pp2048 3525.09 3466.65 0.98
RTX 3090 qwen35moe 35B.A3B Q4_0 2048 pp2048 4272.82 4210.99 0.99
RTX 4090 gemma4 26B.A4B Q4_K_M 16 pp2048 1188.63 1188.28 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 32 pp2048 2026.77 2034.20 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 64 pp2048 3025.67 3033.88 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 128 pp2048 3790.76 3787.95 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 256 pp2048 5562.35 5545.78 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 512 pp2048 8202.06 8170.95 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 1024 pp2048 9883.31 9913.95 1.00
RTX 4090 gemma4 26B.A4B Q4_K_M 2048 pp2048 11224.74 11257.98 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 16 pp2048 1718.25 1718.71 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 32 pp2048 2861.08 2859.05 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 64 pp2048 4222.59 4205.38 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 128 pp2048 5269.95 5271.64 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 256 pp2048 7925.13 7944.50 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 512 pp2048 11318.27 11326.73 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 1024 pp2048 13377.21 13414.26 1.00
RTX 4090 gpt-oss 20B MXFP4 MoE 2048 pp2048 13668.38 13679.98 1.00
RTX 4090 llama 8B Q4_0 16 pp2048 1998.63 2013.72 1.01
RTX 4090 llama 8B Q4_0 32 pp2048 3528.79 3547.03 1.01
RTX 4090 llama 8B Q4_0 64 pp2048 5930.41 5923.30 1.00
RTX 4090 llama 8B Q4_0 128 pp2048 7632.76 7724.48 1.01
RTX 4090 llama 8B Q4_0 256 pp2048 9675.69 9684.05 1.00
RTX 4090 llama 8B Q4_0 512 pp2048 11326.07 11364.63 1.00
RTX 4090 llama 8B Q4_0 1024 pp2048 11593.94 11677.75 1.01
RTX 4090 llama 8B Q4_0 2048 pp2048 12010.55 12100.78 1.01
RTX 4090 qwen35moe 35B.A3B Q4_0 16 pp2048 1139.80 1143.64 1.00
RTX 4090 qwen35moe 35B.A3B Q4_0 32 pp2048 1810.42 1819.42 1.00
RTX 4090 qwen35moe 35B.A3B Q4_0 64 pp2048 2608.43 2627.63 1.01
RTX 4090 qwen35moe 35B.A3B Q4_0 128 pp2048 3081.89 3110.14 1.01
RTX 4090 qwen35moe 35B.A3B Q4_0 256 pp2048 4277.06 4333.59 1.01
RTX 4090 qwen35moe 35B.A3B Q4_0 512 pp2048 6308.46 6382.83 1.01
RTX 4090 qwen35moe 35B.A3B Q4_0 1024 pp2048 7853.61 7931.59 1.01
RTX 4090 qwen35moe 35B.A3B Q4_0 2048 pp2048 10005.10 10151.11 1.01
RTX 5090 gemma4 26B.A4B Q4_K_M 16 pp2048 1474.98 1478.94 1.00
RTX 5090 gemma4 26B.A4B Q4_K_M 32 pp2048 2263.02 2263.03 1.00
RTX 5090 gemma4 26B.A4B Q4_K_M 64 pp2048 3358.64 3351.58 1.00
RTX 5090 gemma4 26B.A4B Q4_K_M 128 pp2048 4174.77 4132.73 0.99
RTX 5090 gemma4 26B.A4B Q4_K_M 256 pp2048 6196.37 6267.39 1.01
RTX 5090 gemma4 26B.A4B Q4_K_M 512 pp2048 9342.22 9469.94 1.01
RTX 5090 gemma4 26B.A4B Q4_K_M 1024 pp2048 11931.60 12052.15 1.01
RTX 5090 gemma4 26B.A4B Q4_K_M 2048 pp2048 14110.02 14234.02 1.01
RTX 5090 gpt-oss 20B MXFP4 MoE 16 pp2048 1844.19 1850.13 1.00
RTX 5090 gpt-oss 20B MXFP4 MoE 32 pp2048 3002.57 3000.23 1.00
RTX 5090 gpt-oss 20B MXFP4 MoE 64 pp2048 4728.89 4732.39 1.00
RTX 5090 gpt-oss 20B MXFP4 MoE 128 pp2048 6408.69 6471.24 1.01
RTX 5090 gpt-oss 20B MXFP4 MoE 256 pp2048 11193.05 11186.13 1.00
RTX 5090 gpt-oss 20B MXFP4 MoE 512 pp2048 16129.68 16186.21 1.00
RTX 5090 gpt-oss 20B MXFP4 MoE 1024 pp2048 19234.95 19312.83 1.00
RTX 5090 gpt-oss 20B MXFP4 MoE 2048 pp2048 20653.65 20798.97 1.01
RTX 5090 llama 8B Q4_0 16 pp2048 2525.81 2521.35 1.00
RTX 5090 llama 8B Q4_0 32 pp2048 4155.26 4139.37 1.00
RTX 5090 llama 8B Q4_0 64 pp2048 6414.40 6404.78 1.00
RTX 5090 llama 8B Q4_0 128 pp2048 7515.39 7508.14 1.00
RTX 5090 llama 8B Q4_0 256 pp2048 10734.74 10688.95 1.00
RTX 5090 llama 8B Q4_0 1024 pp2048 14922.79 14919.42 1.00
RTX 5090 llama 8B Q4_0 2048 pp2048 16085.48 16078.99 1.00
RTX 5090 qwen35moe 35B.A3B Q4_0 16 pp2048 1216.47 1228.26 1.01
RTX 5090 qwen35moe 35B.A3B Q4_0 32 pp2048 1849.66 1858.59 1.00
RTX 5090 qwen35moe 35B.A3B Q4_0 64 pp2048 2771.30 2764.67 1.00
RTX 5090 qwen35moe 35B.A3B Q4_0 128 pp2048 3352.96 3337.75 1.00
RTX 5090 qwen35moe 35B.A3B Q4_0 256 pp2048 4389.51 4360.46 0.99
RTX 5090 qwen35moe 35B.A3B Q4_0 512 pp2048 6946.79 6956.82 1.00
RTX 5090 qwen35moe 35B.A3B Q4_0 1024 pp2048 9178.02 9209.05 1.00
RTX 5090 qwen35moe 35B.A3B Q4_0 2048 pp2048 12938.85 12948.33 1.00

In a quick test I am seeing a slight performance impact from the change, I think only the measurements for LLaMA 3 8b are sufficiently precise to be meaningful.

@JohannesGaessler JohannesGaessler merged commit 9725a31 into ggml-org:master Apr 25, 2026
38 of 44 checks passed
nisparks pushed a commit to nisparks/llama.cpp that referenced this pull request Apr 26, 2026
@JohannesGaessler @nisparks
CUDA: reduce MMQ stream-k overhead (ggml-org#22298)
5160b43 
* CUDA: reduce MMQ stream-k overhead

* use 32 bit integers for kbc

(cherry picked from commit 9725a31)
justinappler added a commit to justinappler/llama.cpp-strix-halo that referenced this pull request Apr 27, 2026
@justinappler
strix-halo: 2026-04-27 — rocWMMA FA flag regression resolution
28cfde3 
The patched rocWMMA FA path (commit 030e290, lhl PR ggml-org#16827 port)
silently regressed at D=256 between 2026-04-19 (when rocwmma-tuned.md
benched it flat ±1.5% and concluded "kept anyway") and 2026-04-27.
Re-bench against the same source/host with only GGML_HIP_ROCWMMA_FATTN
toggled showed pp512@d=16k 244 → 853 t/s. server-configs flipped the
flag back OFF; source carried as #if-gated dead code, harmless when
off.

- rocwmma-tuned.md: status block at top + new "Re-bench 2026-04-27"
  section at the bottom with the systematic bisect (eliminated
  container ROCm version, llama.cpp upstream delta, source patches,
  host ROCm package set, modprobe.d state, KFD userptr eviction,
  memory pressure, GPU clocks, firmware versions) and candidate
  mechanisms for the post-landing regression.
- qwen3.6-baseline.md: rewrite the same-day regression note that
  initially blamed ROCm 7.13 nightly progression. Retract the
  amdgpu_amdkfd_restore_userptr_worker hypothesis (zero firings
  observed since boot).
- tg-at-depth-regression.md: mark resolved by upstream PR ggml-org#22298
  (CUDA: reduce MMQ stream-k overhead, merged 2026-04-26). Recovery
  numbers; original investigation kept below for posterity.
- README.md (root): findings table row 6 reflects the flag flip,
  watching-upstream gains a ggml-org#22298 row, and a re-bench checklist for
  future syncs/ROCm bumps so the next "flat at landing" doesn't go
  five weeks unvalidated.
- strix-halo/README.md: status annotations on the rocwmma-tuned and
  tg-at-depth-regression rows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ORippler ORippler ORippler left review comments

@am17an am17an am17an approved these changes

@IMbackK IMbackK IMbackK approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@JohannesGaessler @nisparks @IMbackK @am17an @ORippler