1 | π₯ O1 Preview (Sept 2024)β¨ | β‘89 |
2 | π₯ O1 Mini (Sept 2024)β¨ | β‘89 |
3 | π₯ Qwen2.5-Coder-32B-Instructβ¨ | β‘87.2 |
4 | GPT 4o (Aug 2024)β¨ | β‘87.2 |
5 | DeepSeek-V3 (Nov 2024)β¨ | β‘86.6 |
6 | GPT-4-Turbo (April 2024)β¨ | β‘86.6 |
7 | DeepSeek-V2.5 (Nov 2024)β¨ | β‘83.5 |
8 | GPT 4o Mini (July 2024)β¨ | β‘83.5 |
9 | DeepSeek-Coder-V2-Instructβ¨ | β‘82.3 |
10 | Claude Sonnet 3.5 (June 2024)β¨ | β‘81.7 |
11 | GPT-4-Turbo (Nov 2023)β¨ | β‘81.7 |
12 | Grok Betaβ¨ | β‘80.5 |
13 | Gemini 1.5 Pro 002β¨ | β‘79.3 |
14 | GPT-4 (May 2023)β¨ | β‘79.3 |
15 | CodeQwen1.5-7B-Chatβ¨ | β‘78.7 |
16 | OpenCoder-8B-Instructβ¨ | β‘77.4 |
17 | claude-3-opus (Mar 2024)β¨ | β‘77.4 |
18 | Gemini 1.5 Flash 002β¨ | β‘75.6 |
19 | DeepSeek-Coder-33B-instructβ¨ | β‘75 |
20 | Codestral-22B-v0.1β¨ | β‘73.8 |
21 | OpenCodeInterpreter-DS-33Bβ¨π | β‘73.8 |
22 | WizardCoder-33B-V1.1β¨ | β‘73.2 |
23 | Artigenz-Coder-DS-6.7Bβ¨ | β‘72.6 |
24 | Llama3-70B-instructβ¨ | β‘72 |
25 | Mixtral-8x22B-Instruct-v0.1β¨ | β‘72 |
26 | OpenCodeInterpreter-DS-6.7Bβ¨π | β‘72 |
27 | speechless-codellama-34B-v2.0β¨π | β‘72 |
28 | DeepSeek-Coder-6.7B-instructβ¨ | β‘71.3 |
29 | DeepSeek-Coder-7B-instruct-v1.5β¨ | β‘71.3 |
30 | Magicoder-S-DS-6.7Bβ¨π | β‘71.3 |
31 | starchat2-15b-v0.1β¨π | β‘71.3 |
32 | GPT-3.5-Turbo (Nov 2023)β¨ | β‘70.7 |
33 | code-millenials-34Bβ¨ | β‘70.7 |
34 | databricks/dbrx-instructβ¨ | β‘70.1 |
35 | WaveCoder-Ultra-6.7Bβ¨ | β‘69.5 |
36 | XwinCoder-34Bβ¨ | β‘69.5 |
37 | claude-3-haiku (Mar 2024)β¨ | β‘68.9 |
38 | Magicoder-S-CL-7Bβ¨π | β‘67.7 |
39 | OpenChat-3.5-7B-0106β¨π | β‘67.7 |
40 | Phind-CodeLlama-34B-v2 | β‘67.1 |
41 | GPT-3.5 (May 2023)β¨ | β‘66.5 |
42 | CodeLlama-70B-Instructβ¨ | β‘65.9 |
43 | WhiteRabbitNeo-33B-v1β¨ | β‘65.9 |
44 | speechless-coder-ds-6.7Bβ¨π | β‘65.9 |
45 | WizardCoder-Python-34B-V1.0β¨ | β‘64.6 |
46 | claude-3-sonnet (Mar 2024)β¨ | β‘64 |
47 | Llama3.1-8B-instructβ¨ | β‘62.8 |
48 | speechless-starcoder2-15bβ¨π | β‘62.8 |
49 | Mistral Large (Mar 2024)β¨ | β‘62.2 |
50 | claude-2 (Mar 2024)β¨ | β‘61.6 |
51 | Gemini Pro 1.5β¨ | β‘61 |
52 | DeepSeek-Coder-1.3B-instructβ¨ | β‘60.4 |
53 | starcoder2-15b-instruct-v0.1β¨π | β‘60.4 |
54 | Code-290k-6.7B-Instructβ¨π | β‘59.7 |
55 | Qwen1.5-72B-Chatβ¨ | β‘59.1 |
56 | Phi-3-mini-4k-instructβ¨ | β‘59.1 |
57 | dolphin-2.6-mixtral-8x7bβ¨π | β‘57.3 |
58 | Command-R+β¨ | β‘56.7 |
59 | Llama3-8B-instructβ¨ | β‘56.7 |
60 | Gemini Pro 1.0β¨ | β‘55.5 |
61 | Code-13Bβ¨π | β‘52.4 |
62 | codegemma-7b-itβ¨ | β‘51.8 |
63 | speechless-starcoder2-7bβ¨π | β‘51.8 |
64 | CodeLlama-70B | β‘50.6 |
65 | WizardCoder-15B-V1.0β¨ | β‘50.6 |
66 | claude-instant-1 (Mar 2024)β¨ | β‘50.6 |
67 | speechless-coding-7B-16k-toraβ¨π | β‘50.6 |
68 | Code-33Bβ¨π | β‘49.4 |
69 | OpenHermes-2.5-Code-290k-13Bβ¨π | β‘48.8 |
70 | CodeQwen1.5-7B | β‘45.7 |
71 | WizardCoder-Python-7B-V1.0β¨ | β‘45.1 |
72 | phi-2-2.7B | β‘45.1 |
73 | DeepSeek-Coder-33B-base | β‘44.5 |
74 | CodeLlama-34B | β‘43.9 |
75 | Mistral-codealpaca-7Bπ | β‘42.1 |
76 | MistralHermes-CodePro-7B-v1β¨π | β‘42.1 |
77 | codegemma-7b | β‘41.5 |
78 | speechless-code-mistral-7B-v1.0β¨π | β‘41.5 |
79 | DeepSeek-Coder-6.7B-base | β‘39.6 |
80 | Mixtral-8x7B-Instruct-v0.1β¨ | β‘39.6 |
81 | CodeLlama-13B | β‘38.4 |
82 | StarCoder2-15Bπ | β‘37.8 |
83 | SOLAR-10.7B-Instruct-v1.0β¨π | β‘37.2 |
84 | Mistral-7B-Instruct-v0.2β¨ | β‘36 |
85 | CodeLlama-7B | β‘35.4 |
86 | gemma-1.1-7b-itβ¨ | β‘35.4 |
87 | xDAN-L1-Chat-RL-v1-7Bβ¨π | β‘32.9 |
88 | Python-Code-13Bβ¨π | β‘30.5 |
89 | StarCoder2-7Bπ | β‘29.9 |
90 | Llama3-8B-base | β‘29.3 |
91 | StarCoder-15Bπ | β‘29.3 |
92 | gemma-7b | β‘28.7 |
93 | CodeGen-16Bπ | β‘28 |
94 | StarCoder2-3Bπ | β‘27.4 |
95 | CodeT5+-16Bπ | β‘26.8 |
96 | CodeGen-6Bπ | β‘25.6 |
97 | DeepSeek-Coder-1.3B-base | β‘25.6 |
98 | stable-code-3Bπ | β‘25.6 |
99 | gemma-7b-itβ¨ | β‘25 |
100 | CodeT5+-6Bπ | β‘24.4 |
101 | Mistral-7B | β‘23.8 |
102 | Zephyr Ξ²-7Bπ | β‘23.2 |
103 | CodeGen-2Bπ | β‘22.6 |
104 | CodeT5+-2Bπ | β‘22 |
105 | StarCoderBase-7Bπ | β‘21.3 |
106 | codegemma-2b | β‘20.7 |
107 | gemma-2b | β‘20.7 |
108 | CodeGen2-7Bπ | β‘17.7 |
109 | gemma-1.1-2b-itβ¨ | β‘17.7 |
110 | CodeGen2-16Bπ | β‘16.5 |
111 | StarCoderBase-3Bπ | β‘15.9 |
112 | Vicuna-13Bπ | β‘15.9 |
113 | gemma-2b-itβ¨ | β‘15.2 |
114 | SantaCoder-1.1Bπ | β‘14 |
115 | CodeGen2-3Bπ | β‘12.8 |
116 | InCoder-6.7Bπ | β‘12.2 |
117 | StarCoderBase-1Bπ | β‘12.2 |
118 | Vicuna-7Bπ | β‘11.6 |
119 | GPT-J-6Bπ | β‘11 |
120 | InCoder-1.3Bπ | β‘11 |
121 | CodeGen2-1Bπ | β‘9.1 |
122 | GPT-Neo-2.7Bπ | β‘6.7 |
123 | PolyCoder-2.7Bπ | β‘6.1 |
124 | StableLM-7B | β‘2.4 |
125 | zyte-1Bβ¨π | β‘1.8 |