GitHub Copilot Struggles in Coding Tests, Fails Half of Challenges

SAN FRANCISCO, Calif. — GitHub Copilot, an AI-powered coding assistant, has shown mixed results in recent performance tests, failing half of the challenges it faced. The tool, which is integrated into Microsoft‘s Visual Studio Code (VS Code), is based on OpenAI‘s GPT-4 model but underperformed compared to other AI tools like ChatGPT and Perplexity Pro.

In a series of tests conducted by a developer, GitHub Copilot was tasked with creating a fully functional WordPress plugin, fixing code errors, and solving complex coding problems. While it succeeded in some tasks, it failed in others, particularly in generating JavaScript code and handling edge cases in currency validation.

“GitHub Copilot passed two tests and failed two,” the developer noted. “This was my first test, so I can’t yet determine whether GitHub Copilot is terrible at writing code or if the context in which it operates is too limiting.”

One of the failed tests involved creating a WordPress plugin that sorts names and separates duplicates. GitHub Copilot generated PHP code but failed to produce the necessary JavaScript for interactive features. In another test, it attempted to fix a currency validation function but produced code that would crash on edge cases.

However, GitHub Copilot did succeed in solving a tricky bug related to WordPress API calls, outperforming Microsoft Copilot, Gemini, and Meta Code Llama. “GitHub Copilot got this right,” the developer said. “It solved a problem that required deep understanding of the WordPress framework.”

Despite its mixed performance, GitHub Copilot remains a popular tool among developers. GitHub recently launched a free version of Copilot, making it accessible to individual developers without requiring a subscription. The free tier is integrated into VS Code and JetBrains IDEs, offering limited access to Copilot’s features.

“With this Free Plan, you can enable GitHub Copilot using just your GitHub account—no trials, subscriptions, or credit cards required,” GitHub announced. The free version is aimed at individual developers, while the Pro version offers more advanced features for organizations and enterprises.

Developers can also customize GitHub Copilot by choosing different large language models (LLMs) from the GitHub Marketplace. While mainstream models like GPT-4 are available, some specialized models, such as IBM‘s Granite, are not yet included.

As AI tools continue to evolve, developers are optimistic about their potential. “I’m sure performance will get better,” the developer said. “Let’s stay tuned and check back in a few months to see if the AI is more effective at that time.”