OpenAI Requires Past Job Work for AI Agent Evaluation
- OpenAI is soliciting real-world work assignments from third-party contractors to assess the capabilities of its upcoming AI models, according to records from openai and the training data company...
- The project aims to create a human performance benchmark for various tasks, allowing openai to compare AI model outputs against human professionals.
- OpenAI is specifically requesting contractors to upload assignments and tasks from thier current or previous workplaces.
OpenAI Requests Work Samples from Contractors to Evaluate AI Performance
Table of Contents
OpenAI is soliciting real-world work assignments from third-party contractors to assess the capabilities of its upcoming AI models, according to records from openai and the training data company Handshake AI obtained by WIRED.
Establishing a Human Baseline
The project aims to create a human performance benchmark for various tasks, allowing openai to compare AI model outputs against human professionals. This initiative began in september 2023 with the launch of a new evaluation process designed to measure AI performance across diverse industries. OpenAI considers this a crucial metric in its pursuit of Artificial general Intelligence (AGI), defined as an AI system exceeding human capabilities in most economically valuable tasks.
Data Collection Process
OpenAI is specifically requesting contractors to upload assignments and tasks from thier current or previous workplaces. The company stated they “hired folks across occupations to help collect real-world tasks modeled off those you’ve done in your full-time jobs, so we can measure how well AI models perform on those tasks.”
Significance of AGI Measurement
OpenAI views the ability to accurately measure AI performance against human professionals as a key indicator of progress toward AGI. The company’s evaluation process, launched September 2023, is intended to provide this measurement across a wide range of industries.