Running Inspect AI Benchmarks with Asteroid’ Approvers

This guide explains how to run any benchmark from Inspect AI using Asteroid’ approvers. We’ll provide specific examples for benchmarks like InterCode: Capture the Flag or GAIA and show how to configure approvals for these evaluations.

For setup instructions and details on how the approvers work and integrate, please refer to the Inspect AI documentation.

You can find the list of available evaluations here or in the Inspect AI repository.

Benchmarks

Below is a list of benchmarks with steps to run each one. Click on a benchmark to see the instructions.

First install the Inspect eval benchmark package:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

The GDM In-House CTF evaluates general capabilities in a capture-the-flag setting.

Run the GDM In-House CTF Evaluation

python examples/inspect_evals/gdm_in_house_ctf/run.py

This command will run the GDM In-House CTF evaluation with 2 samples and use the approval configuration specified in the approval.yaml file.

Set Up the Approval Configuration

Edit the approval.yaml file as needed. An example configuration:

approvers:
  - name: asteroid_sdk/bash_approver
    tools: "*bash*"
    allowed_commands: ["ls", "cd", "pwd", "echo", "cat", "grep", "mkdir", "cp", "wget", "curl", "pip"]
    allow_sudo: false
    command_specific_rules:
      pip: ["install", "list", "show"]

  - name: asteroid_sdk/python_approver
    tools: "*python*"
    allowed_modules: ["requests", "json", "csv", "datetime", "re", "math", "random", "time"]
    allowed_functions: ["print", "len", "range", "str", "int", "float", "list", "dict", "set", "tuple", "sum", "max", "min"]
    disallowed_builtins: ["eval", "exec", "compile", "__import__", "open", "input"]
    sensitive_modules: ["os", "sys", "subprocess", "socket"]
    allow_system_state_modification: false

  - name: asteroid_sdk/human_approver
    tools: "*"
    approval_api_endpoint: "http://localhost:8080"
    agent_id: "testing_agent"
    n: 2

More information on how to run the GDM In-House CTF benchmark with Inspect AI can be found here.

Intercode CTF is another capability evaluation in a capture-the-flag format.

Navigate to the Intercode CTF Directory

cd examples/inspect_evals/intercode_ctf

Run the Intercode CTF Evaluation

python inspect eval run.py --approval approval.yaml --model openai/gpt-4o --trace

This command will run the Intercode CTF evaluation with 2 samples and use the approval configuration specified in the approval.yaml file.

Edit the Approval Configuration

You can edit Inspect evals configuration in run.py. Approval configuration can be configured in the approval.yaml file. An example configuration:

approvers:
  - name: asteroid_sdk/bash_approver
    tools: "*bash*"
    allowed_commands: ["ls", "cd", "pwd", "echo", "cat", "grep", "mkdir", "cp", "wget", "curl", "pip"]
    allow_sudo: false
    command_specific_rules:
      pip: ["install", "list", "show"]

  - name: asteroid_sdk/python_approver
    tools: "*python*"
    allowed_modules: ["requests", "json", "csv", "datetime", "re", "math", "random", "time"]
    allowed_functions: ["print", "len", "range", "str", "int", "float", "list", "dict", "set", "tuple", "sum", "max", "min"]
    disallowed_builtins: ["eval", "exec", "compile", "__import__", "open", "input"]
    sensitive_modules: ["os", "sys", "subprocess", "socket"]
    allow_system_state_modification: false

  - name: asteroid_sdk/human_approver
    tools: "*"
    approval_api_endpoint: "http://localhost:8080"
    agent_id: "testing_agent"
    n: 2

More information on how to run the Intercode CTF benchmark with Inspect AI can be found here.

The GAIA (General AI Assistants) benchmark consists of 450 questions testing tool use on realistic assistant tasks (mostly web browsing).

Note: Access to dataset gaia-benchmark/GAIA from Hugging Face is restricted. You must have access to it and be authenticated to access it.

Navigate to the GAIA Directory

cd examples/inspect_evals/gaia

Run the GAIA Evaluation

python inspect eval run.py --approval approval.yaml --model openai/gpt-4o --trace

This command will run the GAIA evaluation with 2 samples and use the approval configuration specified in the approval.yaml file.

Edit the Approval Configuration

You can edit Inspect evals configuration in run.py. Approval configuration can be configured in the approval.yaml file. An example configuration:

approvers:
  - name: asteroid_sdk/bash_approver
    tools: "*bash*"
    allowed_commands: ["ls", "cd", "pwd", "echo", "cat", "grep", "mkdir", "cp", "wget", "curl", "pip"]
    allow_sudo: false
    command_specific_rules:
      pip: ["install", "list", "show"]

  - name: asteroid_sdk/python_approver
    tools: "*python*"
    allowed_modules: ["requests", "json", "csv", "datetime", "re", "math", "random", "time"]
    allowed_functions: ["print", "len", "range", "str", "int", "float", "list", "dict", "set", "tuple", "sum", "max", "min"]
    disallowed_builtins: ["eval", "exec", "compile", "__import__", "open", "input"]
    sensitive_modules: ["os", "sys", "subprocess", "socket"]
    allow_system_state_modification: false

  - name: asteroid_sdk/human_approver
    tools: "*"
    approval_api_endpoint: "http://localhost:8080"
    agent_id: "testing_agent"
    n: 2

More information on how to run the GAIA benchmark with Inspect AI can be found here.

Customizing Approvals

You can customize the approval configurations by editing the approval.yaml files for each benchmark. This allows you to:

Specify which tools require approval.
Define allowlists for commands and functions.
Configure human or LLM approval.

For details on how the approvers work and how to integrate them, please refer to the Inspect AI documentation.

Get Started

Asteroid

Integrations

Cookbook

Inspect Benchmarks

Running Inspect AI Benchmarks with Asteroid’ Approvers

Benchmarks

Customizing Approvals

Additional Resources

Get Started

Asteroid

Integrations

Cookbook

​Running Inspect AI Benchmarks with Asteroid’ Approvers

​Benchmarks

​Customizing Approvals

​Additional Resources

Running Inspect AI Benchmarks with Asteroid’ Approvers

Benchmarks

Customizing Approvals

Additional Resources