---
sidebar_label: Bitbucket Pipelines
description: Integrate promptfoo LLM testing with Bitbucket Pipelines CI/CD to automate evaluations, track results, and catch regressions in your AI models using built-in assertions
---

# Bitbucket Pipelines Integration

This guide demonstrates how to set up promptfoo with Bitbucket Pipelines to run evaluations as part of your CI pipeline.

## Prerequisites

- A Bitbucket repository with a promptfoo project
- Bitbucket Pipelines enabled for your repository
- API keys for your LLM providers stored as [Bitbucket repository variables](https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/)

## Setting up Bitbucket Pipelines

Create a new file named `bitbucket-pipelines.yml` in the root of your repository with the following configuration:

```yaml
image: node:24

pipelines:
  default:
    - step:
        name: Promptfoo Evaluation
        caches:
          - node
        script:
          - npm ci
          - npm install -g promptfoo
          - npx promptfoo eval -o promptfoo-results.json -o promptfoo-results.junit.xml
        artifacts:
          - promptfoo-results.json
          - promptfoo-results.junit.xml
```

## Environment Variables

Store your LLM provider API keys as repository variables in Bitbucket:

1. Navigate to your repository in Bitbucket
2. Go to Repository settings > Pipelines > Repository variables
3. Add variables for each provider API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
4. Mark them as "Secured" to ensure they're not displayed in logs

## Advanced Configuration

### Fail the Pipeline on Failed Assertions

You can configure the pipeline to fail when promptfoo assertions don't pass:

```yaml
script:
  - npm ci
  - npm install -g promptfoo
  - npx promptfoo eval --fail-on-error
```

### Custom Evaluation Configurations

Run evaluations with specific configuration files:

```yaml
script:
  - npm ci
  - npm install -g promptfoo
  - npx promptfoo eval --config custom-config.yaml
```

### Run on Pull Requests

Configure different behavior for pull requests:

```yaml
pipelines:
  default:
    - step:
        name: Promptfoo Evaluation
        script:
          - npm ci
          - npm install -g promptfoo
          - npx promptfoo eval
  pull-requests:
    '**':
      - step:
          name: Promptfoo PR Evaluation
          script:
            - npm ci
            - npm install -g promptfoo
            - npx promptfoo eval --fail-on-error
```

### Scheduled Evaluations

Run evaluations on a schedule:

```yaml
pipelines:
  default:
    - step:
        name: Promptfoo Evaluation
        script:
          - npm ci
          - npm install -g promptfoo
          - npx promptfoo eval
  custom:
    nightly-evaluation:
      - step:
          name: Nightly Evaluation
          script:
            - npm ci
            - npm install -g promptfoo
            - npx promptfoo eval
  schedules:
    - cron: '0 0 * * *' # Run at midnight UTC every day
      pipeline: custom.nightly-evaluation
      branches:
        include:
          - main
```

### Parallel Testing

Test across multiple configurations in parallel:

```yaml
image: node:24

pipelines:
  default:
    - parallel:
        - step:
            name: Evaluate with GPT-4
            script:
              - npm ci
              - npm install -g promptfoo
              - npx promptfoo eval --providers openai:gpt-4 -o promptfoo-results-gpt4.json
            artifacts:
              - promptfoo-results-gpt4.json
        - step:
            name: Evaluate with Claude
            script:
              - npm ci
              - npm install -g promptfoo
              - npx promptfoo eval --providers anthropic:claude-opus-5 -o promptfoo-results-claude.json
            artifacts:
              - promptfoo-results-claude.json
```

### Using Pipes

Leverage Bitbucket Pipes for a more concise configuration:

```yaml
image: node:24

pipelines:
  default:
    - step:
        name: Promptfoo Evaluation
        script:
          - npm ci
          - npm install -g promptfoo
          - npx promptfoo eval -o promptfoo-results.junit.xml
        after-script:
          - pipe: atlassian/junit-report:0.3.0
            variables:
              REPORT_PATHS: 'promptfoo-results.junit.xml'
```

## Troubleshooting

If you encounter issues with your Bitbucket Pipelines integration:

- **Check logs**: Review detailed logs in Bitbucket to identify errors
- **Verify repository variables**: Ensure your API keys are correctly set
- **Pipeline timeouts**: Split long-running evaluations into smaller steps, or raise a step's [`max-time`](https://support.atlassian.com/bitbucket-cloud/docs/step-options/#Max-time)
- **Debug locally**: [Run the failing build step in Docker](https://support.atlassian.com/bitbucket-cloud/kb/debug-pipelines-locally-with-docker/) to reproduce it locally