Skip to main content

Quick Start Guide

Welcome to Testworthy! This guide will help your organization set up a complete testing environment that leverages AI automation, repository integration, and team management features. Whether you're implementing testing infrastructure for a small team or a larger organization, this guide covers everything from initial setup to AI-powered testing workflows.

Guided Tours: Testworthy includes built-in interactive guided tours that automatically appear on first login. A Dashboard Tour walks you through the platform navigation and key features (16 steps, tailored to your role), and a Create Project Tour guides you through the AI-powered project creation wizard. You can re-trigger the Dashboard Tour anytime via the "?" button on the Dashboard.

Phase 1: Organization Setup & Billing​

1. Create Your Enterprise Organization​

Initial Setup:

  • Sign up at the Testworthy platform and create your organization account
  • Choose Custom Subdomain: Select your organization's unique subdomain (yourcompany.testworthy.us)
  • Configure Organization Settings: Set organization name, time zone, and default preferences
  • Verify Your Identity: Complete email verification and any required onboarding steps

Select Subscription Plan:

  • Trial (Free): Up to 3 team members, 350 AI credits/month — Perfect for evaluation
  • Starter ($49/month): Up to 10 team members, 3,500 AI credits/month — Small teams getting started
  • Standard ($99/month): Up to 20 team members, 9,000 AI credits/month — Growing QA teams
  • Professional ($199/month): Unlimited team members, 12,000 AI credits/month — Enterprise organizations

All plans include the full feature set: AI test generation, GitHub/GitLab/Bitbucket integration, Jira integration, automation script export, and email support.

2. Configure Billing & Payment​

Stripe Integration Setup:

  • Navigate to Organization → Billing in your dashboard
  • Add payment method (credit card) via Stripe Checkout
  • Review subscription features and confirm your plan selection
  • Choose monthly or yearly billing cycle (yearly at a 20% discount)

Cost Management:

  • Review Credit Usage: Understand how AI credits are consumed across features (see Credits & Billing page)
  • Distribute Quotas: Tenant owners allocate credits from the organization pool to individual users via the Manage Allocations modal
  • Purchase Top-Ups: Buy additional credit packages via Stripe; purchased credits expire after 365 days
  • Monitor Billing: Access invoices, payment history, and Stripe billing portal

Phase 2: Team Management & Access Control​

3. Create Team Members & Configure Roles​

Role-Based Access Control:

Organization Roles:

  • Platform Admin: System-wide administrative access across all organizations
  • Tenant Owner: Full organization ownership — billing, user management, credit distribution
  • Project Manager: Project management, user coordination, API keys, and integrations
  • QA: Test case creation, execution, defect management, and AI feature access
  • Developer: Test artifact CRUD access (same capabilities as QA)
  • Guest: Read-only access across the organization

Project-Level Roles:

  • Admin: Full project management and configuration
  • Member: Standard project access for testing and collaboration
  • Viewer: Read-only access for stakeholders

Team Setup Process:

  1. Create Users: Navigate to User Management and create users with email, name, password, and role
  2. Allocate Credit Quotas: Set each user's monthly credit quota from the organization pool
  3. Add to Projects: Assign users to specific projects with project-level roles
  4. Verify Limits: Ensure total user count stays within subscription tier limits

4. Security & Compliance Configuration​

Access Control Policies:

  • OTP Verification: Email-based two-factor authentication for enhanced security
  • Password Policy: 8+ characters, uppercase letter, digit, and special character required
  • JWT Sessions: Secure session management with automatic token refresh

Audit & Compliance:

  • Activity Logging: Comprehensive activity tracking for all user actions
  • Data Encryption: Sensitive data (API tokens, credentials) encrypted at rest and in transit

Phase 3: AI Integration & Repository Configuration​

5. AI Setup​

AI Provider Configuration:

  • Navigate to AI Settings in the sidebar
  • Select your preferred AI provider from the dropdown:
    • Vertex AI — Gemini 3.1 Pro Preview, Gemini 3.1 Flash Lite Preview, Gemini 2.5 Pro, Gemini 2.5 Flash
    • Anthropic (via Vertex AI) — Claude 4.5 Haiku, Claude 4.5 Sonnet
    • Llama (via Vertex AI) — Llama 4 Maverick 17B, Llama 4 Scout 17B
    • Llama (via OpenRouter) — Llama 4 Maverick, Llama 4 Scout
  • Choose a model for your provider
  • Adjust temperature and max context window settings
  • API keys are configured at the platform level via environment variables — individual users do not need to supply keys

Credit Management:

  • Organization Pool: Your subscription tier determines the monthly credit quota (350–12,000)
  • User Distribution: Tenant owners allocate credits from the org pool to individual users via the Manage Allocations modal
  • Purchased Credits: Buy additional credit packages that expire after 365 days (FIFO deduction)
  • Usage Monitoring: Each AI operation tracks input/output tokens, provider, model, and estimated cost
  • Quota Enforcement: Operations are blocked when both monthly and purchased credits are exhausted

6. Repository Integration Setup​

GitHub Integration:

  1. Organization Settings: Navigate to Organization → Integrations → GitHub
  2. Install GitHub App: Authorize Testworthy's GitHub application for your organization
  3. Configure Webhooks: Set up secure webhooks for pull request analysis
  4. Repository Selection: Choose which repositories to monitor for automated test recommendations

GitLab Integration:

  1. GitLab Configuration: Navigate to Organization → Integrations → GitLab
  2. OAuth Setup: Connect your GitLab account with appropriate permissions
  3. Webhook Configuration: Set up merge request webhooks for automated analysis
  4. Security Settings: Configure secure token management

Bitbucket Integration:

  1. Bitbucket Configuration: Navigate to Organization → Integrations → Bitbucket
  2. Authentication: Connect with Bitbucket credentials (username/email)
  3. Webhook Configuration: Set up pull request webhooks for automated analysis
  4. Repository Selection: Choose which repositories to monitor

Automated Workflows:

  • PR Analysis: Enable AI-powered test recommendations for code changes
  • Test Impact Analysis: Configure automatic identification of tests affected by code modifications

Phase 4: Advanced Project Setup​

7. AI-Powered Project Creation​

Codebase Analysis & Import:

  1. Project Creation: Navigate to Projects → Create New Project
  2. Choose AI-Powered Setup: Select "Import from Codebase" or "Import from PRD" option
  3. Upload Source Material:
    • Codebase: Prepare ZIP file with your application source code
    • PRD Document: Upload a Product Requirements Document (PDF, Word, TXT, or Markdown)
    • Review file size limits and supported programming languages
    • Cost Estimation: Preview AI credit usage before proceeding with analysis

Tip: On first use, an interactive guided tour will walk you through each step of the AI-powered project creation wizard.

AI Generation Process:

  • Code Structure Analysis: AI analyzes your codebase architecture and identifies testable components
  • Test Case Generation: Comprehensive test cases generated based on code patterns and business logic
  • Suite Organization: Logical test suite structure created based on application modules

Review & Customization:

  • Generated Structure Review: Examine AI-generated test suites, sections, and test cases
  • Quality Assessment: Review test case completeness and relevance
  • Customization: Modify generated content to match your testing standards and requirements

Phase 5: Advanced Test Management​

8. Hierarchical Test Organization​

Multi-Level Structure Setup:

  • Test Suites: Create feature-based or module-based top-level organization
  • Sections: Implement nested hierarchical organization for complex test structures
  • Parent-Child Relationships: Configure nested section hierarchies

Management:

  • Version Control: Implement comprehensive versioning for all test artifacts

9. Automation & Script Generation​

Framework Support:

  • Playwright: TypeScript, JavaScript, Python, Java, C#
  • Selenium: Python, JavaScript, Java, C#, Ruby, Kotlin
  • Cypress: JavaScript, TypeScript

Generation Flow:

  1. Open a test case and navigate to the Automation tab
  2. Select framework, language, and optionally override AI provider/model
  3. Add additional instructions if needed
  4. Generate, review, edit, and save—each save creates a version snapshot
  5. Copy to clipboard or execute via the Testworthy Runner

Phase 6: Execution & Defect Management​

10. Test Execution​

Execution Workflow:

  1. Test Plan Creation: Organize test cases into execution-ready test plans
  2. Resource Assignment: Allocate team members to test runs
  3. Execution Monitoring: Track progress and record results with detailed status options
  4. Result Recording: Capture pass/fail/blocked/retest statuses with comments and attachments

11. Defect Management Integration​

External System Integration:

Jira Integration:

  • Connection Setup: Configure Jira Cloud connection with base URL, email, and API token per project
  • Personal Overrides: Users can save personal Jira credentials that override the project default
  • Issue Operations: Create Jira issues from failed tests, fetch live issue status, and link test cases to Jira keys
  • Bulk Status View: View all Jira-linked defects across a project with current statuses

GitHub Issues / Bugzilla:

  • External ID Linking: Link defects to GitHub or Bugzilla issues by external issue ID
  • Defect Type Tracking: Store defect type (GitHub, Bugzilla) alongside the external reference

Phase 7: Analytics & Reporting​

12. Reporting Configuration​

Scheduled Reports:

  • Report Types: Milestone, Test Plan, and Test Run reports
  • Schedule Options: Daily, weekly, or monthly PDF generation
  • Dashboard Metrics: View pass rates, failure trends, and project health from dashboards

AI Usage Analytics:

  • Credit Tracking: Monitor consumption by operation type, provider, and model
  • Cost Analysis: Track estimated costs and optimize AI usage
  • Quota Management: View monthly quota usage and purchased credit balances

Phase 8: Security & Audit​

13. Compliance & Audit Capabilities​

Audit Trail:

  • Activity Logging: Comprehensive activity tracking for all user actions
  • Version History: Complete version control for test cases and project structures
  • User Attribution: All changes tracked with user and timestamp

Security:

  • JWT Authentication: Secure session management with refresh tokens
  • OTP Verification: Email-based two-factor authentication
  • Data Encryption: Encrypted storage of sensitive data and API keys
  • Role-Based Access: Granular permissions at organization and project level

Troubleshooting & Support​

Common Setup Issues​

Authentication Problems:

  • Verify email addresses and account configurations
  • Check firewall settings for webhook endpoints
  • Review OAuth permissions for repository integrations

Performance Issues:

  • Monitor AI credit usage and optimize consumption patterns
  • Review data volumes for large test repositories

Integration Challenges:

  • Verify API permissions for external system connections
  • Check webhook configuration and security settings

Getting Help​

Documentation Resources:

  • User Guide Sections: Comprehensive feature documentation
  • API Documentation: Complete REST API reference
  • Integration Guides: Detailed setup instructions for external systems

Support Channels:

  • Email Support: Available for all subscription tiers
  • Community Forum: Peer support and best practice sharing

What's Next?​

After completing this setup:

  1. Monitor Usage: Track AI credit consumption and optimize usage patterns
  2. Gather Feedback: Collect team feedback on workflows and identify improvement opportunities
  3. Expand Integration: Connect additional repositories for automated PR analysis
  4. Scale Operations: Add more projects as your testing needs grow

Tip: Start with a pilot project to validate your setup and configuration before rolling out organization-wide. Use the trial period to fine-tune AI usage patterns and optimize workflows for your team's specific needs.

For detailed information about any specific feature mentioned in this guide, refer to the relevant sections in the User Guide documentation.