Private Equity Deal Sourcer

Project Blueprint: Private Equity Deal Sourcer

Subtitle: Identify targets matching investment mandates

1. The Business Problem (Why build this?)

The private equity (PE) landscape is fiercely competitive, with firms constantly seeking an edge in deal sourcing. Traditionally, identifying potential acquisition targets is a labor-intensive, often inefficient process. Investment professionals (associates, VPs) spend countless hours manually sifting through disparate data sources – company websites, news articles, industry reports, M&A databases, LinkedIn profiles, and proprietary data feeds – to find companies that align with their firm's specific investment mandates. This manual approach presents several critical challenges:

• Time & Resource Drain: Analysts dedicate significant time to repetitive data gathering and preliminary screening, diverting them from higher-value activities like due diligence and relationship building. This translates directly to increased operational costs and slower deal cycles.
• Incomplete Market Coverage: Manual sourcing inherently limits the breadth of the search. Firms may miss out on promising targets simply because they weren't on their radar or required an exhaustive search effort across a fragmented information landscape.
• Inconsistent Data Quality & Format: Information gathered from various sources often comes in different formats, with varying levels of accuracy and completeness, making aggregation and comparison difficult. This leads to manual data normalization and potential errors.
• Suboptimal Mandate Matching: The human element in initial screening introduces subjectivity and potential for bias. It's challenging for even experienced professionals to consistently and comprehensively evaluate every nuanced aspect of a company against a complex investment mandate, leading to potential misses or false positives.
• Scalability Issues: As investment teams grow or mandates evolve, scaling the manual sourcing effort becomes prohibitively expensive and complex. There's no inherent leverage in the current process.
• Lack of Proactive Intelligence: Traditional methods are often reactive. Firms struggle to proactively identify emerging trends, new market entrants, or companies poised for growth before they become widely known or enter a formal sale process.

The "Private Equity Deal Sourcer" application addresses these pain points by automating and intelligently augmenting the initial stages of the deal sourcing process. It aims to transform a manual, bottlenecked operation into a scalable, data-driven, and AI-powered engine for identifying high-probability investment targets, thereby increasing efficiency, expanding market coverage, and improving the quality of the deal flow.

2. Solution Overview

The Private Equity Deal Sourcer is a sophisticated web application designed to automate the discovery and initial qualification of potential investment targets that align with a firm's specific investment mandates. It operates as an intelligent overlay on publicly available and selected proprietary data, leveraging advanced AI to extract, normalize, and match company profiles against user-defined criteria.

Core Workflow:

1. Mandate Definition: Investment professionals define detailed investment mandates within the application, specifying criteria such as industry sectors, revenue ranges, geographical focus, growth characteristics, technology stack, employee count, and specific strategic keywords.
2. Web Scraping Orchestration: The system continuously or on-demand scrapes and ingests data from a diverse array of web sources (e.g., company websites, business directories, news portals, industry-specific publications, regulatory filings).
3. Data Ingestion & Normalization: Raw, unstructured data from scraping is processed, cleaned, and transformed into a structured, unified company profile format. This involves entity extraction, data parsing, and deduplication.
4. AI-Powered Mandate Matching: Leveraging the Gemini API, the normalized company profiles are semantically matched against the defined investment mandates. A probabilistic matching score is generated, along with explanations for the match.
5. Target Profiling & Enrichment: Beyond basic firmographics, the system enriches target profiles by extracting strategic insights, growth signals, competitive landscapes, and potential risks through advanced NLP techniques.
6. Review & Refinement: Matched targets are presented in an intuitive dashboard, allowing users to review profiles, adjust matching logic, provide feedback, and qualify/disqualify targets.
7. CRM Export & Integration: Qualified targets, along with their enriched profiles and matching scores, can be exported into standard CRM formats (CSV/JSON) or directly integrated with popular CRM systems.
8. Deal Flow Dashboard: A real-time dashboard provides an overview of sourcing activities, pipeline status, mandate performance, and key analytics.

Key Benefits:

• Automated, Scalable Sourcing: Dramatically reduces manual effort and expands the universe of potential targets.
• Enhanced Matching Accuracy: AI-driven semantic understanding ensures more precise alignment with complex investment criteria.
• Richer Target Intelligence: Provides comprehensive, consolidated profiles for quicker initial assessments.
• Data-Driven Decision Making: Offers metrics and insights into sourcing effectiveness and market trends.
• Streamlined Workflow: Integrates seamlessly into existing deal-sourcing processes, from discovery to CRM entry.

3. Architecture & Tech Stack Justification

The architecture is designed for scalability, modularity, and leveraging cutting-edge AI capabilities while maintaining developer velocity and a robust user experience.

High-Level Architecture:

+---------------------+      +---------------------+      +----------------------+
|     Frontend UI     |      |  Backend Services   |      |   External Services  |
| (Next.js, Tailwind) |      | (Firebase Genkit,   |      | (Web Scraped Sites,  |
|                     | <--->|    Cloud Functions, | <--->|   External APIs)     |
|                     |      |     Firestore)      |      |                      |
+---------------------+      +---------------------+      +----------------------+
       ^                               ^
       |                               |
       v                               v
+--------------------------------------------------+
|               Google Gemini API                  |
+--------------------------------------------------+
       ^                               ^
       |                               |
       v                               v
+--------------------------------------------------+
|               Google Cloud Platform              |
| (Cloud Run, Cloud Storage, Pub/Sub, Cloud Tasks) |
+--------------------------------------------------+

Tech Stack Justification:

• Next.js (Frontend Framework):

  • Justification: Provides a robust framework for building performant, server-rendered React applications. Its features like file-system based routing, API routes, and built-in image optimization accelerate development. Server-side rendering (SSR) enhances initial page load performance and can improve SEO (though less critical for an internal tool, it still contributes to a snappier user experience). The ability to build API routes directly within Next.js simplifies some backend integrations, especially for simpler data fetching.
  • Role: User interface, data visualization, interaction with backend APIs.
  • Styling: Tailwind CSS for utility-first, rapid UI development and consistent design system.

• Firebase Genkit (Backend / AI Orchestration):

  • Justification: A new open-source framework by Google for building AI-powered applications. Genkit acts as the central orchestrator for AI flows, providing a structured way to define, debug, and deploy AI services. It seamlessly integrates with Gemini API, handles prompt templating, model invocation, and observability. Its native integration with Google Cloud services like Cloud Functions or Cloud Run makes deployment straightforward. This significantly reduces the boilerplate code typically associated with building complex AI applications.
  • Role:
    • Defining AI "flows" for mandate matching, entity extraction, summarization.
    • Invoking Gemini API with structured prompts.
    • Serving as the primary backend API for the Next.js frontend.
    • Integrating with other Firebase/GCP services (Firestore for data, Cloud Functions for specific tasks).

• Gemini API (AI Model):

  • Justification: Google's latest family of highly capable, multimodal large language models. Gemini excels at complex reasoning, understanding nuanced language, summarization, entity extraction, and generating human-quality text. Its advanced capabilities are crucial for semantic mandate matching, detailed target profiling, and abstracting insights from unstructured web data. The API provides a scalable and performant way to access these models.
  • Role:
    • Mandate Matching: Semantic comparison of company profiles against investment mandates, generating match scores and explanations.
    • Entity Extraction: Identifying and extracting firmographic data (revenue, employee count, industry, location, key personnel) from unstructured text.
    • Summarization: Condensing lengthy company descriptions, news articles, or reports into concise summaries.
    • Classification: Categorizing companies into specific industry verticals or growth stages.
    • Sentiment Analysis: Identifying positive/negative signals from news and press releases.

• Google Cloud Platform (GCP) Services:

  • Firestore (NoSQL Database):
    • Justification: A flexible, scalable, and fully managed NoSQL document database. Its real-time synchronization capabilities are ideal for the deal flow dashboard, ensuring immediate updates. The document-based model is well-suited for storing dynamic company profiles and investment mandates, where schemas might evolve. Automatic scaling handles varying loads without operational overhead.
    • Role: Storing user-defined mandates, scraped raw data, normalized company profiles, match results, and user activity logs.
  • Cloud Functions / Cloud Run (Serverless Compute):
    • Justification:
      • Cloud Functions: Event-driven, fully managed serverless compute. Ideal for orchestrating individual scraping jobs, processing data after ingestion (e.g., triggered by new data in Cloud Storage), and hosting Genkit backend flows.
      • Cloud Run: Fully managed compute platform for deploying containerized applications. Offers more control than Cloud Functions and is suitable for more complex, long-running processes like the scraping workers themselves, which might require specific libraries or environments (e.g., headless browsers).
    • Role:
      • Hosting Genkit backend APIs.
      • Running individual web scraping tasks (Cloud Run for headless browser, Cloud Functions for simple API calls).
      • Data processing pipelines (e.g., parsing raw HTML).
      • Asynchronous background tasks (e.g., CRM export processing).
  • Cloud Storage (Object Storage):
    • Justification: Highly scalable, durable, and cost-effective object storage. Perfect for storing raw scraped data (HTML, JSON), processed data intermediates, and application logs before ingestion into Firestore or other processing.
    • Role: Storing raw web content, large datasets, backup of processed profiles.
  • Cloud Pub/Sub (Messaging Service):
    • Justification: A globally scalable, asynchronous messaging service. Decouples components, enabling robust, event-driven architectures.
    • Role: Orchestrating scraping jobs (e.g., a "scrape_url" topic), triggering data processing events, handling notifications.
  • Cloud Tasks (Managed Task Queue):
    • Justification: A fully managed service for dispatching asynchronous tasks. Provides reliable execution, retries, and rate limiting.
    • Role: Scheduling and managing long-running scraping tasks, ensuring delivery and execution even with transient failures. Ideal for managing a queue of URLs to scrape.
  • Firebase Authentication:
    • Justification: Fully managed authentication service. Simplifies user management, secure sign-in, and integration with other Firebase services.
    • Role: User authentication and authorization for the internal application.

4. Core Feature Implementation Guide

4.1 Web Scraping Orchestration

Objective: Systematically collect data from diverse web sources, handling scale, reliability, and anti-bot measures.

Pipeline Design:

1. Sources Definition (Firestore): Users define ScrapingSources (e.g., url_patterns, frequency, scraper_type). Examples: crunchbase.com/organization/*, techcrunch.com/*, specific industry directories.
2. Scheduler (Cloud Scheduler + Pub/Sub):
   • Cloud Scheduler triggers a scrape_start event (e.g., every 6 hours, daily).
   • A Cloud Function subscribes to scrape_start, reads ScrapingSources from Firestore, and publishes individual scrape_job messages to a Pub/Sub topic for each target URL or pattern.
3. Scraper Workers (Cloud Run):
   • A pool of Cloud Run services (e.g., scraper-headless-browser, scraper-api-proxy) subscribe to the scrape_job Pub/Sub topic.
   • Upon receiving a job (e.g., {"url": "...", "scraper_type": "headless", "job_id": "..."}), the worker:
     • Acquires a proxy from a managed pool (essential for avoiding IP bans).
     • Executes the scraping logic (e.g., Puppeteer/Playwright for headless, Cheerio/Axios for simple HTTP requests).
     • Applies rate limiting and backoff strategies.
     • Handles CAPTCHAs (potentially with third-party services or manual intervention for specific sites).
     • Stores raw HTML/JSON data in a specific bucket in Cloud Storage (raw-scraped-data/{source_id}/{timestamp}/{filename.html}).
     • Publishes a raw_data_ingested event to another Pub/Sub topic with the Cloud Storage path.
4. Data Parser & Normalizer (Cloud Function / Genkit Flow):
   • A Cloud Function or Genkit flow subscribes to raw_data_ingested.
   • Retrieves the raw data from Cloud Storage.
   • Uses a predefined parsing template (e.g., CSS selectors, regex, or more advanced ML for unstructured layouts) to extract key fields (company name, URL, description, industry, revenue, employee count, location).
   • Gemini Integration: For complex, unstructured text (e.g., company "About Us" pages, news articles), use Gemini for entity extraction and summarization.
     • Pseudo-code for a Genkit parsing step:

       import { defineFlow, generate } from '@genkit-ai/flow';
       import { geminiPro } from '@genkit-ai/vertexai';
       
       export const parseCompanyProfileFlow = defineFlow(
         { name: 'parseCompanyProfile', inputSchema: z.object({ rawHtml: z.string() }) },
         async (input) => {
           const prompt = `Extract the following details from the company's webpage content (HTML provided below) and return them as a JSON object. If a field is not found, use null.
           Fields: company_name, website_url, description, industry, revenue_range, employee_count, headquarters_address, recent_news_summary.
       
           HTML Content:
           ${input.rawHtml}
       
           JSON Output:`;
       
           const result = await generate({
             model: geminiPro,
             prompt: prompt,
             config: {
               output: { format: 'json' },
               temperature: 0.1, // Keep it deterministic for extraction
             },
           });
           return JSON.parse(result.text());
         }
       );
       

   • Stores the normalized, structured data in Firestore (companies collection). Includes source_urls, last_scraped_at, and a status field.
5. Deduplication (Batch Process / Triggered Function): Periodically or upon new ingestion, a Cloud Function/Run job runs to identify and merge duplicate company profiles using fuzzy matching on name, URL, and location.

4.2 Mandate Matching Logic

Objective: Intelligently match scraped company profiles against user-defined investment mandates using Gemini's semantic understanding.

Process:

1. Mandate Definition (Firestore): User defines a Mandate document:

   {
     "id": "mandate-123",
     "name": "SaaS Growth Equity Q3 2024",
     "description": "Seeking B2B SaaS companies with ARR > $10M, positive EBITDA, targeting healthcare/fintech sectors. Strong recurring revenue, high customer retention, and potential for international expansion. Preference for Series B/C stage.",
     "keywords": ["SaaS", "B2B", "healthcare tech", "fintech", "recurring revenue", "Series B", "Series C"],
     "revenue_min": 10000000,
     "ebitda_positive": true,
     "geo_focus": ["North America", "Europe"]
     // ... other structured criteria
   }
   

2. Matching Trigger:
   • When a new company profile is added/updated in Firestore.
   • When a mandate is created/updated.
   • On-demand by user for specific companies/mandates.
3. Genkit Matching Flow:
   • Pseudo-code for Genkit Mandate Matching Flow:

     import { defineFlow, generate } from '@genkit-ai/flow';
     import { geminiPro } from '@genkit-ai/vertexai';
     import * as z from 'zod';
     
     const CompanyProfileSchema = z.object({
       company_name: z.string(),
       description: z.string(),
       industry: z.string(),
       revenue: z.number().optional(),
       employee_count: z.number().optional(),
       // ... other fields
     });
     
     const MandateSchema = z.object({
       name: z.string(),
       description: z.string(), // Natural language mandate
       keywords: z.array(z.string()).optional(),
       revenue_min: z.number().optional(),
       ebitda_positive: z.boolean().optional(),
       // ... other structured criteria
     });
     
     export const mandateMatcherFlow = defineFlow(
       {
         name: 'mandateMatcher',
         inputSchema: z.object({
           company: CompanyProfileSchema,
           mandate: MandateSchema,
         }),
         outputSchema: z.object({
           match_score: z.number(), // 0-100
           explanation: z.string(),
           matched_criteria: z.array(z.string()),
           unmatched_criteria: z.array(z.string()),
           flagged_risks: z.array(z.string()).optional(),
         }),
       },
       async (input) => {
         const { company, mandate } = input;
     
         // Step 1: Combine structured and unstructured matching
         let structured_match_score = 0;
         const matched_criteria = [];
         const unmatched_criteria = [];
     
         if (mandate.revenue_min && company.revenue && company.revenue >= mandate.revenue_min) {
           structured_match_score += 20; // Example weighting
           matched_criteria.push(`Revenue (${company.revenue}) meets minimum (${mandate.revenue_min})`);
         } else if (mandate.revenue_min) {
           unmatched_criteria.push(`Revenue (${company.revenue || 'N/A'}) does not meet minimum (${mandate.revenue_min})`);
         }
         // ... add similar logic for other structured fields (employee_count, ebitda_positive, etc.)
     
         // Step 2: Gemini for semantic matching and explanation
         const prompt = `You are a Private Equity investment analyst. Evaluate the following company profile against the investment mandate.
         Provide a match score (0-100) and a detailed explanation, highlighting specific aspects of the company that match or don't match the mandate.
         Also, explicitly list criteria that were matched and criteria that were not matched, and identify any potential risks.
         Output your response as a JSON object.
     
         Investment Mandate:
         Name: ${mandate.name}
         Description: ${mandate.description}
         Keywords: ${mandate.keywords?.join(', ') || 'N/A'}
         ${mandate.revenue_min ? `Minimum Revenue: $${mandate.revenue_min.toLocaleString()}` : ''}
         ${mandate.ebitda_positive ? `Requires positive EBITDA.` : ''}
         // ... more mandate details
     
         Company Profile:
         Name: ${company.company_name}
         Description: ${company.description}
         Industry: ${company.industry}
         ${company.revenue ? `Revenue: $${company.revenue.toLocaleString()}` : ''}
         ${company.employee_count ? `Employees: ${company.employee_count}` : ''}
         // ... more company details
     
         JSON Output: {
           "match_score": <score>,
           "explanation": "<detailed explanation>",
           "matched_criteria_llm": ["<item>", ...],
           "unmatched_criteria_llm": ["<item>", ...],
           "flagged_risks": ["<risk>", ...]
         }`;
     
         const result = await generate({
           model: geminiPro,
           prompt: prompt,
           config: {
             output: { format: 'json' },
             temperature: 0.5, // Allow some creativity in explanation
           },
         });
     
         const llmOutput = JSON.parse(result.text());
     
         // Combine structured and LLM results
         const final_match_score = Math.min(100, llmOutput.match_score + structured_match_score); // Simple combination
         const final_matched_criteria = [...new Set([...matched_criteria, ...(llmOutput.matched_criteria_llm || [])])];
         const final_unmatched_criteria = [...new Set([...unmatched_criteria, ...(llmOutput.unmatched_criteria_llm || [])])];
     
         return {
           match_score: final_match_score,
           explanation: llmOutput.explanation,
           matched_criteria: final_matched_criteria,
           unmatched_criteria: final_unmatched_criteria,
           flagged_risks: llmOutput.flagged_risks,
         };
       }
     );
     

4. Result Storage: Store match results in Firestore in a mandate_matches subcollection under the mandate document or a top-level deal_matches collection, referencing both company_id and mandate_id.

4.3 Target Profiling

Objective: Consolidate and enrich company data into comprehensive, actionable profiles.

Process:

1. Data Aggregation: The Company document in Firestore aggregates data from all scraped sources.

   • company_name
   • description (potentially summarized by Gemini from multiple sources)
   • industry (categorized/standardized using Gemini)
   • website_url
   • headquarters_address
   • revenue_range
   • employee_count
   • key_executives (extracted names and titles)
   • funding_rounds (if found)
   • recent_news_summary (summarized by Gemini from recent articles)
   • competitors (identified by Gemini)
   • tech_stack (extracted from website/job postings)
   • growth_signals (e.g., "hiring rapidly", "new product launch", "market expansion")
   • potential_red_flags (e.g., "recent layoffs", "legal disputes", "negative news")
   • source_urls: Array of URLs where data was found.
   • last_updated_at: Timestamp.

2. Enrichment using Gemini (Genkit Flow): Triggered when a new company profile is created or updated.

   • Industry Classification:

     // ... within a Genkit flow
     const industryPrompt = `Classify the following company into a single, concise industry category (e.g., "Enterprise SaaS", "Biotechnology", "Fintech", "E-commerce").
     Company Description: ${company.description}
     Industry:`;
     const industryResult = await generate({ model: geminiPro, prompt: industryPrompt, config: { temperature: 0.1 } });
     company.industry = industryResult.text().trim();
     

   • Key Executive Extraction:

     // ... within a Genkit flow
     const execPrompt = `Extract the names and titles of key executives (CEO, CTO, CFO, VP-level) from the following text, and return as a JSON array of objects with "name" and "title" keys.
     Text: ${rawAboutUsPageContent}
     JSON Output:`;
     const execResult = await generate({ model: geminiPro, prompt: execPrompt, config: { output: { format: 'json' }, temperature: 0.1 } });
     company.key_executives = JSON.parse(execResult.text());
     

   • Growth Signals / Red Flags:

     // ... within a Genkit flow
     const analysisPrompt = `Analyze the following news articles and company description to identify potential growth signals and red flags.
     Return as a JSON object with two arrays: "growth_signals" and "red_flags".
     Text: ${company.description} \n\n Recent News: ${company.recent_news_summary}
     JSON Output:`;
     const analysisResult = await generate({ model: geminiPro, prompt: analysisPrompt, config: { output: { format: 'json' }, temperature: 0.5 } });
     const analysis = JSON.parse(analysisResult.text());
     company.growth_signals = analysis.growth_signals;
     company.potential_red_flags = analysis.red_flags;
     

4.4 CRM Export

Objective: Allow users to export qualified targets into their CRM system or a standardized format.

Implementation:

1. User Interface: A "Export to CRM" button on the target profile page or deal flow dashboard.
2. Export Options:
   • CSV/JSON Download: Simple, universal export. The Next.js frontend can trigger a Cloud Function that queries Firestore for selected targets, formats them, and returns a file for download.
     • Schema mapping: User can configure mapping between internal Company fields and their CRM's CSV/JSON schema.
   • Direct CRM Integration (Advanced):
     • For specific CRMs (e.g., Salesforce, HubSpot), implement dedicated API integrations using Cloud Functions.
     • Requires user to provide API keys/credentials (stored securely in Secret Manager).
     • Each CRM integration would be a separate Genkit flow or Cloud Function, handling authentication and API calls to create/update "Company" or "Opportunity" records.
     • Error handling for API limits, validation errors.

4.5 Deal Flow Dashboard

Objective: Provide an intuitive, real-time overview of identified targets, matching progress, and pipeline status.

Implementation (Next.js Frontend + Firestore Realtime):

1. Data Models:
   • Mandate: User-defined criteria.
   • Company: Scraped & enriched target profile.
   • DealMatch: Result of mandateMatcherFlow (links Company to Mandate, includes score, explanation, status: new, reviewed, qualified, rejected, exported).
2. Components:
   • Mandate List: Displays all active mandates, with quick stats (e.g., "50 new matches," "10 qualified targets").
   • Target List (Filtered by Mandate):
     • Table/card view of DealMatch entries for a selected mandate.
     • Columns: Company Name, Match Score, Industry, Revenue, Last Scraped, Status.
     • Filtering: By score range, industry, status, keywords.
     • Sorting: By score, date, name.
     • Search functionality.
   • Target Detail View:
     • Comprehensive view of a Company profile.
     • DealMatch details: Score, explanation, matched/unmatched criteria.
     • User actions: "Qualify," "Reject," "Add to CRM," "Request More Info" (triggering re-scrape or deeper AI analysis).
     • History of interactions/status changes.
   • Analytics & Visualizations:
     • Charts showing distribution of matched companies by industry, revenue band, geography.
     • Match rate over time.
     • Conversion rates (new -> qualified -> exported).
     • Using charting libraries (e.g., Recharts, Chart.js) integrated with Next.js.
3. Real-time Updates:
   • Firestore's real-time listeners are leveraged in Next.js components to automatically update the dashboard when new matches are found, statuses change, or data is enriched.

5. Gemini Prompting Strategy

Effective prompting is crucial for leveraging Gemini's full potential. The strategy focuses on clear instructions, structured output, and context provision.

General Principles:

1. Role-Playing / Persona: Assign Gemini a persona (e.g., "You are a Private Equity investment analyst," "You are an expert data extractor").
2. Clear Task Definition: Explicitly state what to do (e.g., "Extract...", "Summarize...", "Match...").
3. Structured Output: Always request JSON output where structured data is needed. Provide a clear schema if possible. This makes programmatic parsing reliable.
4. Few-shot / Zero-shot:
   • Zero-shot: For general tasks like summarization or basic entity extraction, where Gemini has inherent knowledge.
   • Few-shot: For domain-specific tasks or when consistency is paramount (e.g., classifying into specific industry taxonomies, judging match relevance). Provide 1-3 examples of input/output pairs.
5. Contextualization: Provide all necessary context in the prompt (e.g., the full company description, the entire mandate text).
6. Iterative Prompting / Chaining: For complex workflows, break down tasks into smaller, manageable steps, each potentially being a separate Gemini call (e.g., first extract entities, then summarize, then match). Genkit flows are excellent for orchestrating this.
7. Temperature Control:
   • temperature=0.0 or 0.1: For factual extraction, classification, or deterministic tasks where creativity is undesirable.
   • temperature=0.5 or 0.7: For summarization, explanation generation, or tasks where some creativity/nuance is acceptable.
8. Output Length/Token Limits: Be mindful of token limits. For very long documents, consider pre-summarization or chunking.

Specific Prompt Examples (Illustrative):

• Prompt for Industry Classification (Zero-Shot):

  "You are an expert industry analyst. Classify the following company into ONE specific, commonly recognized industry category (e.g., 'Enterprise SaaS', 'Biotechnology', 'Fintech', 'E-commerce', 'Industrial Manufacturing', 'Healthcare Services'). Be concise.
  
  Company Description:
  Acme Innovations develops AI-powered software for optimizing supply chain logistics and inventory management for large retailers. Their platform uses predictive analytics to reduce waste and improve delivery times.
  
  Industry Category:"
  -> "Enterprise SaaS"
  

• Prompt for Executive Extraction (Structured Output):

  "Extract the names and titles of key executives (CEO, CTO, CFO, President, VP-level) from the following company 'About Us' page content. Return the results as a JSON array of objects, where each object has 'name' and 'title' keys. If no executives are found, return an empty array.
  
  Text:
  [Extremely long "About Us" page content with executive bios]
  
  JSON Output:"
  -> `[{"name": "Jane Doe", "title": "CEO"}, {"name": "John Smith", "title": "CTO"}]`
  

• Prompt for Mandate Matching with Explanation (as seen in Section 4.2): This combines structured output with descriptive text. The key is to instruct Gemini to explain its reasoning, which is invaluable for user trust and refinement.

• Prompt for Growth Signals & Red Flags (Iterative): (Initial call to summarize recent news)

  "Summarize the key positive and negative developments from the following news articles about a company. Focus on impacts related to growth, market position, and operational health.
  
  News Articles:
  [Concatenated text of recent news articles]
  
  Summary:"
  -> "[Concise summary of news]"
  
  *(Second call using the summary)*
  "Based on the following company description and news summary, identify specific 'growth signals' (e.g., new product launch, market expansion, key hires) and 'potential red flags' (e.g., layoffs, legal issues, declining revenue trends). Return as a JSON object with two arrays: 'growth_signals' and 'red_flags'.
  
  Company Description: [company.description]
  News Summary: [summary from previous call]
  
  JSON Output:"
  

Genkit Integration: Genkit simplifies this by allowing you to define generate calls with specific models, prompts (which can be dynamic templates), and config options like output: { format: 'json' } and temperature.

6. Deployment & Scaling

Leveraging Google Cloud Platform (GCP) provides robust, scalable, and secure infrastructure for the Private Equity Deal Sourcer.

Deployment Strategy:

1. Frontend (Next.js):

   • Hosting: Deploy the Next.js application to Cloud Run. This allows for server-side rendering (SSR) and API routes to scale automatically with demand. Cloud Run runs Docker containers, providing flexibility for custom build steps.
   • CDN: Integrate Cloud CDN for caching static assets and serving the UI globally with low latency.
   • CI/CD: Use Cloud Build to automatically build the Docker image and deploy to Cloud Run upon Git pushes to the main branch.

2. Backend (Firebase Genkit / Cloud Functions / Cloud Run):

   • Genkit Flows: Genkit flows (mandateMatcherFlow, parseCompanyProfileFlow, etc.) are typically deployed as Cloud Functions. Each flow can be a distinct HTTP-triggered or Pub/Sub-triggered function. This provides auto-scaling and a pay-per-execution model.
   • Scraper Workers: Deploy the scraper logic (e.g., Puppeteer/Playwright scripts) as Cloud Run services. Cloud Run is ideal for containerized headless browser environments, providing higher resource limits and longer execution times compared to Cloud Functions, necessary for potentially complex scraping tasks.
   • Scheduled Tasks: Use Cloud Scheduler to trigger Pub/Sub topics, which then invoke Cloud Functions (e.g., for starting daily scraping runs, nightly data normalization batches).

3. Data Storage:

   • Firestore: Fully managed, scales automatically. No explicit deployment steps beyond creating the database and configuring security rules.
   • Cloud Storage: Raw scraped data and logs stored in designated buckets. Managed service, scales infinitely.

4. Messaging & Task Queues:

   • Cloud Pub/Sub & Cloud Tasks: Managed services. Configuration involves creating topics/queues and setting up subscriptions/task handlers.

Scaling Considerations:

1. Web Scraping:

   • Horizontal Scaling: Cloud Run services for scrapers will automatically scale up/down based on the number of scrape_job messages in Pub/Sub. Configure appropriate min-instances and max-instances for Cloud Run.
   • Proxy Management: Integrate with a robust proxy provider or build an internal proxy rotation system using Cloud Load Balancing and a pool of proxy servers (e.g., on Compute Engine) to avoid IP bans.
   • Rate Limiting & Backoff: Implement intelligent rate limiting and exponential backoff in scraper workers to respect target website policies and avoid overloading them.
   • Error Handling: Extensive retry mechanisms for transient failures (network issues, temporary blocks) using Cloud Tasks for guaranteed delivery and retries.

2. AI Inference (Gemini API):

   • API Limits: Be aware of Gemini API rate limits. For high-volume requests, batching inputs where possible (e.g., asking Gemini to process multiple company descriptions for entity extraction in one call) can be efficient.
   • Caching: Cache Gemini results for frequently requested or static data (e.g., common industry classifications) in Firestore or Memorystore (Redis) to reduce API calls and latency.
   • Asynchronous Processing: Mandate matching and deep profiling should primarily happen asynchronously via Pub/Sub triggers to avoid blocking user requests.

3. Database (Firestore):

   • Firestore scales automatically, but schema design and query optimization are crucial. Ensure proper indexing to avoid slow queries, especially as the number of companies and mandates grows. Avoid anti-patterns like hot documents.

4. Frontend/Backend:

   • Cloud Run/Cloud Functions inherently scale with traffic. Ensure memory and CPU configurations are sufficient for the Genkit flows.
   • Next.js API routes will also benefit from Cloud Run's auto-scaling.

Monitoring & Logging:

• Cloud Logging: Centralized logging for all application components (Next.js, Genkit, Cloud Functions, Cloud Run). Use structured logging for easier filtering and analysis.
• Cloud Monitoring:
  • Set up dashboards to monitor key metrics: Cloud Run instance count, CPU utilization, request latency, error rates for APIs and scraping jobs.
  • Monitor Pub/Sub queue sizes (backlog) to identify processing bottlenecks.
  • Create custom metrics for business logic (e.g., "new targets identified," "matches generated per day").
  • Configure alerts for critical errors, performance degradation, or scraping failures (e.g., IP bans).
• Genkit Observability: Genkit provides built-in tracing and logging for AI flows, which integrates directly with Cloud Trace and Cloud Logging, allowing deep introspection into prompt execution, model responses, and tool calls.

Security:

• Authentication: Firebase Authentication for user access to the Next.js frontend.
• Authorization: Implement granular IAM roles for service accounts (e.g., scraper-worker only has access to Cloud Storage write, genkit-backend has access to Firestore read/write and Gemini API).
• API Keys: Store Gemini API keys and any third-party API keys (e.g., for proxies) securely in Google Cloud Secret Manager. Do not hardcode them.
• Network Security: Utilize VPC Service Controls for highly sensitive data to create a perimeter around GCP resources, further restricting data exfiltration.
• Data Encryption: All data at rest in Firestore and Cloud Storage is encrypted by default. Data in transit is encrypted using TLS.
• Access Control: Define clear roles and permissions for users within the application (e.g., "Admin," "Analyst," "Read-Only").