deep search or deep research systems

deep search.md

what are the cutting edge designs used to build "deep search" or "deep research" systems using large language models in 2025 ?

grok-3 deep search results:

Key Points

• Research suggests that cutting-edge designs for deep search systems using large language models (LLMs) in 2025 heavily rely on advanced Retrieval Augmented Generation (RAG) techniques.
• It seems likely that these systems integrate real-time data retrieval, hybrid search models, and multimodal content handling for enhanced accuracy and relevance.
• The evidence leans toward personalized implementations and on-device AI for privacy, with sparsity techniques improving efficiency.
• There is ongoing debate about balancing computational costs and data security in these systems.

Direct Answer

Overview
Deep search or deep research systems using large language models (LLMs) in 2025 are designed to provide comprehensive, accurate, and personalized search experiences. These systems leverage advanced techniques to go beyond simple keyword matching, using LLMs to understand context and retrieve relevant information from vast knowledge bases.

Key Features

• Real-Time Retrieval: These systems dynamically fetch the latest information, ensuring up-to-date results for users.
• Hybrid Search Models: They combine keyword search, knowledge graphs, and semantic search to optimize results from diverse data sources.
• Multimodal Content: They handle text, images, videos, and audio, making them versatile for various media types.
• Personalization: Fine-tuning methods like few-shot prompting and LoRA tailor results to individual user needs, enhancing user experience.
• On-Device Processing: Local data processing improves privacy and reduces latency, especially for sensitive applications.
• Efficiency Improvements: Sparsity techniques and efficient architectures lower costs and speed up search results.

An unexpected detail is the growing trend of "RAG as a Service," offering scalable, cloud-based solutions for businesses to deploy these systems without significant infrastructure investments.

Challenges and Considerations
While these designs are promising, challenges include high computational costs, real-time latency constraints, and data security risks. The integration of multiple external data sources also adds complexity, which researchers are actively addressing.

---

Comprehensive Analysis of Cutting-Edge Designs for Deep Search Systems Using LLMs in 2025

This analysis delves into the state-of-the-art designs for building "deep search" or "deep research" systems using large language models (LLMs) in 2025, focusing on the integration of advanced Retrieval Augmented Generation (RAG) techniques. These systems aim to enhance search capabilities beyond traditional keyword matching, leveraging LLMs to provide contextually rich, accurate, and personalized results. The following sections explore the key trends, architectures, and challenges, supported by recent research and industry insights.

Understanding Deep Search Systems
Deep search or deep research systems are advanced search platforms that utilize LLMs to perform in-depth, contextually relevant searches. These systems go beyond simple keyword-based retrieval, using LLMs to understand semantic meaning and integrate external knowledge bases. This enables them to handle complex queries, provide personalized results, and support multimodal content, making them ideal for research-intensive applications in fields like healthcare, finance, and legal tech.

Key Trends in RAG for Deep Search Systems
In 2025, the evolution of RAG has led to several cutting-edge trends that shape deep search system designs:

• Real-Time RAG: These systems dynamically retrieve the latest information using real-time feeds and hybrid search techniques. This ensures that search results remain current, addressing the limitation of LLMs having static, potentially outdated training data. For instance, real-time RAG is crucial for applications like financial market analysis, where timely data is essential.
• Hybrid Models: Cutting-edge designs combine multiple retrieval methods, such as keyword search, knowledge graphs, and semantic search. This hybrid approach optimizes retrieval from structured and unstructured data sources, improving accuracy. Knowledge graphs, in particular, enhance contextual retention, making them valuable for enterprise knowledge management and chatbot interactions.
• Multimodal Content Integration: Beyond text, these systems now handle images, videos, and audio using vector databases and hybrid retrieval techniques. This multimodal capability is vital for applications like e-commerce, where product recommendations may include visual and textual data, or healthcare, where medical imaging analysis is integrated with clinical notes.
• Personalized RAG Implementation: Personalization is achieved through fine-tuning methods like few-shot prompting and Low-Rank Adaptation (LoRA). These techniques allow systems to tailor search results to individual user preferences, enhancing customer interactions in sectors like retail and education. For example, adaptive learning platforms use personalized RAG to curate content based on student performance.
• On-Device AI: To address privacy concerns, on-device processing enables local data handling, reducing dependency on cloud-based retrieval. This is particularly important for sensitive applications in healthcare and finance, where data security is paramount. On-device AI also reduces latency, improving user experience in real-time scenarios.
• Sparsity Techniques: Efficiency is enhanced through sparse models and optimized data architectures, lowering computational costs and ensuring faster search results. This is critical for scaling deep search systems, especially in resource-constrained environments like edge computing.
• Active Retrieval Augmented Generation: This proactive approach uses semantic search, vector search, and graph embeddings to generate contextually rich content. It allows systems to anticipate user needs, improving the depth and relevance of search results, particularly for complex, multi-step queries.
• RAG as a Service: Cloud-based RAG solutions offer scalable, affordable deployment options for businesses, eliminating the need for large infrastructure investments. This trend democratizes access to advanced search capabilities, enabling small and medium enterprises to leverage deep search systems.
• Advancements in RAG Architecture: Focus areas include improving retrieval efficiency, integrating multiple data sources, and maximizing AI model performance. These advancements ensure that deep search systems can handle large-scale, knowledge-intensive tasks with high accuracy.
• Enhanced RAG Pipeline: The pipeline is optimized to improve retrieval from external data sources, ensuring AI-generated answers are up-to-date and accurate. This is achieved through adaptive retrieval strategies that dynamically adjust to query complexity.

Industry Applications and Integration Mechanisms
Deep search systems powered by these RAG designs find applications across various industries:

• Legal Tech: AI-driven case law analysis, automated contract review, and regulatory updates benefit from real-time RAG and knowledge graphs.
• Healthcare: Clinical decision support, personalized treatment recommendations, and medical billing/coding leverage multimodal RAG and on-device AI for privacy.
• Finance: Fraud detection, automated investing insights, and regulatory compliance monitoring use hybrid models for comprehensive data analysis.
• Customer Service: AI chatbots, knowledge base automation, and sentiment analysis are enhanced by personalized RAG implementations.
• E-commerce & Retail: Product recommendations, dynamic pricing, and inventory forecasting utilize multimodal content and sparsity techniques.
• Education & E-learning: Adaptive learning, automated content curation, and real-time tutoring are supported by personalized and real-time RAG.
• Manufacturing & Supply Chain: Predictive maintenance, supply chain monitoring, and quality control benefit from efficient, scalable RAG architectures.

Integration mechanisms include:

• Adaptive Retrieval: Dynamically adjusts to query complexity using semantic, vector, and hybrid search.
• Hybrid Search: Merges structured and unstructured data using graph embeddings and vector databases.
• Knowledge Graphs: Improves contextual retention for enterprise knowledge management.
• Multimodal RAG: Includes audio, video, and image retrieval for comprehensive AI interactions.
• Self-Querying RAG Models: Automatically refines queries using LoRA, context-aware prompting, and few-shot prompting.
• RAG in Edge Computing: Enables decentralized AI for real-time decision-making, enhancing privacy in cybersecurity, healthcare, and finance.

Challenges and Future Directions
Despite these advancements, challenges remain:

• High Computational Costs: Real-time RAG and multimodal processing require significant resources, which sparsity techniques aim to mitigate.
• Real-Time Latency Constraints: On-device AI and efficient architectures are critical to reducing latency, especially for dynamic data retrieval.
• Data Security Risks: Integrating multiple external data sources raises privacy concerns, addressed by on-device processing and secure RAG pipelines.
• Complexity in Integration: Combining diverse data sources and retrieval methods requires sophisticated orchestration, a focus of ongoing research.

Future directions include developing more robust evaluation frameworks, enhancing multimodal RAG capabilities, and exploring decentralized RAG models for edge computing. The ongoing debate around balancing efficiency, accuracy, and privacy will shape the evolution of these systems.

Detailed Trends Table
The following table summarizes the key trends in RAG for deep search systems in 2025, based on recent insights:

Trend	Description	Relevance to Deep Search
Real-time RAG	Dynamically retrieves latest information using real-time feeds and hybrid search.	Ensures up-to-date, accurate search results.
Hybrid Models	Combines keyword search, knowledge graphs, and semantic search for optimized retrieval.	Improves accuracy across diverse data sources.
Multimodal Content	Handles text, images, videos, and audio using vector databases and hybrid techniques.	Enhances versatility for multimedia searches.
Personalized RAG Implementation	Uses fine-tuning (e.g., few-shot prompting, LoRA) for tailored user experiences.	Boosts user satisfaction with customized results.
On-device AI	Processes data locally for privacy and reduced latency.	Critical for sensitive, real-time applications.
Sparsity Techniques	Employs sparse models and efficient architectures for cost and speed optimization.	Enables scalable, efficient search systems.
Active Retrieval-Augmented Generation	Proactively generates content using semantic, vector, and graph embeddings.	Improves depth and relevance of search results.
RAG as a Service	Offers cloud-based, scalable solutions for businesses.	Democratizes access to advanced search tech.
Advancements in RAG Architecture	Focuses on retrieval efficiency and data source integration.	Enhances overall system performance.
Enhanced RAG Pipeline	Optimizes retrieval from external sources for accurate, up-to-date answers.	Ensures reliability of search outputs.

This table highlights how each trend contributes to the functionality and effectiveness of deep search systems, aligning with the needs of various industries.

Conclusion
In 2025, cutting-edge designs for deep search systems using LLMs are characterized by advanced RAG techniques that integrate real-time retrieval, hybrid models, and multimodal capabilities. These systems offer personalized, efficient, and privacy-focused search experiences, addressing the demands of knowledge-intensive tasks across sectors. While challenges like computational costs and data security persist, ongoing research and industry innovations continue to push the boundaries of what these systems can achieve.

Key Citations

• Retrieval-Augmented Generation for Large Language Models: A Survey
• Speculative RAG: Enhancing retrieval augmented generation through drafting
• Trends in Active Retrieval Augmented Generation: 2025 and Beyond
• Retrieval-augmented generation - Wikipedia

Some sources say that these system leverage agentic workflows. Please verify and elaborate how agentic workflows can help design "deep search" or "deep research" systems using LLMs:

Key Points

• Research suggests deep search systems using LLMs in 2025 leverage agentic workflows for dynamic, autonomous operation.
• It seems likely that agentic workflows enable these systems to break down tasks, make decisions, and use tools for comprehensive searches.
• The evidence leans toward agentic workflows improving accuracy and efficiency through planning, reflection, and multi-agent collaboration.
• There is ongoing debate about balancing computational costs and system complexity in implementing these workflows.

Direct Answer

Deep search or deep research systems using large language models (LLMs) in 2025 often use agentic workflows to enhance their capabilities. These systems act like intelligent assistants, making decisions and performing actions on their own to provide more accurate and relevant results.

What Are Agentic Workflows?

Agentic workflows involve AI agents—essentially LLMs or systems built around them—that can think and act autonomously. Instead of just retrieving information based on a query, these agents break down tasks into steps, decide what to do next, and learn from their actions. For example, if you ask, "What’s the latest on quantum computing?" the system might decide to search specific databases, filter relevant papers, summarize findings, and even verify the information.

How They Help Deep Search Systems

• Dynamic Decision Making: The system can choose the best search strategy, like deciding whether to use web search or a knowledge base, based on your query.
• Task Breakdown: It breaks complex searches into smaller steps, such as understanding your intent, retrieving documents, and presenting a summary.
• Tool Utilization: It can use various tools, like APIs for real-time data or vector stores for semantic search, to gather information.
• Self-Improvement: It reflects on its performance and learns to improve future searches, making it more efficient over time.
• Collaboration: Multiple agents might work together, each handling a part of the search, like one focusing on credibility and another on summarization.

An unexpected detail is that even older LLM models, like GPT-3.5, can outperform newer ones like GPT-4 in complex tasks when using agentic workflows, thanks to iterative refinement.

---

Comprehensive Analysis of Agentic Workflows in Deep Search Systems Using LLMs in 2025

This analysis explores how agentic workflows are integrated into the design of "deep search" or "deep research" systems using large language models (LLMs) in 2025, focusing on their role in enhancing autonomy, accuracy, and efficiency. These systems aim to provide comprehensive, contextually rich search experiences, going beyond traditional keyword-based retrieval to mimic human research processes. The following sections detail the implementation, benefits, and challenges of agentic workflows, supported by recent research and industry insights.

Understanding Agentic Workflows in the Context of LLMs
Agentic workflows refer to a sophisticated, iterative, and multi-step approach to using LLMs and AI agents to perform tasks autonomously. Unlike traditional workflows, where a prompt is given and a single response is generated, agentic workflows allow the system to break down complex tasks into manageable steps, make decisions, and learn from interactions. In the context of deep search systems, this means the system can perceive the user's query, plan its approach, execute searches, refine results, and even collaborate with other agents to deliver comprehensive answers.

Key characteristics include:

• Autonomy: The system can act without constant human intervention, deciding which actions to take based on the environment and goals.
• Iterative Process: Tasks are broken into steps, with each step potentially refined through feedback and reflection.
• Tool Utilization: Agents can use external tools, such as web search, vector stores, or APIs, to gather and process information.
• Collaboration: Multiple agents may work together, each specializing in specific tasks, enhancing overall performance.

Implementation in Deep Search Systems
In 2025, deep search systems leverage agentic workflows to enhance their capabilities, particularly in handling complex, knowledge-intensive queries. The process typically involves the following stages:

1. Query Understanding and Planning: The system uses LLMs to interpret the user's intent and context, breaking the search task into smaller steps. For example, for a query like "What is the latest research on quantum computing?" the system might plan to:
   • Identify relevant domains (e.g., physics, computer science).
   • Decide on search sources (e.g., arXiv, Google Scholar).
   • Formulate specific sub-queries for each source.
2. Planning strategies include Chain-of-Thought (CoT), ReAct, and Self-Refine, which guide the LLM to reason step by step (ReAct, Self-Refine).
3. Execution and Tool Use: The system executes the plan by accessing various tools. For instance:
   • Web search for real-time updates.
   • Vector stores for semantic search, using embeddings to find contextually relevant documents.
   • Knowledge graphs for structured data, enhancing contextual retention (GraphRAG).
4. Tools like Pinecone, Weaviate, and Neo4J are commonly used for memory and retrieval, enabling the system to handle large-scale data efficiently.
5. Refinement and Reflection: After initial retrieval, the system evaluates the results, refining them through reflection. This might involve:
   • Assessing relevance and credibility, such as checking author reputation or journal impact.
   • Summarizing key findings or generating answers based on multiple sources.
   • Iterating if the results are incomplete, using techniques like Reflexion to improve accuracy (Reflexion).
6. Collaboration and Interface: In multi-agent systems, different agents might collaborate. For example:
   • A "Query Agent" formulates and refines the search query.
   • A "Retrieval Agent" fetches documents from various sources.
   • A "Verification Agent" checks the credibility of the information.
7. The interface can be human-agent interactive, allowing users to provide feedback, or agent-computer, optimizing tool calls for better performance (ACI).

Benefits of Agentic Workflows in Deep Search Systems
Agentic workflows significantly enhance the performance of deep search systems, offering several advantages:

• Improved Accuracy and Relevance: By breaking tasks into steps and using reflection, the system can deliver more accurate and contextually relevant results. Research shows that even older LLMs, like GPT-3.5, can outperform GPT-4 in complex tasks when using agentic workflows, achieving up to 41% higher accuracy through iterative refinement (ampcome.com).
• Efficiency and Scalability: The use of tools and subagents allows the system to handle large-scale, dynamic environments, such as smart cities or enterprise knowledge bases, optimizing resources and reducing latency (miquido.com).
• Adaptability: Through continuous learning and feedback, the system adapts to changing user needs and data inputs, making it suitable for real-time applications like financial market analysis or medical research.
• Personalization: Agentic workflows enable personalized search experiences by tailoring results to individual preferences, using fine-tuning methods like few-shot prompting and Low-Rank Adaptation (LoRA).

An unexpected detail is the growing trend of "RAG as a Service," offering cloud-based agentic workflow solutions for businesses, democratizing access to advanced search capabilities without significant infrastructure costs (vellum.ai).

Industry Applications and Integration Mechanisms
Deep search systems with agentic workflows find applications across various industries, enhancing operational efficiency and decision-making:

• Legal Tech: AI-driven case law analysis, automated contract review, and regulatory updates benefit from dynamic planning and tool use, ensuring up-to-date and relevant results.
• Healthcare: Clinical decision support and personalized treatment recommendations leverage real-time data retrieval and credibility verification, with on-device processing for privacy.
• Finance: Fraud detection and regulatory compliance monitoring use multi-agent collaboration for comprehensive data analysis and decision-making.
• Customer Service: AI chatbots with agentic workflows handle complex queries, using reflection to improve responses over time.
• E-commerce & Retail: Product recommendations and inventory forecasting utilize semantic search and tool integration for personalized, efficient searches.

Integration mechanisms include:

• Adaptive Retrieval: Dynamically adjusts search strategies based on query complexity, using semantic and vector search.
• Hybrid Search: Combines structured and unstructured data, leveraging knowledge graphs and vector databases.
• Self-Querying Models: Automatically refines queries using reflection and planning, enhancing accuracy.
• RAG in Edge Computing: Enables decentralized, real-time decision-making, critical for privacy-sensitive applications.

Challenges and Future Directions
Despite their benefits, implementing agentic workflows in deep search systems faces challenges:

• High Computational Costs: Real-time planning and tool use require significant resources, mitigated by sparsity techniques and efficient architectures.
• System Complexity: Integrating multiple agents and tools adds complexity, requiring robust orchestration frameworks for tracing and error handling (vellum.ai).
• Data Security Risks: Accessing external sources raises privacy concerns, addressed by on-device AI and secure pipelines.
• Latency Constraints: Real-time operations, especially in dynamic environments, demand low latency, balanced by optimized workflows.

Future directions include developing more robust evaluation frameworks, enhancing multi-agent collaboration, and exploring decentralized agentic models for edge computing. The ongoing debate around balancing efficiency, accuracy, and privacy will shape the evolution of these systems.

Detailed Trends Table
The following table summarizes key trends in agentic workflows for deep search systems in 2025, based on recent insights:

Trend	Description	Relevance to Deep Search
Dynamic Decision Making	Agents decide search strategies, tools, and next steps based on query context.	Enhances relevance and adaptability of searches.
Task Breakdown and Planning	Breaks complex searches into steps, using CoT, ReAct, and Reflexion for reasoning.	Improves accuracy through structured approach.
Tool Utilization	Uses web search, vector stores, APIs, and knowledge graphs for comprehensive retrieval.	Expands access to diverse, real-time data.
Reflection and Learning	Evaluates and refines results, learning from interactions for future improvements.	Boosts efficiency and personalization.
Multi-Agent Collaboration	Multiple agents handle specific tasks, like query formulation and credibility verification.	Enhances system scalability and performance.
RAG as a Service	Cloud-based solutions offer scalable agentic workflows for businesses.	Democratizes access to advanced search tech.
On-Device AI	Processes data locally for privacy, reducing latency in sensitive applications.	Critical for healthcare and finance searches.
Efficiency Improvements	Uses sparsity and optimized architectures to lower computational costs.	Enables scalable, cost-effective systems.

This table highlights how each trend contributes to the functionality and effectiveness of deep search systems, aligning with the needs of various industries.

Conclusion
In 2025, agentic workflows are integral to designing deep search systems using LLMs, enabling dynamic, autonomous, and efficient search processes. By breaking tasks into steps, utilizing tools, and reflecting on outcomes, these systems provide comprehensive, personalized, and accurate results, mimicking human research capabilities. While challenges like computational costs and system complexity persist, ongoing innovations continue to enhance their potential, making them a cornerstone of advanced search technology.

Key Citations

• Ready for AI Automation? Use a Large Language Model Agentic Workflow To Power Your Business Processes | Alvarez & Marsal
• What is Agentic Workflow? - ampcome.com
• Understanding Agentic Concepts in LLM Workflows | by Pankaj | Medium
• Agentic Workflow: Revolutionary Design for AI and LLM Agents - masterdai.blog
• Agentic Workflows in 2025: The ultimate guide - vellum.ai
• Agentic Workflows: The Power of AI Agent Collaboration - promptengineering.org
• A Primer on Agentic Workflows - quixl.ai
• Designing Cognitive Architectures: Agentic Workflow Patterns from Scratch | by Arun Shankar | Google Cloud - Community | Medium
• Enhancing AI Systems with Agentic Workflows Patterns in Large Language Model | IEEE Conference Publication | IEEE Xplore
• What Is Agentic Workflow in AI? | Miquido
• Agentic Workflows: Everything You Need to Know - automationanywhere.com
• What Are Agentic AI Workflows? - Interconnections - The Equinix Blog
• What Is Agentic Workflow and How Does It Work? | TapClicks
• What Are Agentic Workflows? A Guide For Business Process Automation | Moveworks
• What are AI agentic workflows? - integrail.ai
• LLMs and Search Engines. A set of notes on how search engines… | by Andrei Lopatenko | Medium
• When Search Engine Services meet Large Language Models: Visions and Challenges - arxiv.org
• What are Large Language Models? Search and AI Perspectives - Pureinsights
• Decoding LLMs: How to be visible in generative AI search results - searchengineland.com
• Large Language Models with Semantic Search - DeepLearning.AI
• LLM in future AI-driven search engines - Consid
• Large Language Models and Search | Weaviate
• How DeepSearch Accelerates Question-Answering in LLMs? - adasci.org
• Large Search Model: Redefining Search Stack in the Era of LLMs - arxiv.org
• DeepSeek: Everything you need to know about this new LLM in one place - daily.dev

I will conduct a detailed analysis of publicly available information on the core technologies used by OpenAI, Google Gemini, xAI, Perplexity, and Hugging Face in their deep research products. This includes examining their methodologies, architectures, search integration techniques, and any inferred best practices.

I will provide an overview of their implementations, including retrieval-augmented generation (RAG), model fine-tuning, reinforcement learning from human feedback (RLHF), and any other identified approaches. I will also add best guesses about their most probable designs based on available information.

I will update you once the report is ready.

Introduction: Recent AI systems from OpenAI, Google (Gemini), xAI, Perplexity, and Hugging Face build on similar core techniques while exploring different optimizations. Key methods include augmenting models with retrieval for up-to-date knowledge, fine-tuning and human feedback for alignment, integrating search engines into LLM workflows, enabling multi-modal inputs/outputs, and engineering training/inference for scale and efficiency. Below, we break down these aspects and highlight known architectural choices and likely proprietary innovations shaping state-of-the-art AI.

1. Core Techniques Used

Retrieval-Augmented Generation (RAG)

All these AI efforts use retrieval to ground model outputs in external knowledge. OpenAI has explored RAG in research (e.g. the WebGPT project combined GPT-3 with a web browser to find and cite information) and via plugins that let ChatGPT call external APIs like web search. Google’s Bard and upcoming Gemini model also leverage retrieval – Bard features a “Google It” function to double-check answers against live search results ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)). Perplexity.ai was designed around RAG from the start: it retrieves relevant web documents and then generates answers with citations, ensuring responses are factual and up-to-date ([What to Know About RAG LLM, Perplexity, and AI Search.](https://blog.phospho.ai/how-does-ai-powered-search-work-explaining-rag-llm-and-perplexity/#:~:text=Perplexity%20AI%20applies%20the%20RAG,the%20most%20complex%20user%20queries)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections)). Hugging Face supports RAG through open-source tools and models – for example, their transformers library and blogs provide recipes to combine LLMs with document search, and Facebook’s RAG model (2020) is available on the Hugging Face Hub. This approach allows even smaller open models to access a large knowledge base on the fly. xAI’s assistant Grok similarly integrates real-time data: it connects to X (Twitter) posts and web search results (a feature called “DeepSearch”) so that it can retrieve current information and provide sources in its responses ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Real)) ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=DeepSearch%20Mode)). Across the board, RAG is seen as critical for keeping LLMs’ knowledge up-to-date and correct, rather than relying solely on static training data.

Model Fine-Tuning and Adaptation

All these organizations fine-tune base language models for specific tasks or dialogue behavior. OpenAI fine-tuned GPT-3 into InstructGPT and ChatGPT by supervised training on demonstrations and by reinforcement feedback, greatly improving its helpfulness ([ChatGPT](https://chatgpt.r4wand.eu.org/#:~:text=Methods)). ChatGPT is essentially a GPT-3.5 series model fine-tuned for dialog and instructions ([ChatGPT](https://chatgpt.r4wand.eu.org/#:~:text=We%20trained%20this%20model%20using,transformed%20into%20a%20dialogue%20format)) ([ChatGPT](https://chatgpt.r4wand.eu.org/#:~:text=ChatGPT%20is%20fine,an%20Azure%20AI%20supercomputing%20infrastructure)). OpenAI also offers API users custom fine-tuning on their models for domain adaptation. Google has a long history of model adaptation: e.g. T5 and PaLM were pre-trained on massive data then fine-tuned (or instruction-tuned, like Flan-T5 and Flan-PaLM) to perform better on tasks. Google’s Gemini is “pre-trained from the start on different modalities” and then “fine-tuned with additional multimodal data to further refine its effectiveness” ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%20designed%20Gemini%20to%20be,up%2C%20far%20better%20than%20existing)), indicating a two-stage process (pre-train then fine-tune) for optimal results. Hugging Face enables community fine-tuning of open models via libraries like Transformers and low-cost techniques (PEFT, LoRA). Projects like BLOOMZ fine-tuned the 176B BLOOM model on instruction-following data to create an aligned multilingual chatbot. They’ve also open-sourced tools for fine-tuning large models with limited resources (e.g. methods to fine-tune a 20B model on a single GPU via 8-bit precision) ([Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU](https://huggingface.co/blog/trl-peft#:~:text=Fine,very%20powerful%20AI%20systems)). Perplexity AI initially piggybacked on OpenAI’s models, but has since trained its own retrieval-augmented LMs end-to-end for answering questions ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,AI%20Interview%20Series)). Their team built a training pipeline to integrate the search retrieval step with the answer generation model, effectively fine-tuning the model to use retrieved evidence (reducing hallucinations and improving relevance) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,AI%20Interview%20Series)). xAI likewise iterated on its model: Grok 1 was a base LLM, and subsequent versions (Grok 2, Grok 3) reflect additional training data, tuning, and improvements in reasoning. By February 2025 xAI claimed Grok 3 had significantly enhanced reasoning and real-time knowledge integration compared to earlier versions ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=It%20builds%20on%20the%20foundations,time%20data%20integration)), which suggests they fine-tuned and expanded the model’s capabilities (possibly incorporating more feedback and updated data). Overall, fine-tuning (whether supervised or via feedback) is a common thread to adapt large pre-trained models into specialized products.

Reinforcement Learning from Human Feedback (RLHF)

RLHF has become a standard technique for aligning AI assistant behavior with human preferences. OpenAI’s models heavily rely on RLHF: after pre-training GPT-4, OpenAI applied “post-training alignment” with human feedback to improve factuality and adherence to desired behavior (). In fact, GPT-4 was “fine-tuned using Reinforcement Learning from Human Feedback (RLHF)” (), as were its predecessors (InstructGPT and ChatGPT). This involves training a reward model from human preference comparisons and using it to further optimize the model’s responses via reinforcement learning. The result is more helpful, polite, and factually cautious behavior. Google has followed suit for its chat models – the Gemini team confirms they are “further refining the model using fine-tuning and reinforcement learning from human feedback (RLHF)” before broad deployment ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=For%20Gemini%20Ultra%2C%20we%E2%80%99re%20currently,before%20making%20it%20broadly%20available)). Google’s earlier LaMDA dialogue model and DeepMind’s Sparrow research also used human feedback to tune for safer, more correct answers. Hugging Face supports RLHF in open models through its TRL library (Transformers Reinforcement Learning) which implements Proximal Policy Optimization for language models. Researchers have used these tools to replicate RLHF on models like LLaMA, showing that “LLMs combined with RLHF…seems to be the next go-to approach for building very powerful AI systems” ([Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU](https://huggingface.co/blog/trl-peft#:~:text=Fine,very%20powerful%20AI%20systems)). xAI’s public stance has been to make their AI “truthful” and less censored; presumably they will also use human feedback, but possibly optimizing for different values. Notably, xAI’s first release (Grok-1) had not yet been safety-tuned with RLHF, which the company acknowledged – the base model was released “unrefined…not yet optimized for specific applications…not tuned with human feedback (RLHF)” ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=Grok,any%20ethical%20or%20safety%20guidelines)). This suggests xAI prioritized getting a large pretrained model out, and is likely applying RLHF or similar feedback techniques in later versions (Grok 2/3) to improve usability and alignment. Perplexity AI likely uses implicit feedback signals (clicks, user upvotes, etc.) and may fine-tune its models based on user ratings of answers. While Perplexity hasn’t published details of RLHF, its focus on factual accuracy aligns with using feedback to penalize hallucinations. In summary, RLHF is now ubiquitous: OpenAI and Google have made it a cornerstone of their model training, and even newer entrants (xAI, open-source efforts) consider it essential for aligning LLMs with human intent () ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=For%20Gemini%20Ultra%2C%20we%E2%80%99re%20currently,before%20making%20it%20broadly%20available)).

Search-Based LLM Integration

Beyond static retrieval from a database, these companies integrate live web search directly into their AI systems. OpenAI’s ChatGPT gained a browsing mode that queries the Bing search API and feeds the results into the model. OpenAI’s research showed that letting an LLM search the web and read results can significantly improve factual accuracy – their WebGPT project had the model issue search queries and quote from webpages, guided by an RLHF-trained reward model that favored correct answers with citations. Microsoft’s Bing Chat, built on GPT-4, is a prime example of search-LLM integration: it uses the Bing search index in real time and then the GPT-4 model formulates an answer citing those sources. Google similarly is fusing search with generative AI. Bard can now double-check its output by implicitly searching: when a user clicks the “Google it” button, Bard uses Google Search to find content supporting or contradicting its answer, and highlights which parts of its response are verified ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)). This feature, powered by Google’s search engine, helps mitigate hallucinations by referencing the live web after generation. Looking ahead, Google Gemini is expected to have even tighter integration, possibly using search results during generation (not just after) to ensure up-to-date knowledge. Google has immense search infrastructure (crawlers, index, PageRank) which it can pair with LLM reasoning. In fact, Google Research has explored tool use for LLMs (e.g. Toolformer), enabling models to decide when to call an API like Search. Perplexity.ai is built entirely around an LLM + search hybrid. When a query is asked, Perplexity’s system runs a search (originally using Bing, now using its own index), retrieves top results, and then feeds relevant text into an LLM to synthesize the answer ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=The%20evolution%20of%20Perplexity%20from,end%20of%20the%20answer%20engine)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections)). The process is tightly integrated: Perplexity engineered a custom web crawler and indexing pipeline (PerplexityBot) to gather high-quality content, using traditional IR algorithms (BM25, n-grams, authority signals) to rank results ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=PerplexityBot%20follows%20links%20and%20fetches,them%20in%20the%20news%20recently)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%E2%80%99s%20search%20index%20is%20smaller,to%20be%20relevant%20and%20trustworthy)). The LLM is instructed not to introduce information beyond what’s retrieved, ensuring “you’re not supposed to say anything that you don’t retrieve”, so if sources are insufficient it will respond that it lacks a good answer ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,434%20%282%3A08%3A56)). This disciplined approach effectively prevents the model from hallucinating unsupported facts, by design ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,%E2%80%99%E2%80%9D)). xAI’s Grok also has a search-based workflow: its “DeepSearch” mode will tap into real-time data from the web and even X/Twitter content, then present a well-cited answer ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=DeepSearch%20Mode)). This suggests Grok’s architecture treats the web as an extension of its knowledge, similar to Perplexity. Overall, integrating web search allows these AI systems to perform as answer engines rather than just predictive text generators. They each handle it slightly differently – some do retrieval before generation (Perplexity, Bing Chat), others offer a post-answer verification (Bard’s double-check) – but all recognize that coupling LLMs with search is crucial for reliable, up-to-date information ([What to Know About RAG LLM, Perplexity, and AI Search.](https://blog.phospho.ai/how-does-ai-powered-search-work-explaining-rag-llm-and-perplexity/#:~:text=Perplexity%20AI%20applies%20the%20RAG,the%20most%20complex%20user%20queries)) ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)).

Multi-Modal Processing Capabilities

Another major trend is extending models beyond text, into multi-modal inputs and outputs. OpenAI’s GPT-4 is explicitly “multimodal, accepting image and text inputs” and producing text outputs (). This was a significant advancement – for example, GPT-4 can analyze an image, describe it, or solve visual problems when given an image prompt. OpenAI achieved this by combining a vision encoder with the language model (details were not fully disclosed, but likely similar to techniques in Flamingo or BLIP where visual features are fed into the transformer). Google’s Gemini was “built from the ground up to be multimodal,” aimed at natively handling text, code, audio, images, and video ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Gemini%20is%20the%20result%20of,code%2C%20audio%2C%20image%20and%20video)). Rather than stitching separate models together, Google states “we designed Gemini to be natively multimodal, pre-trained from the start on different modalities” ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Until%20now%2C%20the%20standard%20approach,images%2C%20but%20struggle%20with%20more)) ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Gemini%20is%20the%20result%20of,code%2C%20audio%2C%20image%20and%20video)). In practice, this means Gemini’s architecture and training data cover multiple types of input so it can seamlessly combine them (e.g. understanding a question that includes an image and text together). Early benchmarks show Gemini Ultra outperforming previous state-of-the-art on image and audio understanding tasks, indicating the success of this unified approach ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=With%20the%20image%20benchmarks%20we,Gemini%27s%20more%20complex%20reasoning%20abilities)). Hugging Face has been a leader in open multi-modal models: the HuggingFace Hub hosts vision-language models like CLIP (image embeddings), BLIP-2 (which connects a vision encoder to a language model), and various speech models. HuggingFace’s Transformers library also supports multi-modal pipelines (for example, image captioning by feeding an image through a ViT model and text through a GPT-2). While HuggingFace hasn’t released a single monolithic multimodal model of its own, it enables researchers to combine modalities – e.g. LLaVA, an open-source variant of GPT-4’s vision capability, uses a LLaMA model fine-tuned to take visual features as input, and it’s available via HuggingFace. xAI has hinted at long-term goals of understanding “the true nature of the universe,” which may imply plans for multi-modal integration (for instance, interpreting images or scientific data), but as of Grok 3 their focus has been textual and programming domains. Perplexity currently focuses on text Q&A (with possible voice input in their mobile apps), and it outputs text (or code snippets) – they have not yet rolled out image understanding or generation. However, industry-wide the trend is clear: advanced AI systems are becoming increasingly multi-modal. Both GPT-4 and Gemini treat images as first-class inputs, and Gemini is even evaluated on audio and video tasks ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%27ve%20been%20rigorously%20testing%20our,LLM%29%20research%20and%20development)) ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=With%20the%20image%20benchmarks%20we,Gemini%27s%20more%20complex%20reasoning%20abilities)). This broadens the range of applications (e.g. describing an image, analyzing charts, transcribing audio) and moves these models closer to general AI assistants that can process any form of data.

Training and Inference Optimizations

Training giant models and serving them in production requires heavy optimization. Each of these organizations has developed techniques to improve efficiency and scalability. OpenAI invested greatly in a robust distributed training infrastructure for GPT-4, emphasizing “infrastructure and optimization methods that have very predictable behavior across multiple scales” (). This allowed them to scale up to GPT-4 (which was trained on unprecedented compute) while accurately predicting performance from smaller-scale experiments (). They likely used NVIDIA GPU clusters on Microsoft Azure – OpenAI noted ChatGPT/GPT-3.5 were trained on “an Azure AI supercomputing infrastructure” (ChatGPT). Techniques like model parallelism and pipeline parallelism (splitting the neural network across many GPUs) and memory optimization (such as DeepSpeed’s ZeRO redundancy) were certainly employed. Indeed, the open BLOOM model (176B) – which Hugging Face’s collaboration trained – used Megatron-Deepspeed (a fusion of NVIDIA and Microsoft’s frameworks) on 384 A100 GPUs for 3.5 months ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)). The BLOOM engineers describe mixing data, tensor, and pipeline parallelism to distribute the model, training in bfloat16 precision for speed ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)). These are the same kinds of optimizations likely under the hood for OpenAI and Google’s models (albeit on even larger clusters). Google developed its Pathways system to enable a single model to be trained across thousands of TPU v4 chips. Google’s research into efficient training includes the Chinchilla strategy (smaller model, more training data for better efficiency) and techniques like Mixture-of-Experts (MoE) (which they explored in the Switch Transformer to scale to trillions of parameters by sparsely activating sub-models). While Google hasn’t confirmed if Gemini uses MoE, they have confirmed offering Gemini in different sizes (Ultra, Pro, Nano) ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%E2%80%99ve%20optimized%20Gemini%201,version%2C%20for%20three%20different%20sizes)), which implies they trained a range of model scales – possibly by distilling knowledge from the largest down to the smaller ones for efficiency on device. They also mention plans to “increase the context window” and improve memory in future versions ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%E2%80%99ve%20made%20great%20progress%20on,information%20to%20give%20better%20responses)), which suggests work on architectures for longer sequences (like efficient attention mechanisms or retrieval-based long-term memory). xAI as a smaller newcomer had to optimize to compete. Notably, xAI’s Grok-1 model adopted a Mixture-of-Experts architecture with 8 expert networks totaling 314 billion parameters, but with only 2 experts (25% of weights) active per token ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)). This design dramatically cuts the computation needed per inference compared to a dense 314B model, acting instead like ~78B parameters per token (since only a subset of the network is used) ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)). By using MoE, xAI managed to train a model on a shorter timeline (the base model was developed in ~4 months) and then open-sourced it. They built their training on JAX (Google’s TPU-friendly framework) and likely leveraged cloud TPUs or GPUs efficiently ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=any%20ethical%20or%20safety%20guidelines)). For inference, all these companies use methods like half-precision (FP16/BF16) or lower, and optimize the model serving. OpenAI, for instance, has optimized kernels for faster matrix multiplies and might be employing model sharding across GPUs for each query to serve GPT-4. Perplexity optimized for real-time interactive inference: they mention a system of “tens of LLMs (ranging from big to small) working in parallel to handle one user request quickly and cost-efficiently” ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%20also%20routes%20queries%20to,and%20Pro%20users%20can%20choose)). In practice, Perplexity routes queries to an appropriate model (complex questions to bigger models like GPT-4, simpler ones to smaller internal models) and even runs multiple components concurrently (e.g. retrieving documents while another model generates an initial summary) to minimize latency. This dynamic allocation of compute is an innovative optimization to balance quality vs. speed on a per-query basis ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%20also%20routes%20queries%20to,and%20Pro%20users%20can%20choose)). Hugging Face focuses on enabling such optimizations for the community: they provide the Accelerate library to easily distribute models across devices, and Optimum for optimized inference (including graph optimization and quantization). Quantization in particular has been a major theme – reducing model precision to 8-bit or 4-bit can drastically speed up inference and lower memory use. HuggingFace has integrated support for 8-bit transformers and even 4-bit fine-tuning (QLoRA), allowing models like 65B LLaMA2 to be run on single GPUs with only minor performance loss. This democratizes running “deep” models without requiring the enormous infrastructure that OpenAI or Google have. In summary, the cutting-edge AI players are pushing on all fronts of efficiency: distributed training at massive scale, new architecture tricks (sparse experts, etc.), smarter use of hardware (GPUs/TPUs), and software innovations (better parallelism, caching, quantization) to make these models feasible to train and serve globally () ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)).

2. Architectural Insights

Model Architectures: Transformers with a Twist

All the models in question are fundamentally based on the Transformer architecture, but each has taken it to new levels. OpenAI’s GPT-3/4, Google’s PaLM/Gemini, xAI’s Grok, and most open models are transformer-based generative models. The GPT-4 technical report explicitly states “GPT-4 is a Transformer-based model pre-trained to predict the next token” (). Google hasn’t published Gemini’s details yet, but it’s almost certainly a Transformer backbone as well (Google’s previous PaLM 2 and DeepMind’s models are all Transformers). Hugging Face’s flagship projects like BLOOM also use a standard Transformer decoder architecture (BLOOM’s design is akin to GPT-3 with some improvements) ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)). Where architectures diverge is in the scale and composition: OpenAI has kept GPT-4’s exact size and structure secret, leading to speculation. Some industry analysts inferred GPT-4 might be an ensemble or MoE of multiple sub-models to reach its performance. In fact, xAI’s team pointed out that “GPT-4…are also based on this [Mixture-of-Experts] architecture” when discussing their own MoE model ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)) (though OpenAI has not confirmed this claim). Mixture-of-Experts (MoE) is a notable architectural idea where several expert networks are trained and a gating mechanism activates only a few for each input. This allows a model with, say, hundreds of billions of parameters in total, but each token only passes through a fraction of them, saving compute. Google’s Switch Transformer (2021) demonstrated this at up to 1.6 trillion params (with 32 experts) and inspired others. xAI’s Grok-1 adopted an 8-expert MoE architecture (314B total) with 2 experts active per token ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)). This design choice indicates xAI traded off some complexity to achieve a very high parameter count within their resource constraints. Meanwhile, OpenAI and Google so far appear to stick with dense Transformer models for their flagship (GPT-4, PaLM2, etc.), but with many enhancements in training. Dense means every parameter can affect every token – which is simpler to implement but extremely costly at huge scales. DeepMind’s research found that a 70B dense model (Chinchilla) trained on 4× more data can outperform a 175B model – so the trend has been to find the right model size vs data trade-off rather than blindly scaling parameters. Google’s Gemini might not be vastly larger than GPT-4 in parameter count; instead it may rely on better training (multimodal data, high-quality instructions, etc.) and efficient use of parameters. One clue: Gemini Ultra surpassed GPT-4 on many benchmarks ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%27ve%20been%20rigorously%20testing%20our,LLM%29%20research%20and%20development)) ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Gemini%20Ultra%20also%20achieves%20a,different%20domains%20requiring%20deliberate%20reasoning)), yet Google is comfortable deploying smaller variants (Pro, Nano), suggesting the architecture is scalable and distillable. Each company is also exploring modular architectures: for example, multi-modality often means bolting on encoders or decoders for other data types. GPT-4’s image input comes from a vision module that encodes images into embeddings the Transformer can understand. Google’s older multi-modal efforts (like Perceiver, or Flamingo) tried mixing modalities within one Transformer. Gemini claims to be “native” multi-modal, likely meaning a single unified model rather than separate networks per modality ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Until%20now%2C%20the%20standard%20approach,images%2C%20but%20struggle%20with%20more)). Another architectural innovation is how these models incorporate tool use and reasoning. Rather than change the neural architecture, OpenAI and others have found they can get better reasoning by techniques like chain-of-thought prompting or by letting the model output and execute code. This doesn’t change the transformer’s structure per se, but it’s an emerging functional architecture: e.g. OpenAI’s Code Interpreter plugin allowed GPT-4 to use a Python sandbox for calculations, effectively extending the system’s capabilities without altering the neural network. We can expect future architectures to blend neural and symbolic or neural and programmatic components more deeply. In summary, the core model for all these AI systems remains the Transformer (decoder-style for generation), with differences in scale (number of layers/heads) and extensions (experts, multi-modal encoders, tool interfaces). OpenAI and Google have kept their largest architectures proprietary, but it’s believed they experiment with combinations of dense and sparse subsystems. xAI has been more transparent by open-sourcing an MoE model, illustrating one path to scale that others have hinted at. Hugging Face’s facilitation of research means many novel architectures (from retrieval-augmented transformers like Retro, to alternative formalisms like RNN-Transformer hybrids) are available, but none have yet surpassed the basic Transformer for language tasks. Thus, today’s cutting-edge models are still “Transformers under the hood” – albeit optimized and augmented by every trick in the book.

Size and Scale of Models

The scale of these models is immense, though each company balances size with efficiency. OpenAI’s GPT-3 was 175 billion parameters (2020), and GPT-4 is rumored to be on the order of a few hundred billion (exact number not disclosed) – yet GPT-4 significantly outperforms models much larger than 175B due to training improvements. OpenAI observed that performance improved predictably as they scaled up, following power-law scaling laws () (). They also leveraged “no more than 1/1,000th the compute of GPT-4” runs to predict GPT-4’s performance, indicating a strong application of scaling law research () (). Google’s PaLM (2022) had 540B parameters, one of the largest dense LLMs publicly described. However, after DeepMind’s scaling law findings (the “Chinchilla” paper) suggested PaLM was under-trained relative to its size, Google pivoted to more optimized scaling. PaLM 2 (2023) was actually smaller (reportedly in the tens of billions range for the largest version) but trained on higher-quality data, enabling better performance than the original PaLM. Gemini Ultra’s size isn’t published, but given it outperforms GPT-4, it could be in the same ballpark (maybe ~>200B dense or an effective higher size if using experts). Google also broke Gemini into Ultra / Pro / Nano tiers ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%E2%80%99ve%20optimized%20Gemini%201,version%2C%20for%20three%20different%20sizes)) – this implies they trained a massive model (Ultra) and then likely distilled or architecturally scaled it down. Gemini Nano running on-device suggests ~billion-scale parameters, which is remarkable if it retains strong capabilities. This multi-scale approach echoes earlier Google work on Mixture-of-Experts where a huge model could be pruned or gated for smaller deployments. xAI’s Grok-1 at 314B (MoE) is one of the largest openly released models by parameter count ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)). Because only a subset of those weights are active at a time, its effective capacity is around 40–80B parameters per token, but the high total param count helps it store a lot of knowledge. By comparison, Meta’s LLaMA-2 tops out at 70B dense parameters (also hosted by Hugging Face) and BLOOM is 176B dense. So xAI’s model is an attempt to leapfrog other open models in sheer size. It did require massive compute – xAI used ~200 million GPU-hours on a cluster of 200k GPUs (as one report claims) to train Grok 3 ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Unprecedented%20Computational%20Power)), which if accurate, is an order of magnitude more than earlier open efforts. This highlights that size alone is not a silver bullet; the scaling laws (how to best spend a compute budget between model size and training data) are key. DeepMind’s analysis suggests for a given compute budget, there is an optimal model size and data size – undertraining a giant model is wasteful, and overtraining a small model underfits. All players are now mindful of these trade-offs. OpenAI likely increased not just model size but also the quality/quantity of training data for GPT-4, and used techniques to make training more compute-efficient (e.g. better optimizers, parallelism). Google explicitly cites “advances in scaling laws and efficiency” in their work – Gemini’s team leveraged Google’s TPU v4 pods and possibly algorithms like AdaFactor, sharded training, and parallel layers to push throughput. They also mention working on “increasing the context window” ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%E2%80%99ve%20made%20great%20progress%20on,information%20to%20give%20better%20responses)), which relates to architecture (positional encoding) but also to efficient attention mechanisms (standard self-attention cost grows with the square of context length, so longer contexts require memory optimization like FlashAttention or chunked processing). Hugging Face via BigScience showed that a volunteer coalition could train a 176B model given enough optimization – 384 GPUs over 3.5 months for BLOOM ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)). They published that BLOOM saw 350B tokens of training data ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Hardware%20384%2080GB%20A100%20GPUs,5%20months)); for comparison GPT-3 saw about 300B tokens. This indicates that modern LLMs in the 100B+ range are trained on a few hundred billion to a trillion tokens of text. (OpenAI and Google likely curated even larger corpora, possibly including multi-modal data for GPT-4 and Gemini). As for compute usage: GPT-4’s training was estimated to cost tens of millions of dollars in GPU time. These companies also optimize inference at scale – serving millions of user queries. Here distillation and model compression are important. OpenAI hasn’t shared if they distilled GPT-4 into smaller models, but OpenAI did distill GPT-3 into smaller 13B and 6B “InstructGPT” models in early alignment research (finding a 6B model fine-tuned on human feedback could surpass a 175B model on instruction following ([ChatGPT](https://chatgpt.r4wand.eu.org/#:~:text=Methods))). It’s plausible they use a similar idea internally to reduce load (e.g. a smaller model handles simple prompts, the big model only when needed). Perplexity explicitly takes this approach: easy questions or follow-ups might be handled by a smaller model or a cached answer, while complex ones invoke the big guns, thereby optimizing cost/latency. To summarize scale: OpenAI and Google are in an arms race for quality, but not simply by maxing out parameter count – they carefully choose model size, train on as much high-quality data as possible, and use architectural tricks (like sparse experts or improved attention) when beneficial. xAI, with fewer resources, boldly used an MoE approach to reach a high param count and open-sourced it to gather community support. Hugging Face focuses on enabling broad access to large models rather than pushing absolute state-of-art; even so, it helped prove that 100B+ models can be trained outside Big Tech with enough collaboration. The future likely lies in effective scaling – getting more out of each parameter (through smarter training, better data, and combining retrieval or tools so the model doesn’t need everything “in its weights”).

Innovations in Efficiency and Scaling Laws

A clear trend in cutting-edge AI research is improving efficiency – achieving more intelligence without simply using more parameters. One pillar of this is applying insights from scaling law research. OpenAI’s early scaling law paper (Kaplan et al. 2020) posited that model performance follows smooth power-law improvements with more model parameters and more data, suggesting that if you want better results you can just scale up. Indeed, GPT-3’s creation was guided by those laws, and it validated the approach. However, later research by OpenAI and DeepMind found that for a fixed compute budget, there’s an optimal balance (a smaller model trained on more tokens can outperform a bigger model trained on fewer). This was exemplified by DeepMind’s Chinchilla model (70B), which was trained on 4× more data and outperformed Gopher (280B) that had been under-trained. It’s likely that GPT-4 and Gemini followed the Chinchilla strategy – not dramatically increasing parameters beyond their predecessors, but greatly increasing training data and training time to fully utilize the model’s capacity () (). Both OpenAI and Google have hinted that they achieved major gains by “predictable scaling” and methodology to project performance from smaller models (), meaning they probably found that sweet spot of model size vs data. On the other side, Meta’s LLaMA showed that even a 13B or 30B model can perform extremely well if trained on excellent data (their models were trained on 1+ trillion tokens of diverse text) – indicating that efficient training can make smaller models punch above their weight. We see this in the open-source world where a fine-tuned 13B LLaMA-2 can often match a 175B GPT-3 on many tasks, thanks to inheritance of those efficient scaling lessons.

Beyond data, there are architectural efficiency innovations: we discussed Mixture-of-Experts for parameter sparsity. Another is sparsity in the weights or activations – e.g. pruning unnecessary neurons after training, or using sparse attention patterns. Models like GPT-4 and Gemini likely use dense attention across the full context, but research like Sparse Transformers and BigBird (Google) introduced attention patterns that scale to longer texts by not attending to every token pair. Long context handling is itself an efficiency challenge being tackled: GPT-4 offers up to 32K context tokens (and a 128K version for some users), achieved by positional encoding improvements (OpenAI used techniques like rotary positional embeddings or segmentation to extend context). Other efficient attention methods (such as FlashAttention, which is a faster exact attention algorithm) are being adopted in libraries and could be in use to speed up these models’ inference. Training optimization algorithms also contribute: OpenAI mentions developing “optimization methods that behave predictably at a wide range of scales” (), which hints at tuning hyperparameters (learning rates, etc.) in a scale-invariant way. Google’s Optimizers like Adafactor help reduce memory use on giant models, and techniques like gradient checkpointing reduce memory at the cost of extra compute, enabling deeper models to train. Hugging Face’s BigScience experiment applied many of these: ZeRO stage-3 to shard optimizer states, gradient checkpointing, BF16 precision – all necessary to train BLOOM within 80GB GPU memory limits ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)). They also open-sourced their training logs and lessons, contributing to industry knowledge on large-scale training.

Another area of efficiency is Parameter-Efficient Fine-Tuning (PEFT). Instead of fine-tuning all 175B weights for each new task (which is costly), methods like LoRA (Low-Rank Adaptation) inject a small number of trainable parameters. OpenAI hasn’t publicly used LoRA on GPT-4 (it fine-tunes full models for RLHF), but in the open community, LoRA and adapter modules let individuals fine-tune huge models on consumer hardware. This dramatically lowers the barrier to adapting models to niche domains and is something Hugging Face has championed (with the PEFT library).

Finally, a crucial strategy for both accuracy and efficiency is augmenting models with tools or external knowledge – which we’ve covered in RAG and search integration. By offloading certain tasks to tools (search, calculator, code execution), a model can get away with being smaller or less trained in that niche. OpenAI’s plugin ecosystem is exactly about this: why force the model to memorize every factual detail or how to do math, if it can call a database or Python interpreter? This philosophy increases effective capability without proportionally increasing the model size. Google’s approach with extensions (Bard connecting to Maps, Gmail, etc. ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=Google%20has%20updated%20Bard%20with,more%20capable%2C%20the%20company%20announced)) ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=Bard%20extensions))) follows the same logic. We can view these as architectural extensions at the system level: the AI consists of the core LLM plus a suite of external modules it can consult. This modular approach is likely to grow, because it improves both accuracy (via specialization) and efficiency (not every query needs the full LLM power – sometimes it’s a database lookup or a simple script).

In summary, the design trend is smart scaling, not just brute-force scaling. The core architectures are being pushed to their limits, but through smarter training regimes (following scaling laws), new forms of sparsity, and incorporating external resources to avoid reinventing the wheel. This means future “deep research” AI models might not be mind-boggling 10 trillion-parameter behemoths, but rather more clever combinations of moderately large models, retrieval systems, and tool-use policies – all optimized to work in concert.

3. Search and Information Retrieval Integration

LLMs Augmented with Live Search

A hallmark of these next-gen AI products is tight integration with search and information retrieval systems, effectively marrying NLP with IR. Traditional search engines (Google, Bing) use keyword and link analysis to find relevant documents; LLMs excel at understanding context and generating fluent answers. Combining the two yields a powerful conversational search experience with grounded answers. Perplexity AI demonstrates this clearly: it crawls and indexes the web, then for a given query, retrieves top results (using ranking signals like BM25, n-grams, domain trust) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Aravind%20has%20also%20spoken%2C%20in,like%20PageRank)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%E2%80%99s%20search%20index%20is%20smaller,to%20be%20relevant%20and%20trustworthy)), and feeds the most relevant snippets to its LLM. The LLM then produces a direct answer, but with inline citations linking back to sources ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections)). Users get a concise answer and can click to verify the source – a huge improvement in experience over a list of blue links. Perplexity engineered this pipeline to emphasize factual grounding: the assistant is instructed not to introduce facts that weren’t in the retrieved text ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,434%20%282%3A08%3A56)). If the query is obscure and the search doesn’t find good information, the system will admit it doesn’t have an answer rather than hallucinate ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,434%20%282%3A08%3A56)). This design addresses misinformation by construction.

Bing Chat (OpenAI & Microsoft) uses a similar approach on the backend. It performs a Bing web search for the user query (or the conversation context), then uses the retrieved web content as additional context for GPT-4. The model’s prompt is prepended with snippets from webpages, and the model is prompted to ground its answer in those. That’s why Bing Chat often responds with references [^1^][^2^] – it is citing the web articles it was given. This is essentially a RAG system in production. The benefit is real-time knowledge (Bing Chat knows about events up to the minute, because it can search news) and higher factual accuracy. The challenge is ensuring the model doesn’t ignore the sources or generate something contradictory. Microsoft and OpenAI likely did some fine-tuning so that the model learned to incorporate the provided text and prefer it over its parametric knowledge when answering factual questions.

Google’s integration of LLMs with search is evolving rapidly. Initially, Bard was separate from Google Search (and famously, Bard made a factual error in its first demo when it didn’t consult the latest James Webb Telescope data). Google quickly moved to leverage its core competency – search – to boost Bard. By September 2023, Google introduced a “Double-check response with Google” feature ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=Double,Search)). This effectively takes Bard’s answer, runs a Google search on key claims, and then highlights which parts of the answer are supported by search results and which might be unsupported ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=Google%20DeepMind)). In other words, Bard uses search post hoc as a verification layer. Google’s blog explained that this uses research from Google Research/DeepMind and will highlight phrases with relevant search hits ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)). While this is slightly different from feeding content to the model before answer (it’s after-the-fact), it still provides the user a way to assess correctness and source the information. Google likely did it this way initially to avoid changing Bard’s generative process too much – Bard generates freely, then the system checks. However, looking forward, Gemini might integrate search more directly. Since Gemini is being positioned as a foundation for future Google products, we might see a mode where Gemini actively performs retrieval during generation (a bit like an agent). Google’s SGE (Search Generative Experience) already shows short AI summaries at the top of some search queries, with links to sources. Those summaries are produced by a variant of their LLM (PaLM 2) that is constrained to use the top search results. Google has also researched training LLMs that inherently cite sources – e.g. their “AGReE” framework for attributions in generation ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=I%E2%80%99m%20not%20sure%20how%20Perplexity%E2%80%99s,part%20of%20the%20response%20generation)). So we can expect Google to push toward fully integrated systems where the LLM and search engine work in tandem in real time.

xAI’s Grok integration with search (and X data) is an attempt to differentiate it from ChatGPT by being more up-to-date and possibly more “truth-seeking.” Grok’s “real-time knowledge retrieval” means it will use the latest posts and information available to it ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Grok%203%20is%20an%20AI,time%20knowledge%20retrieval)) ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Real)). Given Elon Musk’s access to Twitter data, Grok might incorporate a specialized index of Tweets to answer questions about current events or opinions. This could make it behave somewhat like a social media search + AI summarizer. One can imagine asking Grok for “What’s the latest on topic X?” and it pulls recent trusted tweets or articles and gives an answer. That would be a niche that leverages xAI’s unique data access. Technically, Grok likely uses an API to query relevant information (similar to how it would query the web) and then conditions the model on that text.

Indexing and Ranking Techniques

One may wonder: how do these AI systems choose what information to retrieve, especially if they maintain their own search index? Perplexity provides some insight. They built a custom index focusing on high-quality content: “a much more compact index optimized for quality and truthfulness…less focused on the tail of the distribution” ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%E2%80%99s%20search%20index%20is%20smaller,to%20be%20relevant%20and%20trustworthy)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,AI%20Interview%20Series)). In an interview, Perplexity’s team noted that many user queries concern popular topics and authoritative sources (the “head” of the distribution), so they prioritized indexing those well ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=niche%20queries%3A)). This improves the odds that the retrieved evidence is correct and easy for the model to use. They use classic IR algorithms under the hood – e.g. BM25 for term matching, which improves upon simple TF-IDF by normalizing for document length and term saturation ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Aravind%20has%20also%20spoken%2C%20in,like%20PageRank)). They also consider domain authority (similar to Google’s PageRank concept) to favor reputable sites ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Aravind%20has%20also%20spoken%2C%20in,like%20PageRank)). Interestingly, Perplexity’s CEO mentioned skepticism about over-relying on vector similarity search alone ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=PerplexityBot%20follows%20links%20and%20fetches,them%20in%20the%20news%20recently)). While they likely use embedding-based retrieval for semantic matches (and definitely use embeddings to select the most relevant snippets for the LLM ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections))), they found that traditional keyword search is very strong for initial candidate retrieval. This mirrors real Google: Google uses BERT embeddings to understand queries, but still uses a massive inverted index and hundreds of hand-crafted signals to retrieve and rank results. So these AI search systems are not throwing out decades of IR research – they are combining it with LLM capabilities. For Google, their search ranking algorithms (RankBrain, neural matching, etc.) are already heavily optimized; integrating LLMs mostly affects how results are presented or used, not the core ranking. However, Google has an experimental Search Generative Experience where the top results are distilled by an LLM into a paragraph. The ranking there likely feeds directly – i.e. the LLM summarizes the top N results that the classic algorithm chose. Over time, if LLM answers become the default, one could imagine a feedback loop where the LLM’s ability to find relevant info might influence what it asks the search index for.

Bing’s approach (which Perplexity initially used via API) is to use the existing Bing ranking to get, say, 5–10 relevant pages. Then a separate module reads those pages and extracts key passages. It may use a lightweight model to score passages or just take the top ones. Those passages are then concatenated (truncated if too long) and given to the main LLM. This multi-step retrieval is important: just feeding raw full webpages could exceed the context window, so systems filter and chunk information. We saw Perplexity does this explicitly, “extracting relevant paragraphs or sentences using LLMs and embedding techniques” before sending to the answer generator ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections)). The use of a smaller LLM or embedding model to identify the most relevant snippets is a form of neural reranking, refining the search results so the big LLM sees only the cream of the crop.

xAI’s data indexing likely includes not just web but possibly proprietary datasets (e.g. the Twitter firehose). They might rank tweets by engagement or author credibility when answering questions, to avoid pulling in random low-quality content. If Grok is asked a factual question, it should prefer news articles or Wikipedia over someone’s tweet – unless the query is specifically about public opinion.

Fact-Checking and Misinformation Mitigation

Integrating retrieval is itself a mitigation against hallucinations, but these companies add other safeguards too. We discussed how Perplexity’s system rules forbid adding unsupported info ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,434%20%282%3A08%3A56)). This drastically lowers misinformation, at the cost that the assistant will sometimes say “I don’t know” (which is arguably a good thing when info is absent). OpenAI and Google use RLHF to penalize blatantly false or harmful outputs. Human annotators in the training loop will give low ratings to answers that contain false facts, teaching the model to prefer saying “I’m not sure” or to use hedging when uncertain. OpenAI’s models improved in factuality from GPT-3.5 to GPT-4, partly due to fine-tuning on expert demonstrations and feedback. The GPT-4 system card mentions efforts to reduce “specific dangerous misunderstandings.”

Google Bard’s double-check feature is an explicit fact-check: it marks sentences green or orange depending on whether web evidence supports them ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)). This not only helps users spot errors, it creates a reputational incentive for the model to get things right (since obvious contradictions will be highlighted). In the future, Google could integrate this check into the generation process itself – e.g. Bard might internally verify each claim before finalizing an answer. Their research on “self-consistency” and “fact-checking via search” points in this direction.

Another mitigation is source citation. Providing sources (as Bing, Perplexity, and others do) helps users verify information and potentially deters the model from making things up (since it knows a source is expected). Google’s SGE and experimental AGReE approach try to have the model generate answers with references attached ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=I%E2%80%99m%20not%20sure%20how%20Perplexity%E2%80%99s,part%20of%20the%20response%20generation)). This is non-trivial because the model has to keep track of which part of its answer came from which source, but it’s an active area of research.

OpenAI so far hasn’t made ChatGPT cite by default (outside of the plugin that shows sources), but with the Bing integration in their ChatGPT (Plus) browsing, they do show citations. Perhaps in the future ChatGPT will internally adopt a retrieval-and-cite mechanism for questions asking for factual data.

One more method to prevent misinformation is model moderation filters – separate models that analyze outputs for likely false or harmful content. OpenAI and Google both have moderation layers. For instance, OpenAI has a system that will refuse to answer certain categories (self-harm, medical, etc.) unless a proper source is given. Google will soon incorporate its Knowledge Graph or fact databases to catch obvious factual errors (for example, if Bard claimed “Paris is the capital of Germany,” a knowledge base check could immediately block or correct that).

xAI interestingly has taken a stance of fewer refusals (“won’t do pointless censorship” as Musk said), which means they rely even more on factual grounding to avoid misinformation. By open-sourcing their model, they invite community scrutiny which can uncover and correct mistakes. They might also use community feedback (e.g. if users report an answer as wrong, it can be fixed via fine-tuning).

In conclusion, search integration is the primary tool to combat hallucinations – by giving the model access to real information in real time, the system anchors answers in reality. All the companies we’re analyzing leverage this: either directly (LLM reads the data) or indirectly (LLM’s output is checked against data). On top of that, human feedback training and explicit citation/verification features provide additional layers to ensure correctness. It’s a recognition that no matter how large a language model is, if it operates purely on internal learned knowledge, it will eventually be outdated or confidently wrong. The solution is making these models open-book and cross-checking by design ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,434%20%282%3A08%3A56)) ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)).

4. Best Guesses on Proprietary Designs and Future Trends

(The companies guard many implementation details as trade secrets, but we can infer some design choices and likely directions from performance and public hints.)

• OpenAI (GPT-4 and beyond): OpenAI has not revealed GPT-4’s architecture or size due to the competitive landscape (). It’s plausible that GPT-4 introduced new optimizations under the hood. Some in the industry speculate GPT-4 could be an ensemble of models or use a Mixture-of-Experts with several large experts (one report suggested 8 experts) ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)), which would explain its leap in capability without an outrageous single-model size. OpenAI’s CEO denied a specific MoE rumor, but even if GPT-4 is a single dense model, it likely benefited from training on diversified data (code, images, human feedback, etc.) and longer context. OpenAI also might have improved the fine-tuning pipeline: using GPT-4 itself to assist in training (e.g. generating high-quality synthetic data or feedback, a form of bootstrapping). The system behavior of ChatGPT (how it follows instructions like the “system” prompt) suggests a lot of careful prompt engineering baked into the model. They probably did multi-stage fine-tuning: first on supervised data, then several rounds of RLHF (as described in the ChatGPT paper) ([ChatGPT](https://chatgpt.r4wand.eu.org/#:~:text=Methods)) (ChatGPT). In terms of efficiency, OpenAI leveraged Azure’s optimized GPUs and networking – possibly using NVLink/NVSwitch heavily to allow model parallelism across many GPUs. Given GPT-4’s ability to handle 32k-token contexts, OpenAI likely implemented rotary position embeddings (RoPE) or a similar technique to extend context length without retraining from scratch. Observers have noted GPT-4 is much better at “chain-of-thought” reasoning when asked, hinting that OpenAI trained it with methods to improve reasoning (for example, having it generate rationales and then final answers during training). In the future, OpenAI might incorporate modular experts – perhaps specialized subsystems for different domains (coding, math, law) that the model can internally consult. There is also interest in automated feedback: OpenAI has done research where models critique other models’ outputs. GPT-4 could be refined by using GPT-4 (or GPT-5) as a judge, not just humans. This could massively scale RLHF-style alignment without as much human labor. Another guess: OpenAI will continue to integrate tools deeply – possibly a version of GPT that can use a browser, a calculator, etc. on its own (building on the plugin work) rather than requiring user to choose a plugin. Architecturally this might mean a planning module that decides when to invoke a tool. OpenAI’s patents and hiring also point to working on memory systems, so that the assistant can remember long-running conversations or a user’s preferences over time beyond the immediate context window.

• Google Gemini (and Bard): Google has openly stated Gemini is being built by combining “Google Research & DeepMind’s” expertise, including techniques from AlphaGo (which suggests reinforcement learning and planning) ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%E2%80%99ve%20made%20great%20progress%20on,information%20to%20give%20better%20responses)). So one educated guess is that Gemini will incorporate a form of reinforcement learning for reasoning or tool use. For example, it might internally simulate steps to solve a problem (like game moves or multistep reasoning) and use an algorithm like Monte Carlo Tree Search (MCTS) to decide on the best answer – akin to AlphaGo’s approach but for language tasks. Demis Hassabis hinted that Gemini will have capabilities like planning and problem-solving that go beyond what LLMs usually do. This could manifest as Gemini being able to break down tasks into subtasks internally (perhaps an explicit scratchpad or prompt that it iterates on, which is something we already see via chain-of-thought prompting, but Google might formalize it). They are also aiming for memory enhancement ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=We%E2%80%99ve%20made%20great%20progress%20on,information%20to%20give%20better%20responses)) – one approach could be an external memory database that Gemini writes important facts to and reads from (somewhat like a differentiable neural wiki). Or simply a much larger context window using efficient attention or retrieval. Given Google’s expertise in retrieval, Gemini might dynamically retrieve not just from the web, but also from internal knowledge bases (Google’s Knowledge Graph, scientific databases, etc.) as part of its generation. On the efficiency side, Google will leverage TPU v5 perhaps, and could use sparsity if it yields gains – maybe a mixture-of-experts like Switch Transformer hidden inside Gemini Ultra to allow scaling beyond dense limits. Since they offer Gemini Nano for mobile, they likely distilled the big model’s knowledge into a smaller model using knowledge distillation or reward distillation (learning to mimic the larger model’s responses and behavior). Google could also be using feedback from billions of Search queries to inform Gemini’s training (anonymized, aggregated). For instance, they might fine-tune Gemini on Search’s query->click data so it better knows what users consider relevant. Another likely design: multimodal embedding space – Gemini may have a single representation space that jointures text, images, audio, etc., making it very fluid in understanding mixed inputs. That’s hinted by “natively multimodal” ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Until%20now%2C%20the%20standard%20approach,images%2C%20but%20struggle%20with%20more)). In summary, we expect Gemini to be not just a bigger chat model, but one with tool-using smarts and efficient scaling. It’s positioned as a platform for many Google products (from cloud to Android), so it’s being designed with adaptability (the fact it has Ultra/Pro/Nano variants suggests a Pathways approach where one architecture can be scaled down or certain parts activated for different needs).

• xAI (Grok): xAI’s strategy so far has been to iterate quickly and leverage openness. Grok-1 was trained fast with MoE, Grok-3 claims new reasoning prowess ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Grok%203%20%E2%80%94%20ever%20since,the%20news%20in%20the%20industry)). One guess is that xAI heavily utilized Twitter data in training – Elon Musk has indicated using “X” as a data source. This means Grok might have a different “personality” or strength: possibly very good at current events, casual internet language, and technical discussions (since Twitter has a lot of tech chatter). On the core model side, since they open-sourced Grok-1, outside developers might fine-tune and improve it (community RLHF, multi-modal fine-tunes, etc.), which xAI can then incorporate – a crowdsourced improvement loop. To catch up with competitors, xAI likely focuses on efficiency techniques like MoE (already done) and maybe low-precision training (JAX/Flax is good for bfloat16, maybe even int8 training experiments). Grok-3’s announcement highlighted “chain-of-thought process” and the ability to “backtrack and correct errors” ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Advanced%20Reasoning%20with%20Reinforcement%20Learning)), implying xAI has trained the model to do step-by-step reasoning similarly to how researchers fine-tuned models to output their reasoning steps. They mention reinforcement learning for reasoning in Grok 3 ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Advanced%20Reasoning%20with%20Reinforcement%20Learning)) – perhaps they had the model practice solving puzzles or math problems with trial and error, rewarding it for correct solutions, which could yield better logical skills. In proprietary terms, xAI might trade off some of the strict safety filtering that others have, thereby giving the model more “freedom” to output edgy or unfiltered content. Musk’s philosophy was that AI should give users information even if it’s politically incorrect; implementing that means setting RLHF rewards differently (i.e. not penalizing answers for being controversial unless factually wrong). So xAI’s model might be optimized for “truthfulness” over “inoffensiveness.” From a performance standpoint, if Grok-3 truly “outperforms all competitors” (a bold claim), it suggests xAI might have expanded beyond the initial MoE – possibly increasing experts or training a larger dense model now that they have more compute (Musk reportedly bought thousands of GPUs). They could also integrate Dojo, Tesla’s supercomputer, if they align with Tesla for hardware. This could give them an edge in running physics simulations or video data through the model (a unique angle given Tesla’s autopilot data). In short, xAI’s proprietary twist may be in aggressively adopting new research ideas (they were quick on MoE and open-sourcing, which others haven’t done at that scale) and leveraging Musk’s ecosystem (Twitter data, Tesla hardware) to differentiate.

• Perplexity AI: As a smaller startup, Perplexity’s “secret sauce” is in how it orchestrates multiple components. We learned that they use a network of specialized sub-models and processes (query classification, document retrieval, snippet extraction, answer generation) in parallel ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%20also%20routes%20queries%20to,and%20Pro%20users%20can%20choose)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections)). Their proprietary design likely includes a lot of caching and smart routing. For example, if a question has been asked before, they might detect it and quickly return a cached answer (with sources) rather than recompute everything – important for efficiency at scale. They may also fine-tune their main answer-generation model on the domain of “answering from web snippets”, which is a slightly different distribution than general chat; this fine-tuning can make it better at quoting and abstaining when info is insufficient. Since Perplexity now has its own web crawler and index, a design challenge is how to keep that index fresh and avoid getting stale info. They likely use incremental indexing where popular sites or frequently queried domains are recrawled often, while others less so. They acquired a retrieval startup (Carbon) in late 2024, which suggests they are working on more advanced indexing, possibly vector indexes to complement keyword search for certain queries (like a semantic search for similar questions). They might also be building a knowledge graph or at least linking concepts, so that if one asks a related question, the system can reuse previous retrieval context. On the model side, Perplexity might be experimenting with a smaller in-house LLM that they can deploy at scale without relying on OpenAI. They could fine-tune an open model (say LLaMA-65B or Falcon 40B) on a large Q&A dataset plus conversations, aiming to approximate GPT-4’s quality on the narrow task of web Q&A. If they succeed, that saves cost and gives them more control (no API rate limits). There’s evidence they were training models: the CEO mentioned an “infrastructure to train LLMs together with search end-to-end” ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,AI%20Interview%20Series)). This end-to-end training could mean they backpropagate through the retrieval step – essentially teaching the model to do searches that improve answer quality. Such a setup would be very cutting-edge (differentiable search ranking). Even if not fully end-to-end, they could use feedback data: every time a user clicks a source or gives a thumbs-up, that could reinforce the system. So, Perplexity’s design likely includes a continual learning loop from user interactions, gradually refining its relevance ranking and answer tone. On the front end, they introduced features like “Copilot” (an interactive follow-up question assistant) and likely will add multi-modal search (ask a question about an image, etc.) — since their model is modular, adding an image-to-text module before the question goes into the pipeline is feasible. I suspect Perplexity will also lean into domain-specific agents: e.g. a mode for scholarly research that only searches academic papers, or a code assistant mode that integrates with an API like StackOverflow search. Their architecture is flexible enough to do this by swapping out the search corpus and maybe using a code-specialized LLM for the generation step. In essence, Perplexity’s strength is treating the LLM not as a monolith, but as one component in a complex retrieval and reasoning system – their proprietary know-how is in balancing these components to yield fast, factual answers (and doing so under the hood such that the user just experiences a smooth conversation).

• Hugging Face: As an open platform, Hugging Face’s “proprietary” approach is a bit different – they focus on enabling others’ models and pushing openness. But we can discuss likely design trends they champion. One is community-driven fine-tuning: Hugging Face will likely host more projects like OpenAssistant, where thousands of volunteers help label data or fine-tune an open model to reach ChatGPT-like alignment. This decentralized RLHF could produce models that rival closed ones. Hugging Face might also invest in federated or collaborative training (they had a project where people could contribute their GPU to train BLOOM). Imagine training the next 1 trillion-parameter model not in one data center, but spread across many participants – HF could coordinate such a swarm (with innovations in synchronizing weights, etc.). In terms of model design, Hugging Face supports Mixture-of-Experts research (they have the Tutel MoE library integration). It wouldn’t be surprising if the next BigScience model uses MoE to push beyond 1 trillion parameters without needing an infeasible amount of GPU memory. HF also works on efficient transformers – e.g. they integrate new architectures like MPT, MLC, etc. They might spearhead an open model that uses retrieval-augmentation natively (like Facebook’s Retro or Atlas models). A “HuggingChat” in the future might automatically search a local knowledge base or the web if a query seems fact-based – all powered by open source. Hugging Face is also big on multimodal pipelines, so one can foresee an open-source equivalent of Gemini: perhaps a model that is trained on text, images, and audio together by the community (there’s already the LAION datasets and multimodal models like CLIP and Flamingo to build on).

Another area HF is pushing is on-device and edge deployment. Their involvement in projects like GPT4All and collaborations with Qualcomm/Apple on running models on phones means they will optimize models to be smaller and faster. Quantization, pruning, distillation – these will be key. We may see a 100B+ parameter model compressed into a 5B effective model through clever distillation, and Hugging Face would be the one to host such a variant for people to try.

In alignment and safety, Hugging Face tends to provide tools rather than impose their own tuning. They host both “censored” and “uncensored” model versions (with warnings). They might develop modular safety filters that anyone can plug into their model – e.g. an open-source “toxic content detector” that can work with any LLM output. This fits their philosophy of transparency: rather than hide the model weights or hard-code refusals, give users the building blocks to moderate as they need.

Scaling trends in the open-source world that HF will likely promote include: long context models (there are already open models with 100k context using efficient attention or segment memory), feedback loops where models evaluate other models (e.g. OpenAI published “Guiding GPT-4 with GPT-4” techniques, which open projects will adopt), and specialization – instead of one model doing everything, there will be a family of models each fine-tuned for a domain (medical, legal, coding, etc.), and HF Hub is the place where they live. The “deep research products” from HF are thus the platform and tools enabling these trends: their Transformer library quickly incorporates new research (like FlashAttention for speed, or RoPE for long context), so the community can build on the latest ideas.

Design trends shaping the future: Across all these players, we see converging themes. Models will be more grounded in real-world data (through retrieval, tools, and multimodal inputs) rather than being standalone prediction engines. They will be more efficient in size – either via smarter training (Chinchilla paradigm) or architecture (experts, compression) – so we get better quality without a proportional increase in cost. There’s a focus on interaction with external systems: whether it’s searching the web, controlling apps, or writing and executing code, the future AI will not be a sealed box but an intelligent orchestrator of resources. This is evident from OpenAI plugins, Google extensions, Perplexity’s tool-like modes, etc. Another trend is scalability and deployment versatility: Google making Gemini run from data center down to phone, OpenAI offering GPT-3.5 fine-tuned variants for specific tasks, Hugging Face quantizing models for browsers – the AI models are being designed to scale out (different sizes, different hardware) not just up.

Finally, we anticipate a continued refinement in alignment – using human and AI feedback to ensure these models are accurate and behave as intended. The techniques will likely evolve (e.g. Direct Preference Optimization was proposed as a simpler alternative to RLHF, and such innovations will be quickly adopted in practice). Each company’s proprietary blend of these techniques will define their products’ strengths. For example, OpenAI might focus on generalist intelligence with top-notch alignment, Google on integrated AI that leverages Google’s ecosystem (knowledge graph, services), xAI on an uncensored truth-seeking assistant, Perplexity on the ultimate AI research tool with guaranteed sources, and Hugging Face on open, accessible versions of all such capabilities.

In conclusion, the core technologies – RAG, fine-tuning, RLHF, search integration, multimodal models, and efficiency optimizations – are the foundation that all these organizations build upon ([What to Know About RAG LLM, Perplexity, and AI Search.](https://blog.phospho.ai/how-does-ai-powered-search-work-explaining-rag-llm-and-perplexity/#:~:text=Perplexity%20AI%20applies%20the%20RAG,the%20most%20complex%20user%20queries)) (). The differences come in how they combine and prioritize them. We’re witnessing a convergence where **language models become more like interactive knowledge bases, and each of these players is contributing to that future in their own way. The likely design trend is an AI that can think, look up, and interact, not just generate text – and achieving this will require continued innovation in training methodology, architectural modularity, and integration with the vast world of data and tools around us.

Sources:

1. OpenAI, “GPT-4 Technical Report”, pp. 1–2 (2023) – Transformer-based multimodal model; post-training alignment via RLHF; scalable infrastructure () ().
2. OpenAI, “ChatGPT: Optimizing Language Models for Dialogue” (2022) – Use of supervised fine-tuning and RLHF on GPT-3.5; model trained on Azure supercomputers ([ChatGPT](https://chatgpt.r4wand.eu.org/#:~:text=Methods)) (ChatGPT).
3. Google, “Introducing Gemini: our largest and most capable AI model” (Nov 2024) – Native multimodal training from text, code, images, audio; sizes (Ultra/Pro/Nano) and use of fine-tuning + RLHF for Gemini ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=Gemini%20is%20the%20result%20of,code%2C%20audio%2C%20image%20and%20video)) ([Introducing Gemini: Google’s most capable AI model yet](https://blog.google/technology/ai/google-gemini-ai/#:~:text=For%20Gemini%20Ultra%2C%20we%E2%80%99re%20currently,before%20making%20it%20broadly%20available)).
4. Google Bard Update – Google Search integration to double-check Bard’s answers (Sep 2023) – Bard can use Google it (the “G” button) to find supporting evidence for its response ([Google Search can now validate responses from Bard](https://searchengineland.com/google-search-can-now-validate-responses-from-bard-432075#:~:text=When%20you%20Google%20the%20answer,contradicting%20information%20found%20by%20Search)).
5. Perplexity AI – Interview with Perplexity team (Unite.AI, Lex Fridman podcast) summarized by Ethan Lazuk (2024) – End-to-end system with multiple LLMs; parallel retrieval and answer generation; custom crawler and ranking focusing on authoritative sources ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,AI%20Interview%20Series)) ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Perplexity%E2%80%99s%20search%20index%20is%20smaller,to%20be%20relevant%20and%20trustworthy)).
6. Perplexity AI – Phospho.ai blog “What to know about RAG LLM, Perplexity, and AI search” (2024) – Describes Perplexity’s use of Retrieval-Augmented Generation to provide up-to-date, source-backed answers ([What to Know About RAG LLM, Perplexity, and AI Search.](https://blog.phospho.ai/how-does-ai-powered-search-work-explaining-rag-llm-and-perplexity/#:~:text=Perplexity%20AI%20applies%20the%20RAG,the%20most%20complex%20user%20queries)).
7. Perplexity AI – System details (E. Lazuk blog continued) – Strict grounding of answers in retrieved documents (no model-made assertions) to ensure factual accuracy ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=,434%20%282%3A08%3A56)); snippet extraction and citation inclusion process ([How Does Perplexity Work? A Summary from an SEO's Perspective - Ethan Lazuk](https://ethanlazuk.com/blog/how-does-perplexity-work/#:~:text=Once%20relevant%20documents%20are%20retrieved%2C,the%20most%20contextually%20relevant%20sections)).
8. xAI – The Decoder news: “xAI releases Grok-1 (314B MoE) open source” (Mar 2024) – Grok-1 architecture (8 experts, 2 active per token) ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=As%20announced%2C%20Grok,also%20based%20on%20this%20architecture)); base model trained without RLHF and released for openness ([Elon Musk's xAI releases Grok-1, the largest open source mixture-of-experts model to date](https://the-decoder.com/elon-musks-xai-releases-grok-1-the-largest-open-source-mixture-of-experts-model-to-date/#:~:text=Grok,any%20ethical%20or%20safety%20guidelines)).
9. xAI – Writesonic blog: “What is Grok 3?” (Feb 2025) – Highlights Grok 3’s training on real-time data via X and web, chain-of-thought reasoning through trial-and-error reinforcement learning ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Real)) ([What is Grok 3? A Detailed Guide to the AI Model [+Examples]](https://writesonic.com/blog/what-is-grok-3#:~:text=Advanced%20Reasoning%20with%20Reinforcement%20Learning)).
10. Hugging Face – “The Technology Behind BLOOM Training” (BigScience, 2022) – Engineering for training a 176B param model: 384 A100 GPUs, Megatron-DeepSpeed, 350B tokens, GPT-3-like architecture ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)).
11. Hugging Face – TRL (Transformer RL) library docs (2022) – Emphasizes RLHF as a go-to method for aligning LLMs with human preferences, and tools to do this at various model scales ([Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU](https://huggingface.co/blog/trl-peft#:~:text=Fine,very%20powerful%20AI%20systems)).
12. Hugging Face – BigScience BLOOM release (2022) – An open 176B model with GPT-3 architecture “with extras,” demonstrating that non-corporate entities can train large models and openly share them ([The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#:~:text=Here%27s%20a%20quick%20summary%20of,project)).