#
Build an agent with tool-calling superpowers 🦸 using smolagents

Authored by: Aymeric Roucher

This notebook demonstrates how you can use smolagents to build awesome agents!

What are agents? Agents are systems that are powered by an LLM and enable the LLM (with careful prompting and output parsing) to use specific tools to solve problems.

These tools are basically functions that the LLM couldn’t perform well by itself: for instance for a text-generation LLM like Llama-3-70B, this could be an image generation tool, a web search tool, a calculator…

What is smolagents? It’s an library that provides building blocks to build your own agents! Learn more about it in the documentation.

Let’s see how to use it, and which use cases it can solve.

Run the line below to install required dependencies:

    !pip install smolagents datasets langchain sentence-transformers faiss-cpu duckduckgo-search openai langchain-community --upgrade -q

Let’s login in order to call the HF Inference API:

    from huggingface_hub import notebook_login

    notebook_login()

##
1. 🏞️ Multimodal + 🌐 Web-browsing assistant

For this use case, we want to show an agent that browses the web and is able to generate images.

To build it, we simply need to have two tools ready: image generation and web search.

- For image generation, we load a tool from the Hub that uses the HF Inference API (Serverless) to generate images using Stable Diffusion.
- For the web search, we use a built-in tool.

    >>> from smolagents import load_tool, CodeAgent, HfApiModel, DuckDuckGoSearchTool

    >>> # Import tool from Hub
    >>> image_generation_tool = load_tool("m-ric/text-to-image", trust_remote_code=True)

    >>> search_tool = DuckDuckGoSearchTool()

    >>> model = HfApiModel("Qwen/Qwen2.5-72B-Instruct")
    >>> # Initialize the agent with both tools
    >>> agent = CodeAgent(tools=[image_generation_tool, search_tool], model=model)

    >>> # Run it!
    >>> result = agent.run(
    ...     "Generate me a photo of the car that James bond drove in the latest movie.",
    ... )
    >>> result

    TOOLCODE:
     from smolagents import Tool
    from huggingface_hub import InferenceClient

    class TextToImageTool(Tool):
        description = "This tool creates an image according to a prompt, which is a text description."
        name = "image_generator"
        inputs = &#123;"prompt": &#123;"type": "string", "description": "The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc."}}
        output_type = "image"
        model_sdxl = "black-forest-labs/FLUX.1-schnell"
        client = InferenceClient(model_sdxl)

        def forward(self, prompt):
            return self.client.text_to_image(prompt)

##
2. 📚💬 RAG with Iterative query refinement & Source selection

Quick definition: Retrieval-Augmented-Generation (RAG) is “using an LLM to answer a user query, but basing the answer on information retrieved from a knowledge base”.

This method has many advantages over using a vanilla or fine-tuned LLM: to name a few, it allows to ground the answer on true facts and reduce confabulations, it allows to provide the LLM with domain-specific knowledge, and it allows fine-grained control of access to information from the knowledge base.

- Now let’s say we want to perform RAG, but with the additional constraint that some parameters must be dynamically generated. For example, depending on the user query we could want to restrict the search to specific subsets of the knowledge base, or we could want to adjust the number of documents retrieved. The difficulty is: how to dynamically adjust these parameters based on the user query?

- A frequent failure case of RAG is when the retrieval based on the user query does not return any relevant supporting documents. Is there a way to iterate by re-calling the retriever with a modified query in case the previous results were not relevant?

🔧 Well, we can solve the points above in a simple way: we will give our agent control over the retriever’s parameters!

➡️ Let’s show how to do this. We first load a knowledge base on which we want to perform RAG: this dataset is a compilation of the documentation pages for many `huggingface` packages, stored as markdown.

    import datasets

    knowledge_base = datasets.load_dataset("m-ric/huggingface_doc", split="train")

Now we prepare the knowledge base by processing the dataset and storing it into a vector database to be used by the retriever. We are going to use LangChain, since it features excellent utilities for vector databases:

    from langchain.docstore.document import Document
    from langchain.text_splitter import RecursiveCharacterTextSplitter
    from langchain.vectorstores import FAISS
    from langchain_community.embeddings import HuggingFaceEmbeddings

    source_docs = [
        Document(page_content=doc["text"], metadata={"source": doc["source"].split("/")[1]}) for doc in knowledge_base
    ]

    docs_processed = RecursiveCharacterTextSplitter(chunk_size=500).split_documents(source_docs)[:1000]

    embedding_model = HuggingFaceEmbeddings(model_name="thenlper/gte-small")
    vectordb = FAISS.from_documents(documents=docs_processed, embedding=embedding_model)

Now that we have the database ready, let’s build a RAG system that answers user queries based on it!

We want our system to select only from the most relevant sources of information, depending on the query.

Our documentation pages come from the following sources:

    >>> all_sources = list(set([doc.metadata["source"] for doc in docs_processed]))
    >>> print(all_sources)

    ['datasets-server', 'datasets', 'optimum', 'gradio', 'blog', 'course', 'hub-docs', 'pytorch-image-models', 'peft', 'evaluate', 'diffusers', 'hf-endpoints-documentation', 'deep-rl-class', 'transformers']

👉 Now let’s build a `RetrieverTool` that our agent can leverage to retrieve information from the knowledge base.

Since we need to add a vectordb as an attribute of the tool, we cannot simply use the simple tool constructor with a `@tool` decorator: so we will follow the advanced setup highlighted in the advanced agents documentation.

    import json
    from smolagents import Tool
    from langchain_core.vectorstores import VectorStore

    class RetrieverTool(Tool):
        name = "retriever"
        description = (
            "Retrieves some documents from the knowledge base that have the closest embeddings to the input query."
        )
        inputs = {
            "query": {
                "type": "string",
                "description": "The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.",
            },
            "source": {"type": "string", "description": ""},
            "number_of_documents": {
                "type": "string",
                "description": "the number of documents to retrieve. Stay under 10 to avoid drowning in docs",
            },
        }
        output_type = "string"

        def __init__(self, vectordb: VectorStore, all_sources: str, **kwargs):
            super().__init__(**kwargs)
            self.vectordb = vectordb
            self.inputs["source"]["description"] = (
                f"The source of the documents to search, as a str representation of a list. Possible values in the list are: {all_sources}. If this argument is not provided, all sources will be searched.".replace(
                    "'", "`"
                )
            )

        def forward(self, query: str, source: str = None, number_of_documents=7) -> str:
            assert isinstance(query, str), "Your search query must be a string"
            number_of_documents = int(number_of_documents)

            if source:
                if isinstance(source, str) and "[" not in str(source):  # if the source is not representing a list
                    source = [source]
                source = json.loads(str(source).replace("'", '"'))

            docs = self.vectordb.similarity_search(
                query,
                filter=({"source": source} if source else None),
                k=number_of_documents,
            )

            if len(docs) == 0:
                return "No documents found with this filtering. Try removing the source filter."
            return "Retrieved documents:\n\n" + "\n===Document===\n".join([doc.page_content for doc in docs])

###
Optional: Share your Retriever tool to Hub

To share your tool to the Hub, first copy-paste the code in the RetrieverTool definition cell to a new file named for instance `retriever.py`.

When the tool is loaded from a separate file, you can then push it to the Hub using the code below (make sure to login with a `write` access token)

    share_to_hub = True

    if share_to_hub:
        from huggingface_hub import login
        from retriever import RetrieverTool

        login("your_token")

        tool = RetrieverTool(vectordb, all_sources)

        tool.push_to_hub(repo_id="m-ric/retriever-tool")

        # Loading the tool
        from smolagents import load_tool

        retriever_tool = load_tool("m-ric/retriever-tool", vectordb=vectordb, all_sources=all_sources)

###
Run the agent!

    from smolagents import HfApiModel, ToolCallingAgent

    model = HfApiModel("Qwen/Qwen2.5-72B-Instruct")

    retriever_tool = RetrieverTool(vectordb=vectordb, all_sources=all_sources)
    agent = ToolCallingAgent(tools=[retriever_tool], model=model, verbose=0)

    agent_output = agent.run("Please show me a LORA finetuning script")

    print("Final output:")
    print(agent_output)

What happened here? First, the agent launched the retriever with specific sources in mind (`['transformers', 'blog']`).

But this retrieval did not yield enough results ⇒ no problem! The agent could iterate on previous results, so it just re-ran its retrieval with less restrictive search parameters. Thus the research was successful!

Note that using an LLM agent that calls a retriever as a tool and can dynamically modify the query and other retrieval parameters is a more general formulation of RAG, which also covers many RAG improvement techniques like iterative query refinement.

##
3. 💻 Debug Python code

Since the CodeAgent has a built-in Python code interpreter, we can use it to debug our faulty Python script!

    from smolagents import CodeAgent

    agent = CodeAgent(tools=[], model=HfApiModel("Qwen/Qwen2.5-72B-Instruct"))

    code = """
    numbers=[0, 1, 2]

    for i in range(4):
        print(numbers(i))
    """

    final_answer = agent.run(
        "I have some code that creates a bug: please debug it, then run it to make sure it works and return the final code",
        additional_args=dict(code=code),
    )

As you can see, the agent tried the given code, gets an error, analyses the error, corrects the code and returns it after veryfing that it works!

And the final code is the corrected code:

    >>> print(final_answer)

    numbers=[0, 1, 2]

    for i in range(len(numbers)):
        print(numbers[i])

##
➡️ Conclusion

The use cases above should give you a glimpse into the possibilities of our Agents framework!

For more advanced usage, read the documentation.

All feedback is welcome, it will help us improve the framework! 🚀

< > Update on GitHub