Upload 5 files

model/biobert.ipynb

@@ -0,0 +1,563 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "09df7e79-4112-4a17-b24f-2e9faa6094d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import torch\n",
+    "from torch.utils.data import DataLoader, Dataset\n",
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "from sklearn.metrics.pairwise import cosine_similarity\n",
+    "from tqdm import tqdm\n",
+    "import os\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9270dae8",
+   "metadata": {},
+   "source": [
+    "# Loading the dataset\n",
+    "After performing the cleaning on uscase_1_.csv using clean.ipynb and then merging the criteria field into the csv using the merged.ipynb and the filtered_combined.xlsx was achieved as the output.\n",
+    "\n",
+    "The following code loads an Excel file into a pandas DataFrame.\n",
+    " - `file_name` specifies the path to the Excel file.\n",
+    " - `pd.read_excel(file_name)` reads the Excel file and stores its content in the DataFrame `df`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "df57d056-b6dc-4512-8b40-aa0d77581d5b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Loading the dataset\n",
+    "file_name = \"../Data/filtered_combined.xlsx\"\n",
+    "df = pd.read_excel(file_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a05a0104",
+   "metadata": {},
+   "source": [
+    "# Connecting to the available device\n",
+    "\n",
+    "The following code detects if a GPU is available and sets the device accordingly. If a GPU is available, it sets the device to \"cuda\"; otherwise, it sets the device to \"cpu\".\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "de885007-b219-4a94-b004-c05c448afae5",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Using device: cuda\n"
+     ]
+    }
+   ],
+   "source": [
+    "\n",
+    "# Detecting if GPU is available and setting the device accordingly\n",
+    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
+    "print(f\"Using device: {device}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cab709e3",
+   "metadata": {},
+   "source": [
+    "# Initializing the BioBERT Model and Tokenizer\n",
+    "\n",
+    "In this section, we initialize the BioBERT model and tokenizer. BioBERT is a pre-trained biomedical language representation model designed to handle various biomedical text mining tasks. It is based on BERT (Bidirectional Encoder Representations from Transformers) and has been further pre-trained on large-scale biomedical corpora.\n",
+    "\n",
+    "## Model Details\n",
+    "\n",
+    "- **Model Name**: `dmis-lab/biobert-base-cased-v1.1`\n",
+    "- **Architecture**: BERT (Bidirectional Encoder Representations from Transformers)\n",
+    "- **Pre-training**: The model has been pre-trained on PubMed abstracts and PMC full-text articles, making it highly suitable for biomedical text mining tasks.\n",
+    "\n",
+    "The BioBERT model has the following architecture details:\n",
+    "\n",
+    "- **Number of Layers**: 12\n",
+    "- **Hidden Size**: 768\n",
+    "- **Number of Attention Heads**: 12\n",
+    "- **Total Parameters**: 110M\n",
+    "\n",
+    "These architectural details make BioBERT a powerful model for understanding and processing biomedical text, leveraging the transformer architecture to capture complex patterns and relationships within the data.\n",
+    "\n",
+    "## Code Explanation\n",
+    "\n",
+    "The following code initializes the BioBERT model and tokenizer:\n",
+    "\n",
+    "```python\n",
+    "model_name = \"dmis-lab/biobert-base-cased-v1.1\"\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
+    "model = AutoModel.from_pretrained(model_name).to(device)  \n",
+    "```\n",
+    "\n",
+    "### Steps:\n",
+    "\n",
+    "1. **Model Name**: We specify the model name `dmis-lab/biobert-base-cased-v1.1`.\n",
+    "2. **Tokenizer Initialization**: We use `AutoTokenizer.from_pretrained(model_name)` to load the tokenizer associated with the BioBERT model. The tokenizer is responsible for converting text into tokens that the model can process.\n",
+    "3. **Model Initialization**: We use `AutoModel.from_pretrained(model_name)` to load the pre-trained BioBERT model. This model is capable of generating embeddings for biomedical text.\n",
+    "4. **Device Assignment**: We move the model to the appropriate device (GPU or CPU) using `.to(device)`. This ensures that the model computations are performed on the available hardware, optimizing performance.\n",
+    "\n",
+    "By initializing the BioBERT model and tokenizer, we are now equipped to process and generate embeddings for biomedical text, which will be used in subsequent steps for tasks such as similarity computation and information retrieval."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "95340ec0-8476-498b-b805-c480bced2ff7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initializing the BioBERT model and tokenizer\n",
+    "model_name = \"dmis-lab/biobert-base-cased-v1.1\"\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
+    "model = AutoModel.from_pretrained(model_name).to(device)  # Move model to the device\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4c855f4d",
+   "metadata": {},
+   "source": [
+    "### Initializing Output and Model File Names\n",
+    "\n",
+    "In this cell, we are initializing two variables to store the names of the output and model files.\n",
+    "\n",
+    "- `output_file`: This variable is assigned the string `\"similar_trials_results.xlsx\"`, which indicates that the results of similar trials will be saved in an Excel file with this name.\n",
+    "- `model_file`: This variable is assigned the string `\"biobert_embeddings.pt\"`, which indicates that the BioBERT model embeddings will be saved in a PyTorch file with this name.\n",
+    "\n",
+    "These variables will be used later in the code to save and load the respective files."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "8408d3bc-80a4-4850-8620-d035ace58293",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initializing the output and model file names\n",
+    "output_file = \"similar_trials_results.xlsx\"\n",
+    "model_file = \"biobert_embeddings.pt\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e1efa86c",
+   "metadata": {},
+   "source": [
+    "# ClinicalTrialsDataset Class\n",
+    "\n",
+    "## Description\n",
+    "The `ClinicalTrialsDataset` class is a custom dataset class designed for tokenizing a list of clinical trial texts using a specified tokenizer. It inherits from the `Dataset` class provided by PyTorch.\n",
+    "\n",
+    "## Parameters\n",
+    "- `data` (list): A list of texts to be tokenized.\n",
+    "- `tokenizer` (Tokenizer): A tokenizer object used to tokenize the texts.\n",
+    "- `max_length` (int, optional): The maximum length of the tokenized sequences. Default is 512.\n",
+    "\n",
+    "## Methods\n",
+    "\n",
+    " `__init__(self, data, tokenizer, max_length=512)`\n",
+    "Initializes the dataset with the provided data, tokenizer, and maximum sequence length.\n",
+    "\n",
+    " `__len__(self)`\n",
+    "Returns the number of texts in the dataset.\n",
+    "\n",
+    " `__getitem__(self, idx)`\n",
+    "Tokenizes the text at the specified index and returns the tokenized encoding as tensors. The tensors are moved to the specified device.\n",
+    "\n",
+    "#### Parameters\n",
+    "- `idx` (int): The index of the text to be tokenized.\n",
+    "\n",
+    "#### Returns\n",
+    "- `dict`: A dictionary containing the tokenized encoding as tensors, with each tensor moved to the specified device."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "ae405a5e-d501-4d87-80d6-d33944e2d17f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Defining a dataset class for tokenization\n",
+    "class ClinicalTrialsDataset(Dataset):\n",
+    "    def __init__(self, data, tokenizer, max_length=512):\n",
+    "        self.data = data  # This should be a list of texts\n",
+    "        self.tokenizer = tokenizer\n",
+    "        self.max_length = max_length\n",
+    "\n",
+    "    def __len__(self):\n",
+    "        return len(self.data)\n",
+    "\n",
+    "    def __getitem__(self, idx):\n",
+    "        text = self.data[idx]  # Access list item directly\n",
+    "        encoding = self.tokenizer(\n",
+    "            text,\n",
+    "            max_length=self.max_length,\n",
+    "            padding=\"max_length\",\n",
+    "            truncation=True,\n",
+    "            return_tensors=\"pt\",\n",
+    "        )\n",
+    "        return {key: val.squeeze(0).to(device) for key, val in encoding.items()}  # Return tensors and move to device\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12afd4b1",
+   "metadata": {},
+   "source": [
+    "## Function: generate_embeddings\n",
+    "\n",
+    "### Description\n",
+    "Generates embeddings for a list of texts using the BioBERT model. This function tokenizes the input texts, processes them in batches, and extracts the embeddings from the model's output.\n",
+    "\n",
+    "### Parameters\n",
+    "- **texts** (`list` of `str`): A list of input texts to generate embeddings for.\n",
+    "- **tokenizer** (`transformers.PreTrainedTokenizer`): The tokenizer associated with the BioBERT model, used to convert texts into token IDs.\n",
+    "- **model** (`transformers.PreTrainedModel`): The BioBERT model used to generate embeddings.\n",
+    "- **batch_size** (`int`, optional): The number of samples to process in each batch. Default is 16.\n",
+    "\n",
+    "### Returns\n",
+    "- **torch.Tensor**: A tensor containing the embeddings for the input texts. The shape of the tensor is `(number_of_texts, embedding_dimension)`.\n",
+    "\n",
+    "### Example Usage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "1f65941c-317f-43eb-be61-c58c5ef3f67f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Generate embeddings using BioBERT\n",
+    "def generate_embeddings(texts, tokenizer, model, batch_size=16):\n",
+    "    dataset = ClinicalTrialsDataset(texts, tokenizer)\n",
+    "    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False)\n",
+    "\n",
+    "    embeddings = []\n",
+    "    model.eval()\n",
+    "    with torch.no_grad():\n",
+    "        for batch in tqdm(dataloader, desc=\"Generating embeddings\"):\n",
+    "            input_ids = batch[\"input_ids\"]\n",
+    "            attention_mask = batch[\"attention_mask\"]\n",
+    "\n",
+    "            # Move tensors to the device\n",
+    "            input_ids = input_ids.to(device)\n",
+    "            attention_mask = attention_mask.to(device)\n",
+    "\n",
+    "            outputs = model(input_ids=input_ids, attention_mask=attention_mask)\n",
+    "            cls_embeddings = outputs.last_hidden_state[:, 0, :]\n",
+    "            embeddings.append(cls_embeddings.cpu().numpy())  # Move embeddings back to CPU for numpy conversion\n",
+    "\n",
+    "    return torch.tensor(np.vstack(embeddings))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fa6950e0",
+   "metadata": {},
+   "source": [
+    "# Preprocess the text data\n",
+    " This code performs the following steps:\n",
+    "1. Creates a new column \"Combined_Text\" in the DataFrame `df` by filling any missing values in the \"Combined Column\" with an empty string.\n",
+    "2. Converts the \"Combined_Text\" column into a list of strings and assigns it to the variable `texts`.\n",
+    "\n",
+    " - `df[\"Combined Column\"].fillna(\"\")`: Replaces any NaN values in the \"Combined Column\" with an empty string.\n",
+    " - `df[\"Combined_Text\"].tolist()`: Converts the \"Combined_Text\" column to a list."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "b6f7f631-455c-43c5-9721-acd56f28b31b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Preprocess the text data\n",
+    "df[\"Combined_Text\"] = df[\"Combined Column\"].fillna(\"\")\n",
+    "texts = df[\"Combined_Text\"].tolist()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "86d9fa2b",
+   "metadata": {},
+   "source": [
+    "## Loading or Generating Embeddings\n",
+    "\n",
+    "This code snippet checks if the embeddings are already saved in a file. If the file exists, it loads the embeddings using `torch.load()`. Otherwise, it generates embeddings for all clinical trials, saves them to a file, and informs the user of the save location.\n",
+    "\n",
+    "### Code Explanation:\n",
+    "1. **Check File Existence**: \n",
+    "   - `os.path.exists(model_file)` checks if the embeddings file already exists.\n",
+    "   - If it exists, the embeddings are loaded, and a message is displayed: `\"Loaded embeddings from saved model.\"`.\n",
+    "\n",
+    "2. **Generate and Save Embeddings**:\n",
+    "   - If the file doesn't exist, embeddings are generated using the `generate_embeddings()` function.\n",
+    "   - `torch.save()` saves the generated embeddings to `model_file`.\n",
+    "   - A confirmation message is printed: `\"Embeddings saved to {model_file}\"`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "74aea002-314e-4c01-aed1-42d05badefc7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "C:\\Users\\LENOVO\\AppData\\Local\\Temp\\ipykernel_18736\\4225845386.py:3: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  embeddings = torch.load(model_file)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loaded embeddings from saved model.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Check if embeddings are already saved\n",
+    "if os.path.exists(model_file):\n",
+    "    embeddings = torch.load(model_file)\n",
+    "    print(\"Loaded embeddings from saved model.\")\n",
+    "else:\n",
+    "    # Generate embeddings for all clinical trials\n",
+    "    embeddings = generate_embeddings(texts, tokenizer, model)\n",
+    "    torch.save(embeddings, model_file)\n",
+    "    print(f\"Embeddings saved to {model_file}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "38232b28",
+   "metadata": {},
+   "source": [
+    "## Retrieving Top N Similar Clinical Trials\n",
+    "\n",
+    "This function computes the top N similar clinical trials based on cosine similarity between embeddings. \n",
+    "\n",
+    "### Code Functionality:\n",
+    "1. **Input Parameters**:\n",
+    "   - `query_embedding`: The embedding of the query trial for which similar trials are to be found.\n",
+    "   - `embeddings`: The embeddings of all clinical trials.\n",
+    "   - `top_n`: The number of similar trials to retrieve (default is 10).\n",
+    "\n",
+    "2. **Steps in the Function**:\n",
+    "   - **Convert to CPU and Numpy**:\n",
+    "     - Move `query_embedding` and `embeddings` to CPU (if not already) using `.cpu()` to ensure compatibility with `cosine_similarity`.\n",
+    "     - Convert them to NumPy arrays using `.numpy()`.\n",
+    "   - **Compute Cosine Similarity**:\n",
+    "     - Use `cosine_similarity` from sklearn to compute the similarity between the query embedding and all other embeddings.\n",
+    "   - **Retrieve Indices of Similar Trials**:\n",
+    "     - Sort the similarity scores and retrieve indices of the top N most similar trials using `argsort` (in descending order).\n",
+    "\n",
+    "3. **Output**:\n",
+    "   - The function returns the indices of the top N most similar trials."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "5951ac38-149f-4d63-8150-4dd53d3ec7a8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.metrics.pairwise import cosine_similarity\n",
+    "import pandas as pd\n",
+    "\n",
+    "def get_similar_trials(query_embedding, embeddings, top_n=10):\n",
+    "    # Ensure both tensors are on the CPU before calling cosine_similarity\n",
+    "    query_embedding_cpu = query_embedding.cpu().detach().numpy()\n",
+    "    embeddings_cpu = embeddings.cpu().detach().numpy()\n",
+    "\n",
+    "    # Compute cosine similarity between the query and all embeddings\n",
+    "    similarities = cosine_similarity(query_embedding_cpu, embeddings_cpu)\n",
+    "    \n",
+    "    # Get the indices of the top_n most similar trials (excluding the query itself)\n",
+    "    similar_indices = similarities.argsort(axis=1)[:, -top_n-1:-1][:, ::-1]\n",
+    "    \n",
+    "    return similar_indices\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "087ef2e5",
+   "metadata": {},
+   "source": [
+    "## Specifying Clinical Trials for Evaluation\n",
+    "This list contains the NCT IDs of the trials on which the project is to be tested.\n",
+    "\n",
+    "**Attributes**:\n",
+    "   - `evaluation_trials`(list): A list of strings representing the NCT IDs of the trials."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "257628d0-22af-494a-9c83-88525e1267fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Trials to evaluate\n",
+    "evaluation_trials = [\"NCT00385736\", \"NCT00386607\", \"NCT03518073\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a7776f99",
+   "metadata": {},
+   "source": [
+    "\n",
+    "### Create a mapping of NCT IDs to indices.\n",
+    "\n",
+    "This dictionary comprehension iterates over the `nct_id` column of the DataFrame `df`, \n",
+    "enumerates the values, and creates a dictionary where each NCT ID is mapped to its \n",
+    "corresponding index.\n",
+    "\n",
+    "**Returns**:\n",
+    "\n",
+    "   - `nct_id_to_index `: A dictionary with NCT IDs as keys and their respective indices as values.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "4854660f-1e28-444a-929b-b40258d90b49",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a mapping of NCT IDs to indices\n",
+    "nct_id_to_index = {nct_id: idx for idx, nct_id in enumerate(df[\"nct_id\"])}\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "814bcdd6",
+   "metadata": {
+    "vscode": {
+     "languageId": "markdown"
+    }
+   },
+   "source": [
+    "## Generating Similar Trials for Evaluation NCT IDs\n",
+    "\n",
+    "The following code generates similar trials for a list of evaluation NCT IDs and saves the results to an Excel sheet.\n",
+    "\n",
+    "### Code Explanation\n",
+    "\n",
+    "1. **Initialize Output List**:\n",
+    "    - `output_data = []`: A list to collect the results for each NCT ID.\n",
+    "\n",
+    "2. **Iterate Over Evaluation Trials**:\n",
+    "    - For each `trial_id` in `evaluation_trials`:\n",
+    "      - Check if the `trial_id` exists in the `nct_id_to_index` dictionary.\n",
+    "      - Retrieve the index of the query trial and its embedding.\n",
+    "      - Move the query embedding to the appropriate device (GPU or CPU).\n",
+    "\n",
+    "3. **Get Similar Trials**:\n",
+    "    - Use the `get_similar_trials` function to get the indices of similar trials based on cosine similarity.\n",
+    "\n",
+    "4. **Retrieve and Process Similar Trials**:\n",
+    "    - Retrieve the similar trials from the DataFrame using the indices.\n",
+    "    - Calculate and store similarity scores for each similar trial.\n",
+    "    - Add the query NCT ID as a new column to track which trial it corresponds to.\n",
+    "    - Append the results to the `output_data` list.\n",
+    "\n",
+    "5. **Combine and Save Results**:\n",
+    "    - Combine all results into a single DataFrame.\n",
+    "    - Save the results to an Excel sheet using `pd.ExcelWriter`.\n",
+    "\n",
+    "### Code\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16c402ba-90f5-4e90-82f0-f22093ead740",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Generate similar trials for evaluation NCT IDs\n",
+    "output_data = []  # List to collect the results for each NCT ID\n",
+    "\n",
+    "for trial_id in evaluation_trials:\n",
+    "    if trial_id in nct_id_to_index:\n",
+    "        query_idx = nct_id_to_index[trial_id]\n",
+    "        query_embedding = embeddings[query_idx].unsqueeze(0).to(device)  # Move query embedding to device\n",
+    "        \n",
+    "        # Get similar trial indices\n",
+    "        similar_indices = get_similar_trials(query_embedding, embeddings)\n",
+    "\n",
+    "        # Retrieve the similar trials from the DataFrame\n",
+    "        similar_trials = df.iloc[similar_indices[0]]\n",
+    "        \n",
+    "        # Calculate and store similarity scores\n",
+    "        similar_trials[\"Similarity_Score\"] = [\n",
+    "            cosine_similarity(query_embedding.cpu().detach().numpy().reshape(1, -1), embeddings[idx].cpu().detach().numpy().reshape(1, -1)).item()\n",
+    "            for idx in similar_indices[0]\n",
+    "        ]\n",
+    "        \n",
+    "        # Add the NCT ID (trial_id) as a new column to track which trial it corresponds to\n",
+    "        similar_trials[\"Query_NCT_ID\"] = trial_id\n",
+    "        \n",
+    "        # Append the results to the output list\n",
+    "        output_data.append(similar_trials)\n",
+    "\n",
+    "# Combine all results into a single DataFrame\n",
+    "final_results = pd.concat(output_data, ignore_index=True)\n",
+    "\n",
+    "# Save the results to an Excel sheet\n",
+    "output_file = \"similar_trials_with_nct_id.xlsx\"\n",
+    "with pd.ExcelWriter(output_file, engine='xlsxwriter') as output_writer:\n",
+    "    final_results.to_excel(output_writer, index=False, sheet_name='Similar Trials')\n",
+    "\n",
+    "print(f\"Similar trials with NCT IDs saved to {output_file}\")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "pytorch_env",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

model/clean_file.ipynb

@@ -0,0 +1,140 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Data Cleaning Workflow\n",
+        "\n",
+        "This Jupyter Notebook demonstrates the process of cleaning a dataset. The steps include loading the dataset, removing rows with garbage values, and saving the cleaned dataset."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 1: Import necessary libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "jyDxPzc_zI5U"
+      },
+      "outputs": [],
+      "source": [
+        "import pandas as pd\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 2: Load the dataset\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 335
+        },
+        "id": "9Lh3U2P5zJZp",
+        "outputId": "d390e6e4-364d-4140-fd13-4795f189e26a"
+      },
+      "outputs": [],
+      "source": [
+        "# Load the dataset\n",
+        "file_path = \"./usecase_1_merged.csv\"\n",
+        "df = pd.read_csv(file_path)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 3: Remove garbage value rows based on 'Unnamed: 0.1'"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 146
+        },
+        "id": "pHkNZFlmzMIo",
+        "outputId": "0c28cdb8-75f5-49ec-b37d-328867a3d00d"
+      },
+      "outputs": [],
+      "source": [
+        "# Step 1: Remove garbage value rows based on 'Unnamed: 0.1' (keep only numeric values)\n",
+        "df = df[pd.to_numeric(df['Unnamed: 0.1'], errors='coerce').notnull()]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 4: Remove garbage value rows based on 'nct_id'"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 146
+        },
+        "id": "MBdaKv2gzUWA",
+        "outputId": "b9923f80-01e2-461b-fd7f-a36db9d36332"
+      },
+      "outputs": [],
+      "source": [
+        "# Step 2: Remove garbage value rows based on 'nct_id' (keep only values starting with \"NCT\" followed by numbers)\n",
+        "df = df[df['nct_id'].str.match(r'^NCT\\d+$', na=False)]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 5: Save the cleaned dataset"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b3QaaKdhzU2O"
+      },
+      "outputs": [],
+      "source": [
+        "# Save the cleaned dataset\n",
+        "cleaned_file_path = \"usecase_1_merged_cleaned.csv\"\n",
+        "df.to_csv(cleaned_file_path, index=False)\n",
+        "\n",
+        "print(f\"Cleaned data saved to {cleaned_file_path}\")"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

model/merge.ipynb

@@ -0,0 +1,129 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "be6478b2",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# Data Loading and Preparation\n",
+    "\n",
+    "In this section, we will load and prepare the data from two sources: `eligibilities.txt` and `usecase_1_.csv`.\n",
+    "\n",
+    "## Steps:\n",
+    "\n",
+    "1. **Import the pandas library**:\n",
+    "\n",
+    "2. **Load the `eligibilities.txt` data**:\n",
+    "    - Use the `read_csv` method from pandas to load the data.\n",
+    "    - Specify the separator as `|`.\n",
+    "\n",
+    "\n",
+    "3. **Select the necessary columns**:\n",
+    "    - We are interested in the `nct_id` and `criteria` columns.\n",
+    "\n",
+    "\n",
+    "4. **Load the `usecase_1_.csv` data**:\n",
+    "    - Use the `read_csv` method from pandas to load the data.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "20cfb7ee-0fd8-4b37-bae1-5ab98125ad10",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The column 'criteria' has been added to usecase_1_.csv and saved as usecase_1_merged.csv.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "# Load the eligibilities.txt data\n",
+    "eligibilities = pd.read_csv('../eligibilities.txt', sep='|')\n",
+    "\n",
+    "# Select the necessary columns\n",
+    "eligibilities = eligibilities[['nct_id', 'criteria']]\n",
+    "\n",
+    "# Load the usecase_1_.csv data\n",
+    "usecase = pd.read_csv('../usecase_1_.csv')\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e0c015c5",
+   "metadata": {
+    "vscode": {
+     "languageId": "markdown"
+    }
+   },
+   "source": [
+    "# Data Merging and Saving\n",
+    "\n",
+    "In this section, we will merge the datasets and save the merged data to a new CSV file.\n",
+    "\n",
+    "## Steps:\n",
+    "\n",
+    "1. **Rename the column in `usecase`**:\n",
+    "    - Rename the column **'NCT Number'** to **'nct_id'** for merging.\n",
+    "\n",
+    "2. **Merge the datasets**:\n",
+    "    - Merge the `usecase` and `eligibilities` datasets on the **'nct_id'** column.\n",
+    "    - Use a left join to ensure all records from `usecase` are retained.\n",
+    "\n",
+    "3. **Save the merged data**:\n",
+    "    - Save the merged data to a new CSV file named **'usecase_1_merged.csv'**.\n",
+    "    - Do not include the index in the saved file.\n",
+    "\n",
+    "4. **Confirmation**:\n",
+    "    - Print a message to confirm that the column **'criteria'** has been added and the file has been saved."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b5fe97ff-8301-4b2b-9a2c-8564f9912054",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Rename 'NCT Number' in usecase to 'nct_id' for merging\n",
+    "usecase.rename(columns={'NCT Number': 'nct_id'}, inplace=True)\n",
+    "\n",
+    "# Merge the datasets on 'nct_id'\n",
+    "merged_data = usecase.merge(eligibilities, on='nct_id', how='left')\n",
+    "\n",
+    "# Save the merged data to a new CSV\n",
+    "merged_data.to_csv('usecase_1_merged.csv', index=False)\n",
+    "\n",
+    "print(\"The column 'criteria' has been added to usecase_1_.csv and saved as usecase_1_merged.csv.\")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}