# ChEBI Bulk Submission Workflow for GlyGen Glycans This workflow prepares glycan data from the GlyGen database for bulk submission to ChEBI. It filters glycans based on specific criteria, generates chemical structure files (SDF format), and cleans the structures for ChEBI compliance. ## Overview The workflow consists of three main scripts that should be run in sequence: 1. **`step1_make_chebi_bulk_sub.py`** - Filters glycan data and prepares metadata 2. **`step2_generate_chebi_sdf.py`** - Generates SDF file with chemical structures 3. **`step3_clean_sdf_remove_hydrogens.py`** - Post-processes SDF to remove hydrogens 4. **`step4_validate_sdf_file.py`** - Validation script for ChEBI SDF file. ## Requirements ### Python Dependencies ```bash pip install pandas rdkit chembl-structure-pipeline ``` ### Required Input Files #### For Data Filtering (`step1_make_chebi_bulk_sub.py`): - `glycan_monosaccharide_composition.csv` - Main glycan composition data - `glycan_xref_chebi.csv` - Glycans with existing ChEBI IDs (to exclude) - `glycan_xref_pubchem.csv` - PubChem cross-references - `glycan_masterlist.csv` - Glycan type classifications - `glycan_sequences_wurcs.csv` - WURCS notation sequences - `glycan_sequences_iupac_extended.csv` - IUPAC extended sequences - `glycan_citations_*.csv` - All citation files (biomarkers, glytoucan, motif, ncfg, species) - Note: `.stat.csv` files are automatically excluded - Image directory with SVG files (optional for testing, required for production) #### For SDF Generation (`step2_generate_chebi_sdf.py`): - `glycans_for_chebi.csv` - Output from Step 1 - `pubchem_glytoucan_compounds.sdf` - PubChem compound structures - `pubchem_glytoucan_substances.sdf` - PubChem substance structures #### For Hydrogen Removal (`step3_clean_sdf_remove_hydrogens.py`): - `glycans_chebi_bulk_submission.sdf` - Output from Step 2 ## Workflow ### Step 1: Filter and Prepare Glycan Data ```bash python3 step1_make_chebi_bulk_sub.py ``` **What it does:** - Filters glycans where `aldi = 0`, `Xxx = '0'`, and `X = 0` - Removes glycans that already have ChEBI IDs - Keeps only glycans with PubChem IDs (adds `pubchem_id` and `pubchem_type` columns) - Duplicates rows for glycans with both compound and substance PubChem entries - **REQUIRED:** Filters for glycans with SVG images (errors if images folder doesn't exist or is empty) - Collects PubMed IDs from all citation files (pipe-separated) - Adds `glytoucan_type` from masterlist - Removes glycans without `glytoucan_type` - Adds WURCS sequences (`sequence_wurcs`) - Adds IUPAC extended sequences (`sequence_iupac_extended`) - Generates `definition` text based on composition and type - Generates `relationship` (ISA number) based on type **Output:** `glycans_for_chebi.csv` with 20+ columns **Output Location:** `/data/projects/glygen/downloads/chebi_bulk_sub/YYYY_MM_DD/` (date-based folder) ### Step 2: Generate SDF File **First download the PubChem SDF compound and substance files from the GlyTouCan Project** 1. Use this query to find all the PubChem entries from the GlyTouCan project - https://pubchem.ncbi.nlm.nih.gov/#query=%22GlyTouCan%20Project%22 2. Select the "Compounds" tab. Then select the "Download" option on the right, then select to download the Chemical Structure records in an SDF format, and 2D coordinate type. Place in the same folder as the "glycans_for_chebi.csv" (/data/projects/glygen/downloads/chebi_bulk_sub/*latest* folder) and change the filename to "pubchem_glytoucan_compounds.sdf". 3. Repeat step 2 for the "Substances" tab in the PubChem results, but change the name to "pubchem_glytoucan_substances.sdf". ```bash python3 step2_generate_chebi_sdf.py ``` **What it does:** - Reads filtered glycan data - Parses PubChem SDF files to extract chemical structures - Matches PubChem IDs to structures (compound first, then substance) - Generates complete SDF entries with: - Chemical structure (from PubChem or empty block) - `` - TEMP_{glytoucan_ac} - `` - Generated description - `` - GlyTouCan {glytoucan_ac} - `` - WURCS notation (if available) - `` - IUPAC extended sequence (if available) - `` - ISA number - `` - GlyGen, GlyTouCan, and PubChem accessions - `` - PubMed IDs (semicolon-separated) **Output:** `glycans_chebi_bulk_submission.sdf` **Statistics shown:** - Number of structures found vs missing - Total entries generated ### Step 3: Remove Hydrogens (Clean SDF) ```bash python3 step3_clean_sdf_remove_hydrogens.py ``` **What it does:** - Uses ChEBI's libRDChEBI methodology - Transforms R-group aliases to proper R-group representations - Removes hydrogen atoms while preserving stereochemistry-relevant H atoms - Handles empty structures gracefully - Provides error handling for problematic structures **Output:** `glycans_chebi_bulk_submission_cleaned.sdf` **Statistics shown:** - Total entries processed - Entries with structures vs empty structures - Number of entries modified (H removed) ### Step 4: Validate generated SDF file ```bash python3 step4_validate_sdf_file.py ``` **What it does:** Checks that all glycan entries in the SDF: - are formatted correctly - match the source data in glycans_for_chebi.csv - match the data in the GlyGen glycan JSON database files (/data/shared/glygen/releases/data/current/jsondb/glycandb/). --------------------------- ### Step 1 - Glycan Filtering Script File paths are hardcoded for production environment: ```python # Data files data_dir = "/data/projects/glygen/generated/datasets/reviewed" # Images folder images_folder = "/data/shared/glygen/releases/data/current/glycanimages_snfg_svg" # Output folder (auto-created with date) output_dir = "/data/projects/glygen/downloads/chebi_bulk_sub/YYYY_MM_DD" ``` **Note:** Image filtering is mandatory. Script will error if: - Images folder doesn't exist - Images folder is empty - Glycans don't have corresponding SVG files ### Steps 2 & 3 - SDF Generation and Hydrogen Removal If testing locally, update the `testing_mode` variable and configure paths as needed. ## Filtering Criteria ### Glycans are INCLUDED if they have: - `aldi = 0`, `Xxx = '0'`, `X = 0` - NO existing ChEBI ID - At least one PubChem ID (compound or substance) - Corresponding SVG image file - `glytoucan_type` in masterlist ### Publication Handling: - Glycans WITH publications: PubMed IDs are collected and added to `pubmed_ids` column - Glycans WITHOUT publications: `pubmed_ids` column will be empty/NaN - **All glycans are kept** regardless of publication status (no filtering based on citations) ## DEFINITION Field Templates Generated based on `glytoucan_type`: **Saccharide:** > A fully or partially-defined glycan consisting of {# type} groups... (identities are known and connectivity is known or partially known). Composition: {type}({#})... **Topology:** > A partially-defined glycan consisting of {# type} groups... (identities are known but connectivity is partially known). Composition: {type}({#})... **Composition:** > A composition-only glycan consisting of {# type} groups... (identities are either known or partially known, and connectivity is unknown). Composition: {type}({#})... **BaseComposition:** > A composition-only glycan consisting of {# type} groups... (identities and connectivity unknown). Composition: {type}({#})... ## RELATIONSHIP (ISA) Mappings - BaseComposition → ISA167481 - Topology → ISA167503 - Composition → ISA167502 - Saccharide → ISA167559 ## Monosaccharide Type Mappings | Column | Definition Type | Composition Type | |--------|----------------|------------------| | Hex | hexosyl | Hex | | HexNAc | acetaminohexosyl | HexNAc | | dHex | 6-deoxyhexosyl | dHex | | NeuAc | N-acetyl-neuraminic acid | NeuAc | | NeuGc | N-glycolyl-neuraminic acid | NeuGc | | HexA | hexuronic acid | HexA | | HexN | hexosamine | HexN | | S | sulfate | Sulpho | | P | phosphoryl | Phospho | ## Output Files ### `glycans_for_chebi.csv` Filtered dataset with all metadata. Columns include: - `glytoucan_ac` - GlyTouCan accession ID - `glytoucan_type` - Type classification - Monosaccharide counts (Hex, HexNAc, dHex, etc.) - `pubchem_id` - PubChem CID or SID - `pubchem_type` - compound or substance - `pubmed_ids` - Pipe-separated PubMed IDs (empty if no publications) - `sequence_wurcs` - WURCS notation (if available) - `sequence_iupac_extended` - IUPAC sequence (if available) - `definition` - Generated description text - `relationship` - ISA number ### `glycans_chebi_bulk_submission.sdf` Raw SDF file with chemical structures and metadata. ### `glycans_chebi_bulk_submission_cleaned.sdf` **Final submission file** - SDF file with hydrogens removed and structures cleaned. ## Error Handling ### Script 1 (Filtering) - Raises error if any required input file is missing or empty - Raises error if no citation files are found - Raises error if images folder doesn't exist (production mode) - Raises error if no glycans remain after any filtering step - Shows warnings if a filter doesn't remove any rows ### Script 2 (SDF Generation) - Provides empty structure blocks for glycans without PubChem structures - Shows statistics on structures found vs missing - Handles parsing errors gracefully ### Script 3 (Hydrogen Removal) - Preserves original structure if processing fails - Shows warning messages for failed structures - Continues processing remaining entries ## Troubleshooting ### "Images folder not found" - Image filtering is mandatory in production - Ensure the folder exists: `/data/shared/glygen/releases/data/current/glycanimages_snfg_svg/` - Check that SVG files are present in the folder - Verify the path is accessible ### "No SVG files found in images folder" - The images folder is empty - Add SVG image files to the folder - Files should be named with glytoucan_ac (e.g., `G00030VN.svg`) ### "No rows remaining after image filtering" - None of your glycans have matching SVG images - Verify that glycan IDs match the SVG filenames - Check that the images folder path is correct ### "Permission denied" when saving files - Close any CSV/SDF files that might be open in Excel or other programs - The script will create a timestamped backup file automatically if the main file is locked ### "No rows remaining after filtering" Check which filter removed all rows: - All glycans have ChEBI IDs → Expected for re-runs - No glycans have PubChem IDs → Check PubChem file - No matching images → Check images folder and file names - No glytoucan_type → Check masterlist file ### "Structure not found in PubChem SDF" - This is normal - not all glycans have chemical structures - Empty structure blocks will be generated automatically - Check statistics at end of script to see how many structures were found ### Missing columns in CSV - The filtering script should include all required columns - Check the output file was saved successfully - Verify you're using the latest version of the script ## Version History - **v2.0** (2026-02-09) - Production Release - Step 1 script: Production-only with hardcoded paths - Image filtering: Now mandatory (not optional) - Citation handling: Glycans without publications are kept (empty pubmed_ids) - Improved error handling for required image files - Updated documentation for production environment - **v1.0** (2026-02-09) - Initial workflow - Multi-step filtering process with testing/production modes - DEFINITION and RELATIONSHIP generation - SDF generation with PubChem structures - Hydrogen removal using ChEBI methodology - Comprehensive error handling ## Contact For questions or issues, contact the GlyGen data team. ## References - ChEBI libRDChEBI: https://github.com/chembl/libRDChEBI - GlyTouCan: https://glytoucan.org/ - PubChem: https://pubchem.ncbi.nlm.nih.gov/