Parsing SMILES: From SMILES Strings To Molecular Graphs

Introduction

Have you ever wondered how a chemoinformatics toolkit like RDKit takes a SMILES string and converts it to a molecular data structure? In this blog post series, we will build a basic parser that turns a SMILES string into a molecular data structure, or raises an error if it can’t. The series is aimed at scientific software engineers but you can still follow along even if you don’t have a chemoinformatics background.

SMILES

Simplified Molecular Input Line Entry System (SMILES) is one of the common ways molecules are represented in software systems.

The SMILES specification was originally developed in the 1980s and the SMILES notation has since become a widely used way to represent small molecules in computer systems.

Compared to other formats such as connection tables or mol files, SMILES represents molecules as short, linear ASCII strings that are easy to store and exchange.

For example, ethanol is written as CCO: atoms are written as their element symbols, atoms next to each other are bonded even when no bond symbol is written (an omitted bond is single, or aromatic between aromatic atoms), and hydrogens are left out since the toolkit infers each atom’s hydrogen count from its valence.

SMILES can also represent reactions, for example CCO.CC(=O)O>>CC(=O)OCC.O for the formation of ethyl acetate, but this series only covers molecules.

SMILES also has two sister languages from Daylight. SMARTS describes substructure patterns, for example [OH]c1ccccc1 matches phenols, and is used for substructure searching. SMIRKS describes reaction transforms, that is, generic reactions. Where reaction SMILES describes one specific reaction, a SMIRKS transform can be applied to any molecules that match it. Both look like SMILES but describe patterns and transforms rather than molecules, so this series won’t cover them. Extensions such as CXSMILES and BigSMILES are also out of scope.

Molecules as graphs

A SMILES string is just text. To do anything useful with it, for example count atoms, find rings or compare molecules, a toolkit first converts it into a molecular graph.

A graph is a set of nodes connected by edges. In a molecular graph, atoms are the nodes and bonds are the edges. Each node stores atom properties such as the element, charge and hydrogen count, and each edge stores bond details such as the bond type.

A graph data structure is a natural fit because that is what a molecule is: atoms and the connections between them, with no fixed start or order. For example, CCO and OCC are two different strings, but both describe ethanol and produce isomorphic graphs: the same atoms and connections, numbered in a different order.

RDKit’s ROMol object is such a graph, and thus our parser will also produce a graph data structure.

What is parsing

Parsing is the process of turning text (also called the source) into some structured output by following a set of rules. These rules are defined in a specification known as the language grammar, similar to how English grammar rules defines which sentences are valid.

For example, Python’s json.loads parses a JSON string into Python data structures by following the JSON specification. Turning a SMILES string into a molecular data structure is also parsing, and its rules are defined by the SMILES grammar, which we’ll look at next.

The SMILES grammar

Grammars and EBNF

A grammar is a formal description of all the valid string constructs in a language. It is made up of two things: terminals and rules.

Terminals are the leaf tokens of the language, the literal characters that appear in the input, for example {, : or true in JSON. Rules (also called non-terminals) describe how terminals and other rules can be combined. A rule can be composed of other rules but every rule must eventually reduce to terminals.

Grammars are usually written in Extended Backus-Naur Form (EBNF) notation. This series uses the W3C style of EBNF, shown below:

As an example, let’s write a grammar for a tiny subset of SMILES which only supports a chain of carbon, nitrogen, oxygen and chlorine atoms joined by single, double or triple bonds.

chain ::= atom ( bond? atom )*
atom  ::= "C" | "N" | "O" | "Cl"
bond  ::= "-" | "=" | "#"

Here chain, atom and bond are rules while "C", "=" etc are terminals. Read out loud, the chain rule says: a chain is an atom, followed by zero or more atoms, each of which may be preceded by a bond.

With this grammar, CCO (ethanol), C=O (formaldehyde) and ClCCl (dichloromethane) are valid. C==O is not, since the grammar does not allow two bonds in a row. Neither is CX, because X is not a terminal in the grammar.

The full OpenSMILES grammar, which we’ll look at next, is built the same way, just with many more rules. If you want to go deeper, the grammars chapter in Bob Nystrom’s Crafting Interpreters is an excellent read.

OpenSMILES vs IUPAC SMILES+

SMILES started out as a proprietary specification by Daylight Chemical Information Systems. In 2007, the Blue Obelisk community started OpenSMILES, an open specification of the language. It contains a formal grammar, which is what we’ll implement. It also separates syntax from semantics: in its own words, “the syntax specifies how ring closures are written, but the semantics require that they come in pairs”. Our parser reports these two kinds of problem as different errors, as we’ll see in Inside a parser. OpenSMILES also lists “relaxed” rules for common mistakes, for example C((C))O or [Na+]..[Cl-], which a parser may accept only if the user explicitly asks for it.

Since toolkits still disagree on how to read some SMILES, IUPAC is working on a stricter standard, IUPAC SMILES+, derived from OpenSMILES. It is still a draft, so this series uses OpenSMILES as the reference.

The grammar

The SMILES grammar is defined in section 2.2 of the OpenSMILES specification. Here are its core rules (the full grammar is in the appendix):

smiles        ::= chain? terminator?
chain         ::= branched_atom | chain branched_atom | chain bond branched_atom
                | chain dot branched_atom
branched_atom ::= atom ringbond* branch*
branch        ::= '(' chain ')' | '(' bond chain ')' | '(' dot chain ')'
ringbond      ::= bond? DIGIT | bond? '%' DIGIT DIGIT
bond          ::= '-' | '=' | '#' | '$' | ':' | '/' | '\'
dot           ::= '.'
atom          ::= bracket_atom | aliphatic_organic | aromatic_organic | '*'
bracket_atom  ::= '[' isotope? symbol chiral? hcount? charge? class? ']'

It is the same idea as our tiny grammar, just with more rules. A few things stand out:

• chain is the backbone. Apart from the terminator, a SMILES string is a chain of branched_atoms, separated by an optional bond or a dot (which means “not bonded”, as in [Na+].[Cl-]).
• Branches are recursive. A branch contains a chain, which can contain more branches. This is what allows nesting, for example CC(C(C)C)C.
• Ring closures belong to atoms. In C1CCCCC1, the 1 is a ringbond attached to the atom before it. Two-digit ring numbers need a % prefix, for example C%12CCCCC%12.
• Bracket atoms are a small language of their own. Inside [...], an atom can specify its isotope, symbol, chirality, hydrogen count, charge and atom class, in that fixed order, for example [13CH4] or [NH4+]. Each field is optional except the symbol; an omitted hydrogen count or charge means zero.

Some rules are not in the grammar at all. For example, the grammar accepts C1CC, but the spec’s semantics require ring closures to come in pairs. Likewise, '@TB' DIGIT DIGIT? accepts @TB99, but only @TB1 to @TB20 are meaningful. These checks need more than the grammar, as we’ll see in Inside a parser.

SMILES strings vs SMILES files

So far we have only looked at a single SMILES string, but SMILES often come in files, one molecule per line. This is why the grammar has a terminator rule. According to section 3.9 of the spec, a SMILES string ends at the first space, tab, newline or carriage return, or at the end of the string. Anything after it, for example a name or registration number, is ignored by the parser, “although applications that use the SMILES parser will often make use of it”.

Section 4.5 then defines a SMILES file: one SMILES per line, optionally followed by whitespace and other data, with either Unix (LF) or Windows (CR/LF) line endings. Blank lines and lines that start with whitespace are ignored. Such files usually have the .smi extension:

SMILES Name
CCO ethanol
CC(=O)O acetic_acid
c1ccccc1 benzene

RDKit reads these files with SmilesMolSupplier. By default, it expects a space-delimited file with a title line, the SMILES in column 0 and the name in column 1. It can also read CSV and other delimited files, as long as you tell it the delimiter and which column holds the SMILES.

from rdkit import Chem

for mol in Chem.SmilesMolSupplier("molecules.smi"):
    if mol is None:  # this line's SMILES could not be parsed
        continue
    print(mol.GetProp("_Name"), mol.GetNumAtoms())

In other words, splitting a file into lines and columns is the job of a reader that sits on top of the SMILES parser. The parser itself only ever sees one SMILES string at a time. Ours will work the same way: it parses a single SMILES string and stops at the terminator, and reading files is left to the code that calls it.

Parsing 101

Inside a parser

Our SMILES parser works in three stages:

The lexer breaks down the source input into tokens, the parser checks the tokens against the language grammar and builds the molecular graph as it goes, and the sanitizer checks that the molecule makes chemical sense.

Lexical analysis (lexer)

The first step of parsing is to break the input into a list of tokens. A token is the smallest unit of the input that still holds meaning. The program that does this goes by three names: lexer, tokenizer and scanner. They all mean the same thing, and this series uses lexer.

For SMILES, ask yourself: what is the smallest unit that still makes chemical sense? Atoms, bonds, ring numbers, charges, chirality markers, etc. For example, breaking the chlorine symbol Cl into C and l would not make sense, so Cl is a single token.

The lexer goes through the input character by character and emits tokens as it goes, so CCO gives the tokens C, C and O, which it passes to the parser. If the lexer comes across something that cannot form a valid token, it raises a lexical error (also called a tokenization or scanning error). For example, CXC fails at this stage since X is not a valid SMILES symbol.

Syntax analysis (parser)

The parser checks that the tokens are arranged in a way the grammar allows, and adds atoms and bonds to the molecular graph as it reads them. For CCO, that is three heavy atoms (non-hydrogen atoms) connected by two single bonds:

The hydrogens are not separate nodes; the sanitizer later works out each atom’s implicit hydrogen count.

Input made up of valid tokens can still be invalid, and the parser catches two kinds of problem:

1. The tokens break the grammar. C(C fails because the branch is never closed, and so does C==C because two bonds cannot follow each other. These are syntax errors.
2. The graph cannot be built. C1CC follows the grammar, but ring bond 1 is opened and never closed, so the parser has no second atom to attach that bond (edge) to. This kind of error comes from the meaning of the input rather than its form, so it is a semantic error.

Sanitization (sanitizer)

A graph that can be built can still be chemically meaningless. For example, C(C)(C)(C)(C)C follows the grammar and builds a graph just fine, but its central carbon has five bonds, which is chemically nonsense.

Toolkits run these chemistry checks, for example valence and aromaticity, after the graph is built. In RDKit this is called sanitization, and MolFromSmiles does it by default. Our sanitizer will work out each atom’s implicit hydrogen count and check basic rules such as valence.

Types of parsers

Parsers are commonly grouped into two families, based on the direction in which they match the grammar:

• Top-down parsers start from the top-level rule, for example “a molecule”, and work their way down to the individual tokens.
• Bottom-up parsers start from the tokens and combine them into bigger structures until they reach the top-level rule.

A common kind of top-down parser is the recursive descent parser (RDP). In an RDP, each grammar rule becomes a function, after a small rewrite for rules that refer to themselves first. For example, a parse_chain function calls parse_atom for each atom, and when it sees (, it calls parse_branch, which in turn calls parse_chain again for the atoms inside the branch. The functions call each other recursively as they descend through the input, hence the name.

RDPs are simple to write by hand, easy to read and debug, and give you full control over error messages.

We will implement our parser as a recursive descent parser.

Parser generators vs hand-written parsers

A parser doesn’t have to be written by hand. A parser generator is a program that takes a grammar as input and generates the parser code for you. Well-known examples are GNU Bison, usually paired with the lexer generator Flex, and ANTLR.

Each option has its trade-offs:

• Parser generators: the grammar file is the source of truth, so the generated parser always matches the grammar, and there is less code to write. Keeping that grammar faithful to the spec is still up to you. However, you have to learn the tool’s grammar syntax, the generated code is hard to read and debug, and error messages are harder to customize.
• Hand-written parsers: more code to write, but you get full control over error messages, error recovery and performance, with no extra build step or dependency.

Both approaches are used for SMILES. RDKit generates its SMILES parser with Flex and Bison (see smiles.ll and smiles.yy), while Beam, the SMILES library used by CDK, has a hand-written parser.

Since the goal of this series is to learn how parsers work, we will write ours by hand.

How existing toolkits parse SMILES

Let’s look at how three popular open-source toolkits do it: RDKit, Open Babel and CDK.

RDKit

As mentioned above, RDKit uses Flex and Bison. It does not build an abstract syntax tree (AST), the tree representation of the input that compilers usually build between parsing and later stages. Instead, each grammar rule in smiles.yy has an action that adds atoms and bonds to the molecule as soon as the rule is matched.

Open Babel

Open Babel’s SMILES reader, in smilesformat.cpp, is hand-written. It loops over the string one character at a time, adding atoms and bonds to the molecule as they are read and keeping track of open branches and rings. Once the loop ends, it checks that every ring was closed.

CDK (Beam)

CDK reads SMILES using Beam, a separate library written by John May. Beam’s Parser is also hand-written and follows the same single-loop pattern, with strict and relaxed modes that control how forgiving it is with unusual input.

Summary

None of the three builds an AST. Since SMILES is a fairly simple language, they lex and parse in one forward pass and build the molecule as they read the string. This avoids building an intermediate token list and tree, but the stages are harder to tell apart, test and extend.

Our parser’s design

Why separate stages

Unlike Open Babel and CDK, we will keep lexing, parsing and sanitization as separate stages. This is a choice for learning rather than a requirement for correct parsing, and it needs more code than a single pass, but:

• Each stage does one thing. The lexer only knows about characters, the parser only knows about the grammar and the sanitizer only knows about chemistry, so each stage is easier to understand and test on its own.
• Errors map to stages. Lexical errors come from the lexer, syntax errors and graphs that cannot be built come from the parser, and chemistry errors come from the sanitizer, so it is always clear what went wrong.
• Tokens are reusable. Other SMILES tooling can be built on them without building a molecule, for example a syntax highlighter.
• It is easier to learn from. Separate stages make each concept visible.

Each stage will raise its own error type:

Each error will also carry the position in the input where it happened. The OpenSMILES spec encourages this, recommending that error messages “show exactly which character or ‘phrase’ of the SMILES string triggered the error message”.

What’s next

We will thus start with lexing, followed by parsing into a molecular graph, and then sanitization.

Appendix: the full OpenSMILES grammar

/* SMILES strings */
smiles            ::= chain? terminator?
terminator        ::= #x20 | #x9 | #xA | #xD

/* Chains, branches and bonds */
chain             ::= branched_atom | chain branched_atom | chain bond branched_atom
                    | chain dot branched_atom
branched_atom     ::= atom ringbond* branch*
branch            ::= '(' chain ')' | '(' bond chain ')' | '(' dot chain ')'
ringbond          ::= bond? DIGIT | bond? '%' DIGIT DIGIT
bond              ::= '-' | '=' | '#' | '$' | ':' | '/' | '\'
dot               ::= '.'

/* Atoms */
atom              ::= bracket_atom | aliphatic_organic | aromatic_organic | '*'

/* Organic subset atoms */
aliphatic_organic ::= 'B' | 'C' | 'N' | 'O' | 'S' | 'P' | 'F' | 'Cl' | 'Br' | 'I'
aromatic_organic  ::= 'b' | 'c' | 'n' | 'o' | 's' | 'p'

/* Bracket atoms */
bracket_atom      ::= '[' isotope? symbol chiral? hcount? charge? class? ']'
isotope           ::= NUMBER
symbol            ::= element_symbols | aromatic_symbols | '*'
element_symbols   ::= 'H' | 'He' | 'Li' | 'Be' | 'B' | 'C' | 'N' | 'O' | 'F' | 'Ne'
                    | 'Na' | 'Mg' | 'Al' | 'Si' | 'P' | 'S' | 'Cl' | 'Ar' | 'K' | 'Ca'
                    | 'Sc' | 'Ti' | 'V' | 'Cr' | 'Mn' | 'Fe' | 'Co' | 'Ni' | 'Cu' | 'Zn'
                    | 'Ga' | 'Ge' | 'As' | 'Se' | 'Br' | 'Kr' | 'Rb' | 'Sr' | 'Y' | 'Zr'
                    | 'Nb' | 'Mo' | 'Tc' | 'Ru' | 'Rh' | 'Pd' | 'Ag' | 'Cd' | 'In' | 'Sn'
                    | 'Sb' | 'Te' | 'I' | 'Xe' | 'Cs' | 'Ba' | 'La' | 'Ce' | 'Pr' | 'Nd'
                    | 'Pm' | 'Sm' | 'Eu' | 'Gd' | 'Tb' | 'Dy' | 'Ho' | 'Er' | 'Tm' | 'Yb'
                    | 'Lu' | 'Hf' | 'Ta' | 'W' | 'Re' | 'Os' | 'Ir' | 'Pt' | 'Au' | 'Hg'
                    | 'Tl' | 'Pb' | 'Bi' | 'Po' | 'At' | 'Rn' | 'Fr' | 'Ra' | 'Ac' | 'Th'
                    | 'Pa' | 'U' | 'Np' | 'Pu' | 'Am' | 'Cm' | 'Bk' | 'Cf' | 'Es' | 'Fm'
                    | 'Md' | 'No' | 'Lr' | 'Rf' | 'Db' | 'Sg' | 'Bh' | 'Hs' | 'Mt' | 'Ds'
                    | 'Rg' | 'Cn' | 'Nh' | 'Fl' | 'Mc' | 'Lv' | 'Ts' | 'Og'
aromatic_symbols  ::= 'b' | 'c' | 'n' | 'o' | 'p' | 's' | 'se' | 'as'

/* Chirality */
chiral            ::= '@' | '@@' | '@TH' [1-2] | '@AL' [1-2] | '@SP' [1-3]
                    | '@TB' DIGIT DIGIT? | '@OH' DIGIT DIGIT?

/* Hydrogens */
hcount            ::= 'H' | 'H' DIGIT

/* Charges ('--' and '++' are deprecated) */
charge            ::= '-' | '-' DIGIT? DIGIT | '+' | '+' DIGIT? DIGIT | '--' | '++'

/* Atom class */
class             ::= ':' NUMBER

/* Terminals the spec leaves undefined */
DIGIT             ::= [0-9]
NUMBER            ::= DIGIT+