Abstract: Foundation models are emerging to be an important overarching toolset for various machine learning tasks scientific discovery.
In this talk I will first share a novel framework for improving performance on molecule predictions. A key observation is that while some auxiliary downstream molecule prediction tasks help each other, some other tasks might be detrimental and cause negative transfer to certain downstream tasks. Such transferability correlates with task and dataset distribution similarity. Our proposed approach automatically selects a set of auxiliary tasks to be grouped with a target task for joint fine-tuning, building on foundation models, allowing the fine-tuned model to achieve much better performance.
We will further introduce our pre-trained equivariant RNA foundation model, incorporating both molecule structure and 3D information into a transformer architecture based on frame-averaging and a novel edge module. The work paves way for a joint RNA and protein foundation models.

Abstract: Inspired by spectral embeddings and graph kernels, Graph Neural Networks (GNNs) have enjoyed great empirical success for the ability to adapt too different tasks, generalize to new graphs and incorporate attributes.

In this talk, I will go back to the theoretical origins of this model class and present some recent advances in hypergraph spectral algorithms and manifold learning, discuss the insights they give us about the inner workings of GNNs and the techniques they suggest towards improving this already formidable learning framework.

Abstract: The ML/NN community has been buoyed by GPU’s, providing ample flops for training and inference. However, GPU’s are an example of the “Hardware Lottery” where the capabilities of a hardware architectural approach constrains and shapes the software and algorithms. GPU’s encourage dense layers and make sparse structures difficult to program and often poor performance.

We are creating a new class of architectures; the first instance is the UpDown System. These systems provide performance increases of 100x or more on sparse structures such as real-world graphs and programmability effort and style more akin to CPUs. The key is a new approach to compute architecture that eschews deep cache hierarchies, but instead relies on vast memory bandwidth of terabytes/second and flexible vertex/edge level parallelism to achieve high performance. The key is novel architecture features such as efficient short threads (10s of instructions), massive memory concurrency (unlimited outstanding requests/thread), prolific hardware multithreading (200,000 threads/node), and artful design that makes high level programmability natural. Programmers can separately specify function/parallelism, computation placement, and data placement. We will describe the design of the UpDown system, some performance vignettes, and sketch how some complex graph applications are expressed on the system.

Abstract: Data-driven modeling and deep learning present powerful tools that are opening up new paradigms and opportunities in the understanding, discovery, and design of soft and biological materials. I will describe an approach based on denoising diffusion models to restore all-atom resolution to coarse-grained molecular dynamics simulations of proteins to help break the time scale barrier in biomolecular simulations.

Abstract: Deep learning is revolutionizing many areas of science and technology, particularly in natural language processing, speech recognition and computer vision. In this talk, we will provide an overview into latest developments of machine learning and AI methods and application to the problem of quantum chemistry at Isayev’s Lab at CMU. We identify several areas where existing methods have the potential to accelerate computational chemistry research and disrupt more traditional approaches.

First we will present a deep learning model that approximate solution of Schrodinger equation. Focusing on parametrization for drug-like organic molecules and proteins, we have developed a single ‘universal’ model which is highly accurate compared to reference quantum mechanical calculations at speeds 10^6 faster. Second, we propose an improved machine learning framework for simulating molecules in arbitrary spin and charge states.

Abstract: Topological Neural Networks (TNNs) are deep learning architectures that process signals defined on topological domains, such as hypergraphs and cellular complexes -- hence generalizing Graph Neural Networks. The additional flexibility and expressivity of TNN architectures permits the representation and processing of complex natural systems such as proteins, neural activity, and many-body physical systems. This talk synthesizes the recent TNN literature using a single unifying notation and graphical summaries and sheds light on existing challenges and exciting opportunities for future development.

Abstract: We discuss the use of graph neural networks (GNNs) are useful in the context of drug discovery workflows. We will share some vignettes of how GNNs can be leveraged to represent large molecular libraries using molecular building blocks (i.e., fragments, scaffold, linkers/decorations), and how GNNs can be used to navigate latent representations of molecular hypergraphs leveraging transformer networks to operate on molecular building blocks to generate new molecules.

We will demonstrate that GNNs possess some unique representational advantages for molecular building blocks (compared to other techniques), while allowing intuitive discovery of novel molecules that can result in binding to and inhibiting SARS-CoV-2 viral protein targets. Further, we show that GNNs can be used to accelerate virtual screening protocols by at least an order of magnitude while spanning much larger chemical spaces than currently possible. We also discuss how incorporating human feedback within GNNs can potentially result in novel molecules with desirable functional properties in the context of drug discovery.
Collaboration with: Rick Stevens, Anima Anandkumar, Austin Clyde, Ryien Hosseini, Ashka Shah, Filipo Simini.

Abstract: A fundamental constraint on any type of graph neural network is that it must be equivariant to permutations of the vertices. The currently most popular type of GNNs are message passing neural networks (MPNNs), which achieve equivariance by employing an aggregation rule that simply sums the activations of neighboring vertices. This scheme limits the network’s ability to capture structure at the local level, for example, to unambiguously identify functional groups in molecules. To overcome these limitations, researchers are increasingly turning to higher order GNNs, which aggregate messages to subgraphs rather than individual vertices and/or operate with messages that transform according to higher order representations of the group of permutations.

In this talk I will describe a general framework for designing and implementing such networks based on new type of mathematical object called P-tensors and show promising empirical results.
The material presented is based on joint work with Andrew Hands, Tianyi Sun, Richard Xu and Qingqi Zhang.

Abstract: In this talk, we explore the challenges of developing Graph Neural Networks (GNNs) that produce robust out-of-distribution (OOD) predictions, with a focus on physical science applications. Traditional machine learning models, including GNNs, often struggle with OOD generalization. This is particularly critical in the physical sciences, where environmental conditions can vary significantly during inference.

We discuss the vital role of causality-aligned GNN architectures in enhancing OOD prediction robustness. The talk concludes underscoring the importance of integrating causality and symmetries to improve predictive modeling in science and engineering.

Abstract: Graph neural networks (GNNs) have proven successful in various applications, leveraging complex graph data. Biomedical Knowledge Graphs (KG) is an efficient way to capture biological information. We first go over some of the work on constructing and making inference with Biomedical KG. We also dive deep on  BioBridge, a parameter-efficient learning framework, is presented. BioBridge utilizes KG to establish multimodal behavior between independently trained unimodal FMs. Empirical results show BioBridge outperforms baseline KG embedding methods in cross-modal retrieval tasks by approximately 76.3%. Furthermore, BioBridge demonstrates out-of-domain generalization and serves as a general-purpose retriever, aiding biomedical multimodal question answering and enhancing guided drug generation.

Abstract: The emergence of deep learning models designed for graph and network data, often under an umbrella term named graph neural networks (GNNs for short), has largely streamlined many graph learning problems. In the vast majority of the existing works, they aim to answer the following question, that is, given a graph, what is the best GNNs model to learn from it.

In this talk, we introduce the graph sanitation problem, to answer an orthogonal question. That is, given a mining task and an initial graph, what is the best way to improve the initially provided graph? We formulate the graph sanitation problem as a bilevel optimization problem, and further instantiate it by semi-supervised node classification, together with an effective solver named Gasoline. I will also introduce other works we recently did on learning optimal graphs and share my vision for the future directions.

Abstract: In transfer learning, the general goal is to leverage the abundant label information from one or more source domains to build a high-performing predictive model in the target domain with limited or no label information. While many research efforts have been focusing on the IID setting where the examples from both the source and target domains are considered to be independent and identically distributed within each domain, recently more research works have been dedicated to the non-IID setting. In particular, many real applications have motivated the study of transferrable graph learning, where the data from both the source and target domains are represented as graphs.

In this talk, I will introduce our recent work in this direction using graph neural networks for both regression and classification. For regression, starting from the transferrable Gaussian process for IID data, I will discuss a generic graph-structured Gaussian process framework for adaptively transferring knowledge across graphs with either homophily or heterophily assumptions. For classification, I will present a novel Graph Subtree Discrepancy to measure the graph distribution shift between source and target graphs, which will lead to the generalization error bounds on cross-network transfer learning, including both cross-network node classification and link prediction tasks

Abstract: There has been exciting recent advances in our ability to measure single cells in its spatial context. The challenge now lies in effectively modeling and deriving biological insights from this complex spatial data.

I will first give an overview of the new technologies for measuring spatially resolved gene expression and protein abundances. Then I will present SPACE-GM, a flexible and general graph deep learning framework for analyzing spatial omics data. We applied SPAGE-GM to several diseases, including 658 head-and-neck and colorectal cancer tissue samples, to discover spatial cellular motifs that predict patient response to cancer treatments. Analysis of these motifs reveals biological insights into tumor-immune interactions that could affect patient outcomes.

Abstract: Graph mining algorithms are widely developed for decades, but these algorithms are often unfair, hindering their deployment in high-stake applications. To ensure fairness in graph mining, it is crucial to propose a paradigm shift, from answering what and who to understanding how and why.

In this talk, I will present our efforts in studying fair graph mining with respect to various fairness definitions, including group fairness, individual fairness, and degree fairness. And I will conclude the talk with my thoughts and visions for future research in fair graph mining.

Abstract: By recursively summing node features over entire neighborhoods, spatial graph convolution operators have been heralded as key to the success of Graph Neural Networks (GNNs). Yet, despite the multiplication of GNN methods across tasks and applications, the impact of this aggregation operation on their performance still has yet to be extensively analyzed. In fact, while efforts have mostly focused on optimizing the architecture of the neural network, fewer works have attempted to characterize (a) the different classes of spatial convolution operators, (b) how the choice of a particular class relates to properties of the data , and (c) its impact on the geometry of the embedding space. In this talk, we bring elements of solution to these three questions by dividing existing operators into two main classes (symmetrized vs. row-normalized spatial convolutions), and show how these translate into different implicit biases on the nature of the data. Finally, we show that this aggregation operator is in fact tunable, and explicit regimes in which certain choices of operators – and therefore, embedding geometries – might be more appropriate.