Publications

Image generation has emerged as a mainstream application of large generative models. Just as test-time compute and reasoning have improved language model capabilities, similar benefits have been observed for image generation models. In particular, searching over noise samples for diffusion and flow models has been shown to scale well with test-time compute. While recent works explore allocating non-uniform inference-compute budgets across denoising steps, existing approaches rely on greedy heuristics and often allocate the compute budget ineffectively. In this work, we study this problem and propose a simple fix. We propose Verifier-Threshold, which automatically reallocates test-time compute and delivers substantial efficiency improvements. For the same performance on the GenEval benchmark, we achieve a 2-4x reduction in computational time over the state-of-the-art method.

Diffusion Transformers (DiTs) with billions of model parameters form the backbone of popular image and video generation models like DALL.E, Stable-Diffusion and SORA. Though these models are necessary in many low-latency applications like Augmented/Virtual Reality, they cannot be deployed on resource-constrained Edge devices (like Apple Vision Pro or Meta Ray-Ban glasses) due to their huge computational complexity. To overcome this, we turn to knowledge distillation and perform a thorough design-space exploration to achieve the best DiT for a given parameter size. In particular, we provide principles for how to choose design knobs such as depth, width, attention heads and distillation setup for a DiT. During the process, a three-way trade-off emerges between model performance, size and speed that is crucial for Edge implementation of diffusion. We also propose two distillation approaches - Teaching Assistant (TA) method and Multi-In-One (MI1) method - to perform feature distillation in the DiT context. Unlike existing solutions, we demonstrate and benchmark the efficacy of our approaches on practical Edge devices such as NVIDIA Jetson Orin Nano.

The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.

Diffusion transformers (DiTs) are effective for image and video generation, but their large model sizes and high computational requirements hinder their use on resource-constrained platforms. To address these major challenges, this work presents a DiT accelerator which enables efficient deployment on edge devices. The accelerator is co-designed with a state-of-the-art (SOTA) 2.2 M parameter count DiT algorithm achieving a Fréchet Inception Distance (FID) of 10.9. The accelerator adopts a custom instruction-set architecture (ISA) with an optimized vectorized processing execution model. A digital in-memory computing (DIMC) architecture is employed to speed-up the compute-heavy matrix-vector multiply (MVM) kernels in the DiT algorithm. A novel DIMC floorplan called memory-in-compute architecture (MICA) is proposed to alleviate the energy and latency overhead of DIMC’s digital summer. MICA is further employed to efficiently generate independent Gaussian noise samples on-chip to support DiT’s generative functionality. Measured results from a prototype IC fabricated in a 28 nm process indicates the MICA macro operates at a maximum frequency of 525 MHz, achieves a SOTA compute density of 1.52 TOPS/mm² at (1 V, 525 MHz) and a SOTA energy efficiency of 31.2 TOPS/W at (0.52 V, 100 MHz).

Videos with binaural audios provide an immersive viewing experience by enabling 3D sound sensation. Recent works attempt to generate binaural audio in a multimodal learning framework using large quantities of videos with accompanying binaural audio. In contrast, we attempt a more challenging problem -- synthesizing binaural audios for a video with monaural audio in a weakly supervised setting and weakly semi-supervised setting. Our key idea is that any down-stream task that can be solved only using binaural audios can be used to provide proxy supervision for binaural audio generation, thereby reducing the reliance on explicit supervision. In this work, as a proxy-task for weak supervision, we use Sound Source Localization with only audio. We design a two-stage architecture called Localize-to-Binauralize Network (L2BNet). The first stage of L2BNet is a Stereo Generation (SG) network employed to generate two-stream audio from monaural audio using visual frame information as guidance. In the second stage, an Audio Localization (AL) network is designed to use the synthesized two-stream audio to localize sound sources in visual frames. The entire network is trained end-to-end so that the AL network provides necessary supervision for the SG network. We experimentally show that our weakly-supervised framework generates two-stream audio containing binaural cues. Through user study, we further validate that our proposed approach generates binaural-quality audio using as little as 10% of explicit binaural supervision data for the SG network.

The problem of gate sizing to meet timing specification while minimizing functional power/area is well understood and is solved by the use of geometric programs (GPs). While these area minimization GP (AM-GP) formulations minimize functional power, they do not address the problem of glitches. Glitches are extraneous transitions caused by signal arrival time imbalance at the input nodes of logic gates. A gate sizing algorithm, Area-Glitch minimization GP (AGM-GP), is proposed to reduce glitches while constraining area and adhering to a timing specification. Glitch reduction is achieved through signal arrival time balancing posed as posynomials in a GP formulation. Prior art does not exploit the complete power of gate sizing when reducing glitches in an attempt to meet the timing specification. In particular, the proposed formulation allows both upsizing and downsizing without causing any timing violation, at the expense of a marginal increase in area. Traditional downsizing methods can be used to further reduce glitches over and above the AGM-GP solution. Simulation results on the ISCAS-85 benchmark circuits show an overall reduction of 20.4% glitch power and 9.5% total power which is respectively 13.8% and 3.9% better than just downsizing the AM-GP solution. This power reduction was achieved with an average area increase of 4.2%.