Escaping the Big Data Paradigm with Compact Transformers, 2021 (Train your Vision Transformers in 30 mins on CIFAR-10 with a single GPU!)

🔥 [CVPR24] COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction

The initial version of EQCCT (EarthQuake detection and phase picking using the Compact Convolutional Transformer)

[AAAI2023] DC-Former:Diverse and Compact Transformer for Person Re-Identification

This repository contains the code used for distillation and fine-tuning of compact biomedical transformers that have been introduced in the paper "On The Effectiveness of Compact Biomedical Transformers"

This project develops compact transformer models tailored for clinical text analysis, balancing efficiency and performance for healthcare NLP tasks.

Brain tumor images classification with ResNet, EfficientNet, EfficientNet_V2 and Compact Convolutional Transformers architectures with PyTorch

Compact Convolution Transformers

Implementation of "Escaping the big data paradigm with Compact Transformers" by Ali Hassani et al. in Keras

NanoDiffVision explores Differential Attention as a natural evolution of Vision Transformers (ViT), re-implementing and comparing attention mechanisms in compact, resource-efficient models.

Custom caffe with signed power, compact bilinear and spatial transformer layer.

My own ViT (Basic ViT & Compact Transformer) & Swin Transformer implementation.

COMPACT SELECTIVE TRANSFORMER BASED ON INFORMATION ENTROPY FOR FACIAL EXPRESSION RECOGNITION IN THE WILD

MathSnap AI — An end-to-end handwritten mathematical expression recognition (HMER) system powered by Mini-CoMER, a compact encoder-decoder architecture (6.39M params) with DenseNet encoder, Transformer decoder, and Attention Refinement Module (ARM). Trained on CROHME 2013/2016/2019 datasets.

This repo is a compact, dependency-free micro-GPT in pure Python. It includes a minimal autograd engine, a tiny Transformer, a training loop, and inference. The goal is to keep the core LLM pipeline readable and small.

Compact Self-Supervised Vision Transformer (cSiT) on Histopathology Images

Compact transformer model for detecting harmful/unsafe text prompts

Compact, decoder-only Transformer PINN using Fourier features to mitigate spectral bias and self-attention to capture spatiotemporal correlations

A compact language model implementing HybridNorm and Fourier-based attention. Combines CoLA (low-rank projections), FANformer, and hybrid normalization to create an efficient decoder-only transformer. Leverages periodicity modeling and gated residuals to enhance performance while maintaining a small parameter footprint.

Compact Transformers implemented in keras

Implementations of Vision Transformer, Compact Vision Transformer and Compact Convolutional Transformer in PyTorch

HN GROUPS are basically from Bangalore and operating all the construction related project works all over in India on turnkey basis since 1990. MHN GROUPS are working as a consultants, contractors, manufacturers, exporters and importers. We are taking all the construction projects on Engineering - Procurement - Contract [EPC] and contracting basis. MHN GROUPS is an one stop place for getting all types of construction solutions / requirements on professional way. We have associated team of Architects, Engineers, Designers and planners for creating and fast executing all types construction projects. We have vast and experienced team of engineers, machinery and labour force for taking up multiple activities in one time. MHN GROUPS are associated with so many professional and experienced Architects, Structural Engineers for taking up planning and designing and as well as working as building contractors and civil contractors. MHN GROUPS are executing the same projects on in house and contracting basis. We are taking up these type of projects for individual homes, apartment projects, commercial projects, villa projects, row house projects, residential plots developers, residential venture developing contractors, interior contractors, jack well digging contractors and more. We have our own construction team of architects, civil engineers, structural engineers, quantity surveyors, masons, carpenters, bar-benders, electrical contractors, plumbers, painters, flooring masonry, gabion walls / feature walls and more. We are taking properties for developing these projects on out rate and joint venture basis. We have vast experience in this profession for developing, execution and marketing. Also we are into as manufacturers and traders for all types of construction materials in bulk order, in and around Bangalore. We have our own manufacturing units for clay bricks and concrete bricks like country bricks, table mould bricks, wire cut bricks, light weight bricks, flyash bricks, paving bricks, lantern bricks [engineering bricks], brick bat [churiki] and more.. MHN GROUPS are basically from Bangalore and operating all the construction related project works all over in India on turnkey basis since 1990. MHN GROUPS are working as a consultants, contractors, manufacturers, exporters and importers. We are taking all the construction projects on Engineering - Procurement - Contract [EPC] and contracting basis. MHN GROUPS is an one stop place for getting all types of construction solutions / requirements on professional way. We have associated team of Architects, Engineers, Designers and planners for creating and fast executing all types construction projects. We have vast and experienced team of engineers, machinery and labour force for taking up multiple activities in one time. MHN GROUPS are associated with so many professional and experienced Architects, Structural Engineers for taking up planning and designing and as well as working as building contractors and civil contractors. MHN GROUPS are executing the same projects on in house and contracting basis. We are taking up these type of projects for individual homes, apartment projects, commercial projects, villa projects, row house projects, residential plots developers, residential venture developing contractors, interior contractors, jack well digging contractors and more. We have our own construction team of architects, civil engineers, structural engineers, quantity surveyors, masons, carpenters, bar-benders plumbers, painters, flooring masonry, gabion walls / feature walls and more. We are taking properties for developing these projects on out rate and joint venture basis. We have vast experience in this profession for developing, execution and marketing. Also we are into as manufacturers and traders for all types of construction materials in bulk order, in and around Bangalore. We have our own manufacturing units for clay bricks and concrete bricks like country bricks, table mould bricks, wire cut bricks, light weight bricks, flyash bricks, paving bricks, lantern bricks [engineering bricks], brick bat [churiki] and more.. MHN GROUPS are one of the good experienced and professionalism construction company with very good technical experience, excellence, innovation and total dedication for completing all the projects. That's how the MHN GROUPS of success story in construction industry goes on in past 25 years. In addition to construction industry MHN GROUPS are basically from Bangalore and operating all the construction related project works all over in India on turnkey basis since 1990. MHN GROUPS are working as a consultants, contractors, manufacturers, exporters and importers. We are taking all the construction projects on Engineering - Procurement - Contract [EPC] and contracting basis. MHN GROUPS is an one stop place for getting all types of construction solutions / requirements on professional way. We have associated team of Architects, Engineers, Designers and planners for creating and fast executing all types construction projects. We have vast and experienced team of engineers, machinery and labour force for taking up multiple activities in one time. MHN GROUPS are associated with so many professional and experienced Architects, Structural Engineers for taking up planning and designing and as well as working as building contractors and civil contractors. MHN GROUPS are executing the same projects on in house and contracting basis. We are taking up these type of projects for individual homes, apartment projects, commercial projects, villa projects, row house projects, residential plots developers, residential venture developing contractors, interior contractors, jack well digging contractors and more. We have our own construction team of architects, civil engineers, structural engineers, quantity surveyors, masons, carpenters, bar-benders plumbers, painters, flooring masonry, gabion walls / feature walls and more. We are taking properties for developing these projects on out rate and joint venture basis. We have vast experience in this profession for developing, execution and marketing. Also we are into as manufacturers and traders for all types of construction materials in bulk order, in and around Bangalore. We have our own manufacturing units for clay bricks and concrete bricks like country bricks, table mould bricks, wire cut bricks, light weight bricks, flyash bricks, paving bricks, lantern bricks [engineering bricks], brick bat [churiki] and more.. MHN GROUPS are one of the good experienced and professionalism construction company with very good technical experience, excellence, innovation and total dedication for completing all the projects. That's how the MHN GROUPS of success story in construction industry goes on in past 25 years. In addition to construction industry MHN GROUPS ​are (No-1) construction company for taking up all types of construction related project works like as ALL IN ONE PLOT FORM. Company has been entrusted with construction of high value EPC turnkey projects for core organisations including government and private sectors. From a humble start in Building material manufacturers in 1990, now we are in to delivery any type of construction related project works within time and uncompromising quality. MHN GROUPS main strength is our existing customers ​​​and always will remain our dedicated team of highly qualified engineers, managers and workers. Supported by the latest technologies and assets second to none, our team ensures that the strictest quality standards are maintained in every project that we are getting ​repeated​ orders from our clients stand testimony to our professionalism and ability to provide effective results in quick turnaround time. MHN GROUPS are associated with so many professionals for taking up planning and designing and as well as building contractors and civil contractors for individual homes, apartment projects, commercial projects, villa projects, row house projects, park developing contractors, road projects and more. We have our own construction team of civil engineers, structural engineers, quantity surveyors, construction masons, carpenters, bar benders, plumbers, painters, pop contractors, flooring/tiling masons, fabricators, interior designers and more. We are working in following areas...  Architects, Builders, Civil Engineers, Structural Engineers, Civil Contractors, Industrial Contractors, Interior Contractors, Land Developers, Layout Contractors, Landscaping Contractors, Road Contractors, Earth Movers, Rock Breakers, Natural Stones, Stone Designers, Canal / Dam / Reservoir Contractors, Desilting / Dredging, Mining Contractors, UGD Contractors, Pipeline Contractors, Electrical Contractors, Plumbing Contractors, Grit / Guniting / Lining Contractors, jack Wells / Gabion Walls / Feature Walls, Soil Filling / Earth Filling / Embankment, Geogrid Walls / RE Walls, Building Materials, Fly overs / Piling / Bridge Contractors, Highway Road Contractors & more….. MHN GROUPS are taking civil related projects like construction of residential apartments, villa projects, commercial projects, industrial projects, educational projects, hotel projects, resorts projects, railway projects, airport projects, landscaping projects, layout projects, land development projects, pipeline projects, road projects, compound wall projects, guniting projects, lining projects, gabion wall projects, geogrid wall projects, reinforced soil / earth retaining wall projects, flooring projects, parks development projects, soil / earth / mud / gravel / morrum filling projects, stone design projects, cladding projects, de-silting projects, dredging projects, pitching projects, construction of overhead tanks, villa projects, apartment projects, jack well projects, water projects, housing projects, amphitheatre contractors, agricultural projects, concrete piling contractors, farm house projects for developing and construction. MHN GROUPS are working and developing all types of road related works like national highway, state highways, layout roads, land developing project roads, industrial roads, agricultural project roads, villa project roads, row house project roads, farm house project roads, commercial project roads with making asphalting roads, bituminous roads, concrete roads, vdf roads, wbm roads, gsb roads, soil embankment roads, construction of bridges, flyovers and underpasses, foot paths, dismantling contractors, demolishing contractors, side drains, reinforced concrete geo grid walls, reinforced earth wall, reinforced stone facia walls, gabion walls and more.. MHN GROUPS taking all types of earth related projects like canal projects, dam projects, reservoir projects, road projects, pond projects, lake projects, park projects, agriculture projects, industrial projects, layout projects for soil filling, soil cutting, soil embankment, revetment works, pitching works, lining works, guniting works, soil nailing works, soil stabilisation works and more. We are specialised company for taking up soil filling works, filling earth suppliers within Bengaluru and Karnataka. We have a capacity of soil supplying in a day more than 1000 loads for mass earth filling works. MHN GROUPS are working in hard landscaping and soft landscaping with construction of planter boxes, swimming pools, foot ball courts, basketball courts, kabbadi courts, tennis courts, golf courses, children's play areas, water bodies, compound walls, security cabins, granite works, stone works, tiling works, marble works, flooring works, earth filling works, screed concrete, spreading of geo textile cloth, spreading of wire mesh, spreading of aggregates with plants and more… MHN GROUPS are taking all types of properties for developing residential / commercial / industrial / land development projects on out rate and joint venture basis. We have 25 more years of experience in this profession for developing, execution and marketing. MHN GROUPS are taking all types electrical, plumbing works and pipeline works on turnkey basis. We are taking electrical contract works for residential projects, industrial projects, agricultural projects, layout development projects, commercial projects of Government and private sectors. We are taking both liasoning and contracting for underground cabling, open lines, installation of transformers and required panels, internal electrical wirings in all aspects. MHN GROUPS are also into as manufacturers, traders and exporters for all of construction materials in bulk order, in and around Bangalore. We have our own mining for manufacturing all type of required construction raw materials. We have table mould bricks, wire cut bricks, country bricks, light weight fly ash bricks, concrete blocks/bricks, aggregates, river sand, robo sand, silica sand, rubble stones, cinder, fly ash and more. MHN GROUPS are taking up all types flooring related works including materials and laying on turnkey basis. We have our own manufacturing units for manufacturing granites and tiles like sira granite, Sadarahalli granite, black granite, steel grey granite and more. Also we are supplying rough stone, hand dressed stones for construction projects like sira stones, Sadara Halli stones, Koira stones, chappidi stones, Narasapur stones, Mulbagal stones, pink stones,red stones, black stones, yellow stones, grey stones, cobble stones, pebbles stones, paving stones, sand stones, slate stones, mica stones, stacking stones and more. MHN GROUPS we are working in dry stone cladding works, wet stone cladding works, dry granite cladding works, wet granite cladding works, sira stone cladding, sadara halli stone cladding, chappidi stone cladding, rough stone cladding, hand dressed stone cladding, grey colour stone cladding, black colour stone cladding, pink colour stone cladding, red colour stone cladding, yellow colour stone cladding, koira stone cladding, mulabagal stone cladding, narasapur stone cladding, kadapa stone cladding, sandstone cladding, slate stone cladding, stacking stone cladding, mica stone cladding, kota stone cladding, granite stone cladding, bush hammered stone cladding, crazy cladding, butch work stone cladding and more… and in Paving Projects we are working in stone paving, rough stone paving, chapped stone paving, sadarahalli stone paving, sira stone paving, narasapur stone paving, mulabagal stone paving, granite paving, sandstone paving, slate stone paving, kota stone paving, tandoor stone paving, cobble stone paving, pebble stone paving, interlocking paver stone paving, grass pavers paving and more… MHN GROUPS are working & taking projects like - We have own Quarries for manufacturing raw blocks and well cutting units. In the same passion we are into Granites and Stone Designs Manufacturers, Traders, Exporters & Importers. We have Sadarahalli Stones, Sira Stones, Koira Stones, Narasapur Stones, Mulbagal Stones, Pink Stones, Red Stones, Yellow Stones, Black stones, Sand Stones, Slate Stones, Kota Stones, Taandoor Stones, Kadapa Stones, Bethamcherla Stones, Stacking Stones, Mica Stones, Mandana Stones and more. We have stones like Rough Stones, Machine Cut Stones, Rough Blocks, Hand Dressed Stones, Flamed Stones, Shot Blasted Stones, Antique Finished Stones and Stone Designs like Stone Cladding, Stone Paving, Kerb Stones, Coping Stones, Cobble Stones, Pebble Stones, Pergolas, Murals, Monuments, Fountains, Sculptures, Carved Stones, Construction of Gabion Walls & Feature Walls, Reinforced Earth Walls, Reinforced Soil Walls, Geogrid Stone Facia Walls and more… MHN GROUPS are working as also Hard Landscaping Contractors, Soft Landscaping Contractors, Interior Contractors, Fabricators, Carpentry Contractors, Plumbing Contractors, Painting Contractors, Flooring Contractors, Electrical Contractors, Demolishing Contractors, Guniting Contractors, Lining Contractors, Grit Plastering Contractors, Piling Contractors and Building Materials Manufacturers & Suppliers of Country Bricks, Table-mould Bricks, Wire Cut Bricks, Fly Ash Bricks, Concrete Blocks & Bricks, Aerocon Blocks, Size Stones, Rubble Stones, Aggregates, Fly Ash, Cinder, Clay Tiles, Flooring Tiles, River Sand, Robo Sand, M-Sand, Silica Sand, Filter Sand, Chappidi Stones, Red Earth / Red Soil and more… MHN GROUPS are working as excavation and digging of canals, dams, rivers, reservoirs, ponds with mass desilting and dredging works. We are taking mass earth cutting and filling works with supply of soil / earth / gravel / morrum / clay / mud, levelling and compaction with vibro rollers. We have capacity of supply and filling of earth in day of minimum 1000+ loads within Bangalore. We are taking all types of time bond earth filling and earth compaction works anywhere in India. Also we are regularly supplying filling soil, filling earth, filling gravel, filling morrum, filling mud, filling soil, red soil, garden soil and more…In adding to construction we are taking interior projects and exterior projects for residential, industrial, commercial, hotel and resorts projects with civil works, concreting works, flooring works, tiling works, plumbing works, carpentry works, marble works, pop works, false ceiling works, wood works, electrical works, roofing works, nylon mesh suppliers, geotextile suppliers, masonry works, demolishing works, rock breaking works, painting works, plumbing works, vdf flooring works, cabling works, waterproofing works, guniting works, lining works, grit plastering works, rain water harvesting contractors, real-estate promoters, radiography testing, revetment contractors, shot blasting contractors, sand blasting contractors, ugd works, tar road contractors, UPVC contractors, fabrication works and more… MHN GROUPS are one of the leading specialised construction company for taking up various types of Gabion Walls & Feature Walls, Reinforced Earth Wall & Reinforced Soil Walls, Reinforced Stone Walls & Reinforced Concrete Walls, Reinforced Geogrid Walls, Gravity Retaining Walls and more…We are manufacturing, supplying and using Reno Mattresses for construction of gabion walls and are usually used to protect the walls of channels, retaining walls on slopes, landscape construction, protect the pipelines, regulation of river channels, strengthening of the banks, soil stabilisation and embankment stability for flood protection. Reno Mattresses, or revet mattresses, or gabion type structures manufactured in customised size of double twist mesh. These are made with gi wires (GI welded mesh / GI double twisted woven mesh / GI double twisted woven mesh with PVC coated) with more planar area and relatively small in thickness. The blending of Reno Mattress structures into the environment is rapid and pleasing. The voids in the rock fill, geotextile fabric become progressively filled with silt, promoting vegetation growth which is essential to the preservation and maintenance of the ecological balance of the surrounding environment. MHN GROUPS are the one of the experienced firm for developing all types of land development projects with all types of modern amenities and in layout contracting. We have vast experience in past 25 years and we developed more than 7000 acres of land all over in India. We are taking up all types of activities for developing layouts like residential layouts, commercial layouts. We are executing following type of work activities in land development project... *Scrubbing and levelling of land. *Making drains in stone slab / concrete / size stones / concrete blocks. *Drain cover slabs with precast or on cast at site. *UGD lines with pvc pipes / stoneware pipes. *Water Lines with gi / cpvc / upvc / pvc pipes. *Construction of oht / stp / sump tanks. *Construction of compound walls with concrete blocks / precast concrete slabs / size stones / stone slabs / barbed wire fencing / chain link mesh fencing / feature wall designs. *Developing roads with soling stone laying / gsb sub base laying / wmm sub base laying / asphalting of roads / concrete roads / vdf roads paver tile roads / cobble laid roads. *Fixing of kerb stones along with interlock paved foot paths. *Electrical lines with open lines / underground cabling. *Fixing of street lights with electrical / solar. *Marking of sites / roads / other areas. *Fixing of main gates with arched designs.

Unofficial implementation of Compact Convolution Transformers with CBAM Attention

In this project Compact Convolutional Vision CCT is used for Gastrointestinal diseases detection using Gastroscopy

A Generalized Thermal Comfort Model using Thermographic Images and Compact Convolutional Transformers: Towards Scalable and Adaptive Occupant Comfort Optimization

DTM: Diffusion Transformer Model Guided by Compact Prior in Low-dose PET Reconstruction

PyTorch implementation of "Compact Convolutional Transformer for Subject-Independent Motor Imagery EEG-based BCIs"

Compact Clinical NLP pipeline for extracting structured entities from free-text medical notes using transformer-based CNER.

EAL-ICNet, a lightweight segmentation framework, integrates the advantages of both CNNs and Transformers to achieve a strong balance between segmentation accuracy, model compactness, and inference efficiency.

A comprehensive guide on how to implement and train a Compact Convolutional Transformer model that is specifically designed for classifying skin cancer.