Deaf and mute people use sign language to communicate. Unlike acoustically conveyed sound patterns, sign language uses hand gestures, facial expressions, body language and manual communication to convey thoughts. Due to the considerable time required in learning Sign Language, people find it difficult to communicate with specially-abled people, creating a communication gap. Hence conventionally, people face problems in recognizing sign language. Moreover, different countries have their respective form of sign gesture communication which results in non-uniformity. The Indian Sign Language used in India is largely different from the American Sign Language used in the US, mostly because of the difference in culture, geographical and historical context. Somewhere between 138 and 300 different types of sign language are currently being used throughout the world. Sign language structure varies spatially and temporally. We have identified these as a major barrier in communicating with a significant part of society. And hence, we propose to design a system that recognizes different signs and conveys the information to people. The component of any sign language consists of hand shape, motion and place of articulation. When combined, these three components (together with palm orientation) uniquely determine the meaning of the manual sign. For sign language identification, sensor-based and vision-based methods are used. In vision-based gesture recognition technology, a camera reads the movements of the human body, typically hand movements and uses these gestures to interpret sign language; whereas in sensor-based methods, real-time hand and finger movements can be monitored using the leap motion sensor. We aim at developing a scalable project where we will be considering different hand gestures to recognize the letters and words. We plan to use different deep learning models to predict the sign. This may be developed as a desktop or mobile application to enable specially-abled people to communicate easily and effectively with others. However, this project can later be extended to capture the whole vocabulary of ASL (American Sign Language) through manual and non-manual signs.