大数据面试题，从0到1走向架构师之路。Flink、Spark、Hive、HBase、Hadoop、Kettle、Kafka...

Intention is to use different algorithms of Machine Learning in R-Programming and Python to work with various dimension and range of data. The implementation will be based on BigData framework and main point of attraction will be Spark and Hive includinh hadoop

A curated list of awesome frameworks, libraries and software for IOT/BigData/AI/Cloud/Python/Java...

Implementation of Paper "AutoRed: Automated Attack Scenario Generation Framework for Red Teaming of LLMs" [IEEE BigData 2024 Industry & Governance Track]

Official implementation of WaveGNN (IEEE BigData 2025), integrating a decay-aware Transformer and dynamic Graph Neural Network framework for modeling irregular multivariate time series without imputation. Achieves consistent, robust, and interpretable performance on clinical datasets.

A forecasting study based on BigData methods through the example of pesticide sales. Predictions are based on linear regression and two neural networks frameworks, to obtain statistics and compare results.

Consists of few case studies done under BigData using MapReduce framework.

Repo of various examples of frameworks and tools used for bigdata, with annotations about caveats and how to use.

Code to make an analysis of the political campaign spent in 2010 using the bigdata framework storm.

A framework for using multi-threading in operations that can be decomposed into several tasks working just-in-time. Typically to process a flow of objects in a bigdata context.

This project deals with implementation of BigData using Hadoop framework. I have provide the support of BigData HADOOP version 1. My product provide three ways to setup hadoop cluster -} Automatic Configuration -} Manual Configuration -} On Demand Configuration TECHNOLOGY USED: - OS Used RHEL 7.2 - SSH, NFS, DNS and YUM Configuration - Framework (Map Reduce, Pig, Hive, ZooKeeper) - FRONTEND - python CGI integrated with HTML - BACKEND - HTML, CSS, Java-Script, Ajax.

BigData handling with Hadoop framework and Google Dataproc. This program counts the occurances of trigrams on any dataset of txt files using the Hadoop MapReduce framework. It also sorts the trigrams by their size in descending order by implementing a Comparator class. A complimentary report is produced with it highlighting all the additional features to optimise the processing of Big Data, such as a FileInputCobiner.

Demonstrate the knowledge of BigData systems: Hadoop Distributed File System, NoSQL data base HBase, Resource manager YARN, Processing Frameworks such as MapReduce, Hive, Kafka, and also the Spark eco-system. Develop both batch processing and stream processing data-intensive applications using Apache Hadoop and Apache Spark technologies.

Public sector banks deal with large amount of data on a daily basis. To manage such large datasets, Bigdata analytics techniques like Mapreduce that run on Hadoop framework are used. The basic entities in public sector are customers and their transactions. The question of interest here is to analyse the behavioural patterns of the spending habits of customers and patterns of their transactions. These behaviours are encoded into data through certain medium that capture behaviour. Various data sources are then accessed, prepared, consolidated and analyzed. Ultimately, this gives rise to insights into the patterns of the expenses by the customers across a period of time.

The rise of different topics all over the interneton a daily basis is increasing rapidly which leads to a seriesof fluctuating data in all areas of the world and following upa particular topic and getting related information regardingthe topic is essential to keep with the trend. Most of thesetopics trend with their related hash tags on twitter on a dailybasis, and the data related to the trending topics are vastand require a capable and efficient framework to stream,analyse and cluster the topics based on the topic’s hash tag. Bigdata frameworks such as Apache Spark has high computingcapacity to manage such big data at a faster rate in an efficientway.The challenge of analyzing the trending topics on twitterfor real-time topic clustering to get a clear and only relatedtweets and information regarding a topic is the motivation forthe following applied clustering techniques applied using ApacheSpark and clustering algorithms. The clustering algorithm isconstructed using Spark LDA (Latent Dirichlet Allocation),the algorithm takes the live twitter stream data as an inputand the data is Represented using a vector space model,thenon-negative dimension weights highlight the significance of thein accordance term functions, one essential assets of the sortof function space is high dimensions which occurs.The LDAalgorithm takes the approximate assumed topics in the documentand will assign every word in the document to a temporarytopic using LDA which is a probabilistic model that posits aset of global topics and a set of document topics, the LDAprocess is applied iterative by loop each word in the document and update the topic assignment based on the criteria established.

DE Stuff - Programming & DSA in Python Distrubuted computation & storage -Hadoop - imp - spark,spark sql file format - json,avro,parquet type of data - structiured & semi sturectured processing mechniasm - batch & real time pub sub -kafka or AWS Kinesis data warehuse desginng amazon - ecomerce, netflix sql complex transactional & no sql databases - key value ,document,columanr base,graph base db,-mongodb,caseendra,hbase - data visualization tools- ETL tools cloud services- experience 🔥𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀🔥 🔴 Data Engineer Roadmap Document : ➡️https://docs.google.com/document/d/1g...​ 🔴 Python : ➡️ https://www.youtube.com/watch?v=_uQrJ...​ ➡️ https://www.programiz.com/python-prog...​ 🔴 Scala : ➡️ https://www.youtube.com/watch?v=LQVDJ...​ ➡️ http://allaboutscala.com/​ 🔴 Java : ➡️ https://www.youtube.com/watch?v=eIrMb...​ ➡️ https://beginnersbook.com/java-tutori...​ 🔴 Linux, Unix, Shell Scripting : ➡️ https://practice.geeksforgeeks.org/ba...​ ➡️ Above one is free course with invitation code - ELEARNINGBLINUX 🔴 Data Structures & Algorithms : ➡️ https://www.youtube.com/watch?v=5_5oE...​ ➡️ https://www.geeksforgeeks.org/​ ➡️ https://leetcode.com/​ 🔴 DBMS : ➡️ https://www.youtube.com/watch?v=kBdlM...​ ➡️ https://www.studytonight.com/dbms/​ 🔴 SQL Scripting : ➡️ https://www.youtube.com/watch?v=HXV3z...​ ➡️ https://www.youtube.com/watch?v=7S_tz...​ ➡️ https://www.w3schools.com/sql/​ 🔴 Basic Terminologies In BigData : ➡️ https://data-flair.training/blogs/wha...​ ➡️ https://www.edureka.co/blog/what-is-b...​ 🔴 Data Exploration Libraries : ➡️ Pandas - https://www.youtube.com/watch?v=UB3DE...​ ➡️ NumPy - https://www.youtube.com/watch?v=DI8wg...​ 🔴 Data Warehousing Concepts : ➡️ https://www.youtube.com/watch?v=J326L...​ ➡️https://www.tutorialspoint.com/dwh/dw...​. 🔴 BigData Frameworks (Hadoop, Hive, Spark, Sqoop, Nifi, Flume) : ➡️ https://www.youtube.com/results?searc...​ ➡️ https://www.youtube.com/user/edurekaIN​ ➡️ https://data-flair.training/​ ➡️ https://www.edureka.co/​ 🔴 Workflow Schedulers, Dependency Management : ➡️ https://www.youtube.com/watch?v=niJ06...​ ➡️ https://www.youtube.com/watch?v=6RebQ...​ 🔴 NoSQL Databases : ➡️ HBase - https://www.youtube.com/watch?v=NOX6-...​ ➡️ Cassandra - https://www.youtube.com/watch?v=iDhIj...​ ➡️ Elastic Search - https://www.youtube.com/watch?v=1Envk...​ ➡️ MongoDB - https://www.youtube.com/watch?v=pWbMr...​ 🔴 Apache Kafka : ➡️ https://www.youtube.com/watch?v=daRyk...​ 🔴 Dashboarding Tools : ➡️ Tableau - https://www.youtube.com/watch?v=aHaOI...​ ➡️ PowerBI - https://www.youtube.com/watch?v=3u7MQ...​ ➡️ Grafana - https://www.youtube.com/watch?v=CjABE...​ ➡️ Kibana - https://www.youtube.com/watch?v=gQ1c1...​ 🔴 BigData Services in Cloud (AWS) : ➡️ https://www.youtube.com/watch?v=k1RI5...​ ➡️ https://www.youtube.com/watch?v=8PyLr...​ ➡️ https://www.simplilearn.com/aws-big-d...​ 𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝘁𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗦𝗤𝗟 : 📌 Joins 📌 Group By 📌 Nested joins 📌 Case-When conditions 📌 Window functions 𝗧𝗲𝗰𝗵 𝘀𝘁𝗮𝗰𝗸 𝗳𝗼𝗿 𝗿𝗲𝗮𝗹𝘁𝗶𝗺𝗲 𝗱𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 : 📌 Apache Kafka 📌 Apache Flink 📌 Apache Storm 📌 AWS Kinesis 📌 Spark Streaming 𝗕𝗶𝗴𝗗𝗮𝘁𝗮 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 𝗮𝗻𝗱 𝗛𝗮𝗱𝗼𝗼𝗽 𝗲𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 : 📌 Hadoop architecture, Map-Reduce, HDFS, Yarn 📌 Apache Spark 📌 Hive 📌 Flume 📌 Sqoop 📌 Zookeeper 📌 Ambari, Hue 📌 Oozie, Airflow, Azkaban 𝗗𝗮𝘁𝗮 𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗧𝗼𝗼𝗹𝘀 : 📌 Tableau 📌 Power BI 📌 Qlik Sense 📌 Grafana 📌 Kibana 𝗧𝗿𝗮𝗻𝘀𝗮𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 : 📌 Amazon Aurora 📌 PostgreSQL 📌 MySQL 📌 MariaDB 📌 Oracle 📌 SQL Server 𝗡𝗼-𝗦𝗤𝗟 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 : 📌 DynamoDB 📌 Cassandra 📌 MongoDB 📌 ElasticSearch 📌 HBase 📌 Couchbase 📌 Redis Book - Practical Statistics for Data Scientists: 50+ Essential Concepts Using R and Python 2nd Edition,