Problem Statement Introduction So far, in this course, you have learned about the Hadoop Framework, RDBMS design, and Hive Querying. You have understood how to work with an EMR cluster and write optimised queries on Hive. This assignment aims at testing your skills in Hive, and Hadoop concepts learned throughout this course. Similar to Big Data Analysts, you will be required to extract the data, load them into Hive tables, and gather insights from the dataset. Problem Statement With online sales gaining popularity, tech companies are exploring ways to improve their sales by analysing customer behaviour and gaining insights about product trends. Furthermore, the websites make it easier for customers to find the products they require without much scavenging. Needless to say, the role of big data analysts is among the most sought-after job profiles of this decade. Therefore, as part of this assignment, we will be challenging you, as a big data analyst, to extract data and gather insights from a real-life data set of an e-commerce company. In the next video, you will learn the various stages in collecting and processing the e-commerce website data. Play Video2079378 One of the most popular use cases of Big Data is in eCommerce companies such as Amazon or Flipkart. So before we get into the details of the dataset, let us understand how eCommerce companies make use of these concepts to give customers product recommendations. This is done by tracking your clicks on their website and searching for patterns within them. This kind of data is called a clickstream data. Let us understand how it works in detail. The clickstream data contains all the logs as to how you navigated through the website. It also contains other details such as time spent on every page, etc. From this, they make use of data ingesting frameworks such as Apache Kafka or AWS Kinesis in order to store it in frameworks such as Hadoop. From there, machine learning engineers or business analysts use this data to derive valuable insights. In the next video, Kautuk will give you a brief idea on the data that is used in this case study and the kind of analysis you can perform with the same. Play Video2079378 For this assignment, you will be working with a public clickstream dataset of a cosmetics store. Using this dataset, your job is to extract valuable insights which generally data engineers come up within an e-retail company. So now, let us understand the dataset in detail in the next video. Play Video2079378 You will find the data in the link given below. https://e-commerce-events-ml.s3.amazonaws.com/2019-Oct.csv https://e-commerce-events-ml.s3.amazonaws.com/2019-Nov.csv You can find the description of the attributes in the dataset given below. In the next video, you will learn about the various implementation stages involved in this case study. Attribute Description Download Play Video2079378 The implementation phase can be divided into the following parts: Copying the data set into the HDFS: Launch an EMR cluster that utilizes the Hive services, and Move the data from the S3 bucket into the HDFS Creating the database and launching Hive queries on your EMR cluster: Create the structure of your database, Use optimized techniques to run your queries as efficiently as possible Show the improvement of the performance after using optimization on any single query. Run Hive queries to answer the questions given below. Cleaning up Drop your database, and Terminate your cluster You are required to provide answers to the questions given below. Find the total revenue generated due to purchases made in October. Write a query to yield the total sum of purchases per month in a single output. Write a query to find the change in revenue generated due to purchases from October to November. Find distinct categories of products. Categories with null category code can be ignored. Find the total number of products available under each category. Which brand had the maximum sales in October and November combined? Which brands increased their sales from October to November? Your company wants to reward the top 10 users of its website with a Golden Customer plan. Write a query to generate a list of top 10 users who spend the most. Note: To write your queries, please make necessary optimizations, such as selecting the appropriate table format and using partitioned/bucketed tables. You will be awarded marks for enhancing the performance of your queries. Each question should have one query only. Use a 2-node EMR cluster with both the master and core nodes as M4.large. Make sure you terminate the cluster when you are done working with it. Since EMR can only be terminated and cannot be stopped, always have a copy of your queries in a text editor so that you can copy-paste them every time you launch a new cluster. Do not leave PuTTY idle for so long. Do some activity like pressing the space bar at regular intervals. If the terminal becomes inactive, you don't have to start a new cluster. You can reconnect to the master node by opening the puTTY terminal again, giving the host address and loading .ppk key file. For your information, if you are using emr-6.x release, certain queries might take a longer time, we would suggest you use emr-5.29.0 release for this case study. There are different options for storing the data in an EMR cluster. You can briefly explore them in this link. In your previous module on hive querying, you copied the data to the local file system, i.e., to the master node's file system and performed the queries. Since the size of the dataset is large here in this case study, it is a good practice to load the data into the HDFS and not into the local file system. You can revisit the segment on 'Working with HDFS' from the earlier module on 'Introduction to Big data and Cloud'. You may have to use CSVSerde with the default properties value for loading the dataset into a Hive table. You can refer to this link for more details on using CSVSerde. Also, you may want to skip the column names from getting inserted into the Hive table. You can refer to this link on how to skip the headers.

India GDP Analysis Problem Description - I NITI Aayog: Background NITI Aayog (National Institution for Transforming India) is a policy think tank of the Government of India; it provides strategic inputs to the central and the state governments to achieve various development goals. In the past, NITI Aayog has played an important role in initiatives such as Digital India, Atal Innovation Mission and various agricultural reforms and have designed various policies in education, skill development, water management, healthcare, etc. NITI Aayog was established to replace the Planning Commission of India, which used to follow a top-down model for policy making, i.e., it typically designed policies at the central level (such as the 5-year plans). On the other hand, NITI Aayog designs policies specific to the different states or segments of the economy. Finance Minister, Arun Jaitley, made the following observation on the necessity of creating NITI Aayog, "The 65-year-old Planning Commission had become a redundant organisation. It was relevant in a command economy structure, but not any longer. India is a diversified country and its states are in various phases of economic development along with their own strengths and weaknesses. In this context, a ‘one size fits all’ approach to economic planning is obsolete...". Project Brief We are working as the chief data scientist at NITI Aayog, reporting to the CEO. The CEO has initiated a project wherein the NITI Aayog will provide top-level recommendations to the Chief Ministers (CMs) of various states, which will help them prioritise areas of development for their respective states. Since different states are in different phases of development, the recommendations should be specific to the states. The overall goal of this project is to help the CMs focus on areas that will foster economic development for their respective states. Since the most common measure of economic development is the GDP, we will analyse the GDP of the various states of India and suggest ways to improve it. Understanding GDP Gross domestic product (GDP) at current prices is the GDP at the market value of goods and services produced in a country during a year. In other words, GDP measures the 'monetary value of final goods and services produced by a country/state in a given period of time'. GDP can be broadly divided into goods and services produced by three sectors: the primary sector (agriculture), the secondary sector (industry), and the tertiary sector (services). It is also known as nominal GDP. More technically, (real) GDP takes into account the price change that may have occurred due to inflation. This means that the real GDP is nominal GDP adjusted for inflation. We will use the nominal GDP for this exercise. Also, we will consider the financial year 2015-16 as the base year, as most of the data required for this exercise is available for the aforementioned period. Per Capita GDP and Income Total GDP divided by the population gives the per capita GDP, which roughly measures the average value of goods and services produced per person. The per capita income is closely related to the per capita GDP (though they are not the same). In general, the per capita income increases when the per capita GDP increases, and vice-versa. For instance, in the financial year 2015-16, the per capita income of India was ₹93,293, whereas the per capita GDP of India was $1717, which roughly amounts to ₹1,11,605. Problem Description - II Data The data is sourced from https://data.gov.in/, an Open Government Data (OGD) platform of India. The download instructions are provided in the next segment. The data for GDP analysis of the Indian states is divided into two parts: Data I-A: This dataset consists of the GSDP (Gross State Domestic Product) data for the states and union territories. Data I-B: This dataset contains the distribution of GSDP among three sectors: the primary sector (agriculture), the secondary sector (industry) and the tertiary sector (services) along with taxes and subsidies. There is separate dataset for each of the states. We are expected to read the dataset for the available states and join these (in Python) if needed. There are two parts to this project. In the first part, we will analyse and compare the GDPs of various Indian states (both total and per capita). The GDP of a state is referred to as the GSDP (Gross State Domestic Product). Then, we will divide the states into four categories based on the GDP per capita, and for each of these four categories, we will analyse the sectors that contribute the most to the GDP (such as agriculture, real estate, manufacturing, etc.). In the second part, we will analyse whether GDP per capita is related to dropout rates in schools and colleges. Part-I: GDP Analysis of the Indian States For each of the following steps of analysis, choose an appropriate type of plot for comparing the data. Also, ensure that the plots are in increasing or decreasing order for better comparison. For example, if we make a bar plot to compare the GDPs of the states, ensure that the bars are in either increasing or decreasing order of GDP. Part I-A: For the analysis below, use the Data I-A. First, we need to load the data in Python properly and then clean it. This also involves the treatment of missing values, we can choose to drop the row or column as well. Remember this will affect our next analysis and results drastically. Plot a graph for rows " % Growth over previous year" for all the states (not union territories) whose data is available, use as much data as possible for this exercise. Use the best fit line to represent the growth for each state. Draw a similar line graph for the nation as well. How will we compare the growth rates of any two states? Which states have been growing consistently fast, and which ones have been struggling? Rank top 3 fastest and 3 slowest-growing states. What is the Nation's growth rate? What has been the growth rate of my home state, and how does it compare to the national growth rate? Plot the total GDP of the states for the year 2015-16: Which Plot will we use for this? Why? (Remeber to plot the graph in a way such as it is easier to read and compare) Identify the top 5 and the bottom 5 states based on total GDP. What insights can we draw from this graph? What states are performing poorly? (Remember: this will not be solely based on total GDP) Part I-B: For the analysis below, use Data I-B. We can also use Data I-B along with Data I-A if required. Also, perform the analysis only for the duration 2014-15. Filter out the union territories (Delhi, Chandigarh, Andaman and Nicobar Islands, etc.) for further analysis, as they are governed directly by the central, not state governments. Plot the GDP per capita for all the states. Identify the top 5 and the bottom 5 states based on the GDP per capita. Find the ratio of the highest per capita GDP to the lowest per capita GDP. Plot the percentage contribution of the primary, secondary and tertiary sectors as a percentage of the total GDP for all the states. Which plot will we use here? Why? Why is (Primary + Secondary + Tertiary) not equal to total GDP? Can we draw any insight from this? Find correlation of percentile of the state (% of states with lower per capita GDP) and %contribution of Primary sector to total GDP. Categorise the states into four groups based on the GDP per capita (C1, C2, C3, C4, where C1 would have the highest per capita GDP and C4, the lowest). The quantile values are (0.20,0.5, 0.85, 1), i.e., the states lying between the 85th and the 100th percentile are in C1; those between the 50th and the 85th percentiles are in C2, and so on. Note: Categorisation into four groups will simplify the subsequent analysis, as otherwise, comparing the data of all the states would become quite exhaustive. For each category (C1, C2, C3, C4): Find the top 3/4/5 sub-sectors (such as agriculture, forestry and fishing, crops, manufacturing etc., not primary, secondary and tertiary) that contribute to approximately 80% of the GSDP of each category. Note-I: The nomenclature for this project is as follows: primary, secondary and tertiary are named 'sectors', while agriculture, manufacturing etc. are named 'sub-sectors'. Note-II: If the top 3 sub-sectors contribute to, say, 79% of the GDP of some category, we can report "These top 3 sub-sectors contribute to approximately 80% of the GDP". This is to simplify the analysis and make the results consumable. (Remember, the CEO has to present the report to the CMs, and CMs have limited time; so, the analysis needs to be sharp and concise.) Plot the contribution of the sub-sectors as a percentage of the GSDP of each category. Now that we have summarised the data in the form of plots, tables, etc., try to draw non-obvious insights from it. Think about questions such as: How does the GDP distribution of the top states (C1) differ from the others? Which sub-sectors seem to be correlated with high GDP? Which sub-sectors do the various categories need to focus on? Ask other such relevant questions, which we think are important, and note we insights for category separately. More insights are welcome and will be awarded accordingly. Finally, provide at least two recommendations for each category to improve the per capita GDP. Part-II: GDP and Education Dropout Rates In Part-I, we would have noticed that (one) way to increase per capita GDP is by shifting the distribution of GDP towards the secondary and tertiary sectors, i.e., the manufacturing and services industries. But these industries can thrive only when there is an availability of educated and skilled labour. In this part of the analysis, we will investigate whether there is any relationship between per capita GDP with dropout rates in education. Data Data II: This section will require the dropout rate dataset apart from the dataset that we used in Part-1 of the case study. Download instructions are provided in the next segment. Part-II: GDP and Education Analyse if there is any correlation of GDP per capita with dropout rates in education (primary, upper primary and secondary) for the year 2014-2015 for each state. Choose an appropriate plot to conduct this analysis. Is there any correlation between dropout rate and %contribution of each sector (Primary, Secondary and Tertiary) to the total GDP? We have the total population of each state from the data in part I. Is there any correlation between dropout rates and population? What is the expected trend and what is the observation? Write down the key insights we draw from this data: Form at least one reasonable hypothesis for the observations from the data About GDP analysis for India in the year for 2015-16 and recommendation for the individual states for increasing the GDP by focusing on various factor. Topics python statistical-analysis data-analysis gdp-analysis Resources Readme Stars 0 stars Watchers 1 watching Forks 0 forks Releases No releases published Packages No packages published Languages Jupyter Notebook 100.0% Footer © 2022 GitHub, Inc. Footer navigation Terms Privacy Security Status Docs Contact GitHub Pricing API Training Blog About

This repository contains a Fraud Detection System consisting of ingestion, processing, analyses and real-time analytics systems to visualize trends and detect fraudelent activity in transactions

Analysing real time urban crime hotspots by integrating socioeconomic data with crime trends using spatiotemporal and causal inference techniques.

An end-to-end real-time data streaming pipeline that leverages Kafka and Spark Streaming to analyse social media sentiment trends.

Healthcare Business Plan Index 01. What is Healthcare Business Plan 02. A Business Plan should be SMART 03. Components of a Healthcare Business Plan 04. Role of Forte Healthcare 05. Summary 01. What is Healthcare Business Plan? What Is Healthcare Business Plan A Healthcare business plan is the foundation or basis for the development and existence of any Healthcare facility or services. Even healthcare regulators encourage new players in the market to have a robust business plan to ensure they understand the local regulatory landscape, revenue cycle management including insurances, laws protecting patients and physicians, licensing, pricing, working capital, services and treatments, and having a business continuity plan, this ensures quality healthcare and catering to the needs of the market. The Business plan is then cascaded to the facility layouts ensuring it meets all local regulations on the Architectural drawings and the facility infrastructure is developed to offer quality patient care. The Healthcare industry is continuously evolving with technological advancements, hence having a strategic business plan is important for a successful project with allocated budgets CAPEX and OPEX as well as working capital clearly defined for a successful venture. A healthcare business plan includes market feasibility and financial feasibility which helps in identifying the segment and patient base for the facility. A healthcare business plan includes market feasibility and a financial feasibility report. Market & financial Feasibility Market &Amp; Financial Feasibility Market feasibility helps understand market dynamics, stakeholder competition, target audience, and key prospects. The study also includes an analysis of demography, demand, and market capacity. Quantitative market projections can help the business identify opportunities in the region and help to decide what treatment gaps are present. Competitor analysis includes a direct and indirect competitor list, which helps a business evaluate a competitor’s weakness, pricing market share, services provided, and also market reviews. For an existing business, a competitor analysis helps to improvise or add new product or service lines. The financial feasibility study includes: Long-term lease or Purchase of Real Estate Asset Location Analysis and validation Competition Analysis Local regulatory compliance Design and construction budget estimate Staffing and budgets Medical equipment and budgets IT infrastructure Revenue Cycle Management – Insurance Operations budget Working Capital Revenue cycle management and Insurance play a pivotal play in the feasibility and cash flows. Insurances work differently across the EMEA region with a few countries with mandatory insurance and others on a cash basis, it’s crucial that the financial feasibility captures all modes of payments along with actual pricing for Healthcare facilities. 02. A Business Plan should be SMART A Business Plan Should Be Smart - Specific Specifically, the model should cater to the ideology with which the business is being set up. It should define the mission, vision, and long-term goal of the project and should have a core specialty A Business Plan Should Be Smart - Measurable Measurable, the model should provide the estimated budget for the project to complete which can be used as a reference in the various stages of the project. More important is the measurable services and revenue streams need to be aligned for robust operations A Business Plan Should Be Smart - Achievable Achievable, Achievable, the business plan gives an idea if the proposed business idea or concept is realistic and achievable and that investment in the field will be beneficial. Also, all milestones are to be listed. A Business Plan Should Be Smart - Relevant Relevant, in order to align the various stakeholders, the business plan should be designed in a way that it appeals relevant to their interests and the healthcare operations they are interested to venture A Business Plan Should Be Smart - Time Bound Time-bound, a business plan should include a time frame in which the business will break even and when it will become profitable. 03. Components of a Healthcare Business Plan A Healthcare business plan also includes… Your vision for the future A business plan must include the vision of the project. Then, it should be made sure that the plan becomes integral to the culture of the organization. The plan should always be shared with physicians, employees, board members, and top leaders or operators who will be engaged with the core team. Components Of A Healthcare Business Plan -Your Vision For The Future Components Of A Healthcare Business Plan - Swot SWOT Business plans should include a SWOT matrix. The strengths, weaknesses, opportunities, and threats of a business can be assessed using this analysis. SWOT analyses need to be specific, realistic, and compare to other competitors evaluating needs vs gaps, and should be updated as the goals change with time. It should be comprehensive and include all elements of healthcare operations. Operations strategy Operations strategy should be clear in terms of sourcing local physicians vs from overseas and other staff which shall define a well-balanced team that will be attractive to patients who are more comfortable with existing Physicians and hence follow them. Also, the category identification and strategy henceforth are critical whether it’s a Hospital, Day Surgery, or Clinic operating in their respective segment will define the most suitable operations strategy. Components Of A Healthcare Business Plan - Operations Strategy Components Of A Healthcare Business Plan - Define Target Audience Define Target Audience In the opinion of industry experts, marketing efforts are most successful if they are targeted. As a result, a business plan should define the common characteristics of your current patients to identify potential patients. A methodology that helps reach out to the audience and is tailored according to their preferences should be adopted. In UAE / GCC this also expands into segments or categories of individuals where insurances are validated. Furthermore, UAE is becoming a hub of medical tourism and hence this should be considered while formulating patient flow to the facility. 04. Role of Forte Healthcare Role Of Forte Healthcare Who We are? Forte Healthcare has over 23 years of experience in start-ups, including over 120+ in the healthcare sector. It has also successfully completed over 3.6 billion dollars in mergers & acquisitions across sectors including healthcare, hospitality, and special projects. For over 23 years, Forte Healthcare has developed and managed healthcare facilities that have been profitable across Dubai and GCC. Additionally, Forte Healthcare apart from managing the operations also recommends to their clients a list of highly qualified physicians handling costs related to identification, negotiation, and recruitment. Role Of Forte Healthcare - Our Services Our Services Forte Healthcare advises healthcare start-ups and existing businesses on developing and operating their facilities according to unique and segmented business models. We provide comprehensive services from the inception of a project up to developing and building a highly productive healthcare business with state-of-the-art facilities and services and providing value-added care for our patients. Featuring a customer-driven strategy, Forte Healthcare aims to consistently meet customer expectations and ensure sustainable growth The team at Forte closely evaluates the demand and supply across geographical location and suggest the appropriate positioning of the clinic brand. Financial feasibility is expanded across all services and with a cost vs benefit analysis to evaluate what’s relevant in each market, also the segmentation is vital for healthcare facilities given tier-based insurances and facilities in GCC / Middle East countries. An essential goal of vital arranging is to recognize inside qualities and deficiencies as well as outside dangers and opportunities. As a result of this contribution, you will have hierarchical ownership, which is vital to the execution and maintenance of your arrangement. 05. Summary Forte Healthcare Business Plan Summary Forte Healthcare can assist in developing general hospitals, specialized hospitals, aesthetic centers, daycare surgery centers, sports medicine clinics, stem cell laboratories, and specialized dental centers under a format most suitable for operations and business in a country/location. The Directors at Forte Healthcare come from successfully starting and operating successful Aesthetic & Wellness services chains in Asia, and GCC including UAE, Oman, Qatar, Kuwait, Bahrain & KSA. Our expertise starts from the company structure, incorporation, bank accounts, commercial lease negotiations, location validation, facility layouts, regulatory compliance and approvals, design and architecture, construction management, Recruitment and onboarding, Branding, Social media marketing, PR, Launch and manage the facility ensuring business and operations objectives are met. Sample of a Business Plan A business plan made by the team of Forte Healthcare is as follows: Market Overview And Trends Market overview and trends Demographics Regulatory Supply gap Competitor Analysis Competitor analysis Direct competitors Indirect competitors Service Mix &Amp; Analysis Service Mix & Analysis Healthcare Treatments Proposed Pricing Phasing of Treatments Financial Plan Financial Plan CAPEX Project timelines Revenue & EBITDA

A real-time data pipeline for analysing Teamfight Tactics trends

An all-in-one application so you can effectively analyse market trends and obtain real-time stock news.

To get stock pricing trends of various companies altogether; plot, analyse and predict them after intaking real-time data from Yahoo finance using python3

A Python project to analyse Fantasy Premier League player performance, optimise team selection, and explore trends using real-time data from the FPL API

A real-time cryptocurrency price analyser built with Python that fetches live prices, displays 24-hour changes, and visualizes historical price trends through interactive charts. The project focuses on accurate data retrieval, clean visualization, and modular design.

To develop a comprehensive credit card weekly dashboard that provides real-time insights into key performance metrics and trends, enabling stakeholders to monitor and analyse credit card operations effectively.

PulseTag is an intelligent social media utility designed to maximize post engagement through AI-driven trend analysis. It bridges the gap between content creation and discovery by analysing social media posts in real-time and generating optimized hashtag strategies.

The main purpose is to capture tweets, analyse them regarding the most used words and hashtags and classifying them regarding their sentiments(POSITVE, NEGATIVE OR NEURAL). Search real time data and insights about hashtags on tweeter, popularity ranking, related hashtags and trends. Use top trending hashtags to reach a broader audience. Track any tweet to get in-dept analysis

A data-driven project designed to analyse music sales trends on BandCamp. It features an ETL pipeline for data extraction, transformation, and loading into a PostgreSQL RDS database, a Streamlit dashboard for real-time visualisations, automated reporting and alerts to provide insights into popular genres and artists.

Digitally enabled customers like to shop on the run, and that is the reason why online shopping is one of the most popular online activities worldwide. In 2019, global e-commerce sales amounted to 3.53 trillion USD and are projected to grow to 6.54 trillion USD in 2022. The analytical capabilities of big data have had a positive impact across industries, including e-commerce. Big data tools improve business performance by enabling companies to analyse trends and current consumer behavioural patterns and offer better and more customised products. For the purposes of this project, you have been tasked with computing various Key Performance Indicators (KPIs) for an e-commerce company, RetailCorp Inc. You have been provided real-time sales data of the company across the globe. The data contains information related to the invoices of orders placed by customers all around the world. You will get to know the details of the data in the next segment. At the industry level, an end-to-end data pipeline is built for this purpose. Tools such as HDFS(Hadoop Distributed File System) are used to store the data that is processed by the real-time processing framework and then shown on a dashboard with tools such as Tableau and PowerBI.

**Project Description (100 words):** The Real-Time Job Trend Analyzer is a web-based dashboard built using Python and Streamlit that visualizes current job market trends. It scrapes job listings from major platforms and presents insights such as the most in-demand job titles, top hiring cities, and essential skills like Python.

Real-Time Trending Topic Sentiment Analyser

A real-time, reddit trends sentiment analyser

Interactive dashboard to analyse financial asset trends using real time market data

A real time dashboard for cryptocurrency data where its trends and trends prediction can be analysed.

Real time NLP dashboard that analyses news trends and sentiment using NewsAPI and Streamlit

A tool for analysing real-time gym attendance data and visualising trends using Chart.js.

A crypto dashboard with real-time cryptocurrency data to analyse trends, news, and metrics for Bitcoin, Ethereum, and more.

Pull real-time earthquake data from USGS. Display magnitude and frequency on an interactive map. Analyses risk using historical trends.

An Agentic Financial Market Intelligence app that analyses real-time financial events and trends using the GDELT Open Intelligence dataset to predict stock trends and assess investment risks.

A Python-based tool for screening and analysing market trends by combining financial data collection, model development, and real-time web scraping.

A pilot study on data mining and text analytics. Using python to analyse a dataset to mimic real-time news analysis and trend prediction.

pokie analyses tool helps users determine trends and win clusters with real time updates. chat forums to see where hot or cold machines are located.

A portal that helps writers to publish their ideas, views, and knowledge. Real-time user insights of articles provided to an admin for analyses of trends