My Insight Data Engineering Fellowship project. I implemented a big data processing pipeline based on ​lambda architecture​, that aggregates Twitter and US stock market data for user sentiment analysis using open source tools - ​Apache Kafka ​for data ingestions, Apache Spark ​& ​Spark Streaming ​for batch & real-time processing, ​Apache Cassandra f​ or storage, ​Flask​, ​Bootstrap and ​HighCharts f​ or frontend.

The coding has been done on Python 3.65 using Jupyter Notebook. This program fetches LIVE data from TWITTER using Tweepy. Then we clean our data or tweets ( like removing special characters ). After that we perform sentiment analysis on the twitter data and plot it for better visualization. The we fetch the STOCK PRICE from yahoo.finance and add it to the data-set to perform prediction. We apply many machine learning algorithms like (random forest, MLPClassifier, logistic regression) and train our data-set. Then we perform prediction on untrained data and plot it with the real data and see the accuracy.

Stock Market Trend Prediction using sentiment analysis Leveraging machine learning and sentiment analysis, we accurately forecast stock market trends. Our project combines advanced algorithms like BERT and Naïve Bayes with sentiment analysis from Twitter and other sources. By analyzing sentiment and historical price data, we provide insights

A company tracking app that requests real time stock data via websocket connections and draws interactive graphs with d3.js and HTML5 canvas. Uses Twitter streaming data and algorithmic analysis to explore relationship between company performance and Twitter sentiment. HTML/CSS, Javascript, React, Node, Express, MongoDB, d3.js

The information system chosen for the project was a stock investment management website providing live prices, historical data, news articles, etc and also basic analysis and recommendations using data mining techniques. 1. Crawling and parsing Yahoo-Finance, Reuters and Twitter data (Java, twitter4j). 2. Web Interface using J2EE and Struts-2 framework. jQuery (highstocks lib) for showing technical charts. 3. Database integration, data cleaning, feature selection on the collected data and applying linear regression and classification algorithms : SVM, Naive Bayes to produce detailed analysis and recommendations.

No description available

This project addresses the problem of sentiment analysis on Twitter. The goal of this project was to predict sentiment for the given Twitter post using Python. Sentiment analysis can predict many different emotions attached to the text, but in this report, only 3 major were considered: positive, negative and neutral. The training dataset was small (just over 5900 examples) and the data within it was highly skewed, which greatly impacted on the difficulty of building a good classifier. After creating a lot of custom features, utilizing bag-of-words representations and applying the Extreme Gradient Boosting algorithm, the classification accuracy at the level of 58% was achieved. Analysing the public sentiment as firms trying to find out the response of their products in the market, predicting political elections and predicting socioeconomic phenomena like the stock exchange.

The project focused on the use of public data to assess the economic situation in the country based on the state of the stock market and national means of payment, in particular - of the national currency. As sources are used: Open data Ministry of Finance of the Russian Federation These Moscow Exchange Google Finance Data Technologies used: Backend: Databases (relational) - Microsoft SQL Server 2014 Databases (multivariate) models DataMining, OLAP-cube - Microsoft Analysis Services 12.0 Веб-сервер - Windows Server 2012 / Internet Information Services Самописный ASP.NET HTTP Restful интерфейс для взаимодействия с Frontend ETL (загрузка и пре-процессинг данных, управление обновлением данных) SQL Server Integration Services 2014 (разработка в Visual Studio 2013, SSDT) Frontend: AngularJS ChartJS Twitter Bootstrap These were chosen so that the detail (granularity) in the set is not less than 1 day. The result has been created and filled with data analytic repository (Kimball model, topology - star), which was used to build a multi-dimensional databases and OLAP-based cubes on it, as well as models of analysis of data on two main algorithms: Microsoft Time Series, Microsoft Neural Network . To ensure interoperability frontend and backend server for backend-server was set up HTTP-Restful interface JSON-issuing documents in the form of finished sets. The project includes two main areas: Intelligent visualization of open data Analysis of open data and the construction of forecasts based on them Intelligent visualization involves the use of MDX-queries to the OLAP-cube, followed by depression (drilldown) in the data, the system allows the user to quickly find the "weak points" of the economy, as part of the data collected. To predict the time a standard mix of algorithms ARTXP / ARIMA, without the use of queries involving cross-prediction (but it is possible to enroll in the system correct data). These algorithms have been tested primarily on foreign exchange rates (US dollar) and the assets of banks included in the special list of Ministry of Finance. In addition, for assets shows the different customization options algorithms - a long-term, short-term and medium-term (balanced) plan. Assessing the impact of oil prices and foreign currency exchange rate for the total market capitalization was conducted on a sample of the data collected: companies with a total market capitalization of 100 to 500 million rubles, present in the market during 2013-2015 Analytical server builds the neural network receiving the input exchange rates, companies, the weighted average share price, total capitalization of the company and the price of oil to requests received models give the opportunity to evaluate the growth rate of \ fall (if at all) the company's capitalization at historical exchange rates and / or the cost of oil. Built a system can expand to include new indicators, which will significantly increase the accuracy of forecasting.

Archimedes 1 is a bot based sentient based trader, heavily influenced on forked existing bots, with a few enhancements here or there, this was completed to understand how the bots worked to roll the forward in our own manner to our own complete ai based trading system (Archimedes 2:0) This bot watches [followed accounts] tweets and waits for them to mention any publicly traded companies. When they do, sentiment analysis is used determine whether the opinions are positive or negative toward those companies. The bot then automatically executes trades on the relevant stocks according to the expected market reaction. The code is written in Python and is meant to run on a Google Compute Engine instance. It uses the Twitter Streaming APIs (however new version) to get notified whenever tweets within remit are of interest. The entity detection and sentiment analysis is done using Google's Cloud Natural Language API and the Wikidata Query Service provides the company data. The TradeKing (ALLY) API does the stock trading (changed to ALLY). The main module defines a callback where incoming tweets are handled and starts streaming user's feed: def twitter_callback(tweet): companies = analysis.find_companies(tweet) if companies: trading.make_trades(companies) twitter.tweet(companies, tweet) if __name__ == "__main__": twitter.start_streaming(twitter_callback) The core algorithms are implemented in the analysis and trading modules. The former finds mentions of companies in the text of the tweet, figures out what their ticker symbol is, and assigns a sentiment score to them. The latter chooses a trading strategy, which is either buy now and sell at close or sell short now and buy to cover at close. The twitter module deals with streaming and tweeting out the summary. Follow these steps to run the code yourself: 1. Create VM instance Check out the quickstart to create a Cloud Platform project and a Linux VM instance with Compute Engine, then SSH into it for the steps below. The predefined machine type g1-small (1 vCPU, 1.7 GB memory) seems to work well. 2. Set up auth The authentication keys for the different APIs are read from shell environment variables. Each service has different steps to obtain them. Twitter Log in to your Twitter account and create a new application. Under the Keys and Access Tokens tab for your app you'll find the Consumer Key and Consumer Secret. Export both to environment variables: export TWITTER_CONSUMER_KEY="<YOUR_CONSUMER_KEY>" export TWITTER_CONSUMER_SECRET="<YOUR_CONSUMER_SECRET>" If you want the tweets to come from the same account that owns the application, simply use the Access Token and Access Token Secret on the same page. If you want to tweet from a different account, follow the steps to obtain an access token. Then export both to environment variables: export TWITTER_ACCESS_TOKEN="<YOUR_ACCESS_TOKEN>" export TWITTER_ACCESS_TOKEN_SECRET="<YOUR_ACCESS_TOKEN_SECRET>" Google Follow the Google Application Default Credentials instructions to create, download, and export a service account key. export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials-file.json" You also need to enable the Cloud Natural Language API for your Google Cloud Platform project. TradeKing (ALLY) Log in to your TradeKing (ALLY account and create a new application. Behind the Details button for your application you'll find the Consumer Key, Consumer Secret, OAuth (Access) Token, and Oauth (Access) Token Secret. Export them all to environment variables: export TRADEKING_CONSUMER_KEY="<YOUR_CONSUMER_KEY>" export TRADEKING_CONSUMER_SECRET="<YOUR_CONSUMER_SECRET>" export TRADEKING_ACCESS_TOKEN="<YOUR_ACCESS_TOKEN>" export TRADEKING_ACCESS_TOKEN_SECRET="<YOUR_ACCESS_TOKEN_SECRET>" Also export your TradeKing (ALLY) account number, which you'll find under My Accounts: export TRADEKING_ACCOUNT_NUMBER="<YOUR_ACCOUNT_NUMBER>" 3. Install dependencies There are a few library dependencies, which you can install using pip: $ pip install -r requirements.txt 4. Run the tests Verify that everything is working as intended by running the tests with pytest using this command: $ export USE_REAL_MONEY=NO && pytest *.py --verbose 5. Run the benchmark The benchmark report shows how the current implementation of the analysis and trading algorithms would have performed against historical data. You can run it again to benchmark any changes you may have made: $ ./benchmark.py > benchmark.md 6. Start the bot Enable real orders that use your money: $ export USE_REAL_MONEY=YES Have the code start running in the background with this command: $ nohup ./main.py & License Archimedes (edits under Invacio) Max Braun Frame under Max Braun, licence under Apache V2 License. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The application is a cloud service that provides the functionality of performing sentiment analysis on stock market and financial data. The application can be hosted on Google App Engine and makes use of many of the GAE services like Search Service, MemCache, DataStore etc. Given the name of a company, data from various sources like Twitter, Facebook Graph, Google News, Google Finance etc is aggregated. For each source, different models have been pretrained using some prior data. Using different models provided us with a chance to utilize different Machine Learning methodologies based on the type of data from each source. The various techniques that we have built and tested on are :Naive Bayes, Multinomial and Bernoulli text representations, KNN.

LSTM based stock prediction model which also uses twitter data for sentiment analysis

Uses Twitter API, TensorFlow, and Natural Language Processing to predict the performance of a given stock as well as perform sentiment analysis of the stock using Twitter data.

Project on engagement and stock price analysis of CEOs on Twitter. Extracted the data from Twitter API and Yahoo Finance and implemented sentiment analyzer, topic modeling (LDA), stock price regression and engagement analysis to determine the factors that make a CEO influential

Analysis program written in Python that consolidates 3 pillars of investment analysis; sentiment, technical and fundamental. This script takes a stock ticker, performs a discounted cash flow valuation, identifies support and resistance levels alongside entry and exit points. This leads to a final sentiment vs price analysis conducted with data from twitter.

Daily Stock Price Movement Prediction based on Off-market Twitter Data on Daily Basis (Tweepy, Snscrape, NLP, Vader Sentiment Analysis, ML models / Top Tech Companies)

Twitter is an online social networking service with over 300 million monthly active users. This enormous amount of data available on social media platforms can be extracted and analyzed for various purposes. In this paper, we aim to investigate the relationship between sentiment analysis of Twitter data and stock market prices for five companies (Walmart, ExxonMobil, Apple, Berkshire Hathaway Inc., and Amazon) by scraping the Tweets extracted from Twitter based on company hashtags and using the twitter intelligence tool – twint. Sentiment analysis is applied to the extracted tweets and a correlation is analyzed between stock market movements of a company and sentiments in tweets. Elaborately, news and tweets in social media about a company would encourage decision of people to invest or not in the stocks of that company and as a result, the stock price of that company would increase or fall. At the end of the paper, it is shown that a none or very weak correlation exists between the rise and fall in stock prices with the public sentiments in tweets

Sentiment analysis of Twitter feeds and statistical analysis of stock market data

Explore Twitter's stock performance with insights into daily price changes, trading volumes, and average trends. Empower your investment decisions with comprehensive historical data analysis.

In today's financial market,news sentiment plays a crucial role in shaping investor behavior and influencing stock prices. By analyzing the sentiment behind stock-related news articles, investors can gain valuable insights to make informed trading decisions.We have performed sentiment analysis of the twitter data based on a whole day to analyse it.

No description available

StockInsight predicts the stock prices using Neural Networks. It uses Elasticsearch, Twitter data, News Headlines and sentiment analysis to find the effect of emotions on the stock price. How much do emotions on twitter and news headlines affect stock price? What could be the price of stock within next 30 minutes?

Analyze impact of sentiment made in Twitter for Tesla's stock price using NLP

A data-driven algorithmic trading strategy that leverages Twitter sentiment analysis to construct and rebalance a stock portfolio. This project aims to outperform traditional market benchmarks by prioritizing stocks with strong positive social media engagement.

Data analysis of stock and twitter data combining multiple technologies

Data Science project to predict stock close price based on macroeconomic news headlines using twitter sentiment analysis.

Sentiment-driven stock market analysis using Yahoo Finance, Twitter, Reddit, and NLP. CMSC320 Data Science Final Project.

This uses NLP sentiment analysis to analyze the Twitter data and the behaviour of stock prices particularly for Blizzard and CodeProjekt Red.

An AI-powered stock price prediction tool built using Streamlit, LSTM neural networks, and sentiment analysis from news, Twitter, and Google Trends data.

An investment strategy using Twitter sentiment analysis to guide trading decisions. The project processes live tweets, analyzes sentiment trends with NLP, and integrates them with stock data to predict performance. Features data preprocessing, machine learning models, and a backtesting framework for evaluating profitability.

hadoop-projects IBM stock project Get IBM stock dataset Clean the dataset Load dataset on the HDFS Build MapReduce program Process/ Analyse result Hadoop set up Run single node Hadoop cluster /usr/local/Celler/hadoop Check : https://www.slideshare.net/SunilkumarMohanty3/install-apache-hadoop-on-mac-os-sierra-76275019 http://zhongyaonan.com/hadoop-tutorial/setting-up-hadoop-2-6-on-mac-osx-yosemite.html Go to :http://localhost:50070/dfshealth.html#tab-overview Start : hstart Hadoop command: hadoop fs -ls hadoop fs -mkdir /hbp Upload a file in HDFS hadoop fs -put <localsrc> ... <HDFS_dest_Path> go to : http://localhost:50070/explorer.html#/hbp/ibm-stock Dataset head date - opening stock quote - high - low - traded volume - closing price Clean dataset with command : awk,sed,grep Run the program Copy jar to Hadoop Run the program on Hadoop system: hadoop jar /hbp/ibm-stock/ibm-stock-1.0-SNAPSHOT.jar /hbp/ibm-stock/ibm-stock.csv /hbp/ibm-stock/output Check output dir : hadoop fs -ls /hbp/ibm-stock/output Copy file from HDFS to local file system : hadoop fs -get /hpb/ibm-stock/output/part-r-00000 home/Users/hien/results.csv Check head home/Users/hien/results.csv Customer Analysis Collect data Customer master data : MySQL Logs : text file Twitter feeds : JSON Load data from data sources in HDFS Mug data Create table in Hive to store data in format Query and join tables Export data Set up stack: Hortonwork data platform HDP Install HDP sandbox: HDP 2.3 HDP : hive, squoop , Fraud Detection system Clean dataset Create model Using: Spark and Hadoop Problem: predict payment transaction is suspect Build model : Find relevant field: Apache Spark 2 Spark ecosystem : Spark core Spark streaming Spark SQL MLlib GraphX Spark-R Apache Spark component: + navigate to : localhost:4040 run spark-shell : $SPARK_HOME/bin/spark-shell Word count Create pairRDD : valpairRDD=stringRdd.map( s => (s,1)) Run reducebykey to count the occurency of each word : alwordCountRDD=pairRDD.reduceByKey((x,y) =>x+y) Run the collect to see the result : valwordCountList=wordCountRDD.collect Find the sum of integers Create RDD of even number from integers : valintRDD = sc.parallelize(Array(1,4,5,6,7,10,15)) Filter even numbers from RDD : valevenNumbersRDD=intRDD.filter(i => (i%2==0)) Sum the even numbers from RDD : val sum =evenNumbersRDD.sum Count the number of words in file : Read txt file : cat people.txt Read file from Apache Spark shell : val file=sc.textFile("/usr/local/spark/examples/src/main/resources/people.txt") Flaten the file, prcess and split , with each word : valflattenFile = file.flatMap(s =>s.split(", ")) Check the content of RDD : flattenFile.collect Count all words from RDD : val count = flattenFile.count Working with Data and Storage + Chua hoc 4 (RDD transformation),