Predict the Future with Time Series Forecasting and Deep Neural Networks

This collection of resources is a companion to the Predict the Future with Time Series Forecasting and Deep Neural Networks bootcamp.

AWS Resources and References

Start here to learn more about Amazon Sagemaker to Build, Train, and Deploy Machine Learning Models at Scale. Then check out more resources below.

• Amazon Sagemaker Developer Guide summarizes the service and frameworks https://docs.aws.amazon.com/sagemaker/index.html#lang/en_us
• Access more than 30 Amazon Sagemaker Notebooks at https://github.com/awslabs/amazon-sagemaker-examples
• New (circa 27Mar2019)– AWS Deep Learning Containers https://aws.amazon.com/blogs/aws/new-aws-deep-learning-containers/
• Amazon QuickSight Announces General Availability of ML Insights (circa Mar2019) https://aws.amazon.com/blogs/big-data/amazon-quicksight-announces-general-availability-of-ml-insights/
• Overview of containers for Amazon SageMaker https://sagemaker-workshop.com/custom/containers.html
• Find prebuilt ML and DL models on the AWS Marketplace https://aws.amazon.com/marketplace/solutions/machinelearning/

Deep Learning Resources

• Dive into Deep Learning, An interactive deep learning book with code, math, and discussions that is authored and maintained by Amazon AI specialists. https://d2l.ai/. Think textbook and Jupyter notebooks integrated in a learning portal.
• MIT Online Deep Learning book / labs https://www.deeplearningbook.org/ This is a good read for anyone wanting a refresher or first broad look at Deep Learning.

Time Series Forecasting with Neural Networks (module 1)

• For an introduction to Recurrent Neural Networks (RNNs) start with Understanding LSTM Networks, http://colah.github.io/posts/2015-08-Understanding-LSTMs/ then read The Unreasonable Effectiveness of Recurrent Neural Networks https://karpathy.github.io/2015/05/21/rnn-effectiveness/
• Blog post on developing LSTM networks in Python using the Keras deep learning library for time series forecasting. https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/
• Keras documentation on recurrent layers. https://keras.io/layers/recurrent/
• The Unreasonable Effectiveness of Recurrent Neural Networks, a blog post on RNNs by Andrej Karpathy. http://karpathy.github.io/2015/05/21/rnn-effectiveness/
• Research paper cited in Module 1 to illustrate an example of non-linear, complex dependencies. Carbonneau, Réal, Rustam Vahidov, and Kevin Laframboise. “Forecasting Supply Chain Demand Using Machine Learning Algorithms.” Machine Learning, 1652–1686. doi:10.4018/978-1-60960-818-7.ch609. https://pdfs.semanticscholar.org/85c8/66eeeaf05edf872e7d06ff1c32f7fd560757.pdf (Abstract only)
• Deutsche Börse Public Dataset with links to the datasets and alternative visualizations and machine learning examples. https://registry.opendata.aws/deutsche-boerse-pds/
  • Stock Price Movement Prediction Using The Deutsche Börse Public Dataset & Machine Learning is a docker based ML share price predictor https://github.com/Originate/dbg-pds-tensorflow-demo
  • 10 visualizations to try in Amazon QuickSight with sample data https://aws.amazon.com/blogs/big-data/10-visualizations-to-try-in-amazon-quicksight-with-sample-data/
• Efficient Backprop by Yann Le Cunn describes strategies for improving or minimizing the cost function. You can find it and many other interesting ML and DL approaches at http://yann.lecun.com/exdb/publis/index.html
• Predicting stock prices is non trivial. Here is an untested approach using GANs which looks interesting to say the least. Titled ‘Using the latest advancements in AI to predict stock market movements’ https://github.com/borisbanushev/stockpredictionai

Amazon SageMaker and DeepAR for Time Series Forecasting (module 2)

• The seminal research paper on DeepAR. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Network https://pdfs.semanticscholar.org/4eeb/e0d12aefeedf3ca85256bc8aa3b4292d47d9.pdf
• Latest publication (as of March, 2019) related to and building off of DeepAR. Deep State Space Models for Time Series Forecasting https://papers.nips.cc/paper/8004-deep-state-space-models-for-time-series-forecasting.pdf
• DeepAR Forecasting documentation https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html
• Time series forecasting with DeepAR This notebook demonstrates how to prepare a data set of time series for training DeepAR and how to use the trained model for inference. https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/deepar_synthetic/deepar_synthetic.ipynb
• SageMaker/DeepAR demo on electricity dataset This notebook complements the notebook cited above. In this notebook, a real-life use case will be considered to show you how to use DeepAR on SageMaker for predicting energy consumption. This is the notebook from which Lab 2 of this bootcamp was designed. https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/deepar_electricity/DeepAR-Electricity.ipynb

Introduction to MXNet and Gluon (module 3)

• Deep Learning – The Straight Dope A repo containing an incremental sequence of notebooks designed to teach deep learning, Apache MXNet, and the Gluon interface. https://gluon.mxnet.io/ and simoncorstonoliver/DeepLearningWithMXNetGluon forked from ThomasDelteil/DeepLearningWithMXNetGluon https://github.com/simoncorstonoliver/DeepLearningWithMXNetGluon
• Machine Learning with Time Series and Sequence Data, This repo includes instructional materials for the “Modeling timeseries and sequence data on AWS using Apache MXNet and Gluon” workshop run during the Applied ML Days, 2018. The workshop focused around how to model generic sequence data and time series data to do sequence prediction, forecasting future time series, and anomaly detection. It demonstrates how to train neural networks with Apache MXNet. https://github.com/sunilmallya/timeseries
• Pixm takes on phishing attacks with deep learning using Apache MXNet on AWS, One of two MXNet case studies highlighted in this module. https://aws.amazon.com/blogs/machine-learning/pixm-takes-on-phishing-attacks-with-deep-learning-using-apache-mxnet-on-aws/
• Curalate makes social sell with AI using Apache MXNet on AWS https://aws.amazon.com/blogs/machine-learning/curalate-makes-social-sell-with-ai-using-apache-mxnet-on-aws/

LSTNet (module 4)

• Modeling Long and Short-Term Temporal Patterns with Deep Neural Networks, Seminal paper on LSTNet. https://arxiv.org/pdf/1703.07015.pdf
• LSTNet implementation in Apache MXNet Gluon, Github repository that this module’s lab is based off of. https://github.com/safrooze/LSTNet-Gluon
• A short primer on ReLU and Softmax Activation Functions https://github.com/Kulbear/deep-learning-nano-foundation/wiki/ReLU-and-Softmax-Activation-Functions

AWS Machine Learning Resources

• AWS has now released free digital ML training (more than 45 hours) and announced the new Machine Learning Certification https://aws.amazon.com/blogs/machine-learning/amazons-own-machine-learning-university-now-available-to-all-developers/ and https://aws.amazon.com/training/learning-paths/machine-learning/
• AWS created solutions, reference architectures and quickstarts at https://aws.amazon.com/big-data/getting-started/tutorials/ . Look for the self paced labs that are accessible at run.qwiklabs.com
• There are many interesting solutions on the AWS Machine Learning blog at https://aws.amazon.com/blogs/machine-learning/
• Access hundreds of ML algorithms and models from the AWS Marketplae https://aws.amazon.com/about-aws/whats-new/2018/11/awsmarketplace-makes-it-easier-to-build-machine-learning-applications-on-amazonsagemaker/
• Which services are available in which regions? https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
• ReInvent 2018 was a huge event for ML. 13 major ML annoucements by AWS. https://www.businesswire.com/news/home/20181128005660/en/Amazon-Web-Services-Announces-13-New-Machine and https://aws.amazon.com/new/reinvent/?sc_icampaign=event_reinvent2018-recap&sc_ichannel=ha&sc_icontent=awssm-1690-default-hero&sc_iplace=hero&trk=ha_awssm-1690-default-hero
• 8 Machine Learning Algorithms explained in Human language (sort of) https://www.datakeen.co/en/8-machine-learning-algorithms-explained-in-human-language/
• AWS provided ML resources on Github. https://github.com/aws-samples/machine-learning-samples which includes useful tools and the comprehensive Amazon Sagemaker collection including algorithms https://github.com/awslabs/amazon-sagemaker-examples

Other References

• List of mathematical symbols for those who need a refresher. Useful when reading the academic papers referenced here. https://en.wikipedia.org/wiki/List_of_mathematical_symbols
• Online derivative calculator using math symbols https://www.derivative-calculator.net/
• Comparing forecasting methods. https://otexts.com/fpp2/accuracy.html/ Ultimately it’s the use of test sets that is the most conclusive validation method.

One Line List of services

Entry page for AWS Documentation https://docs.aws.amazon.com/index.html#lang/en_us 165 services as of Mar 2019 according to Andy Jassy https://www.cnbc.com/2019/02/28/amazon-cloud-ceo-we-have-a-30-billion-run-rate-in-our-early-stages.html

Continue reading articles in my Amazon Web Services series

• Data Warehousing on AWS
• Migrating to AWS
• AWS Business Essentials
• IAM Demo
• Architecting on AWS
• SysOps on AWS
• S3 Demo
• Predict the Future
• AWS Tech Essentials
• Developing on AWS
• DevOps on AWS
• Advanced Architecting on AWS
• Big Data on AWS
• AWS Deep Dive Toolbox
• Security Engineering on AWS
• Deep Learning on AWS
• AWS List of Services
• Networking on Aws
• AWS Data and Analytics
• Microsoft Immersion Day
• Adelaide Deep Racer Hints and Tips
• Deep Racer Awards
• Windows on AWS
• AWS Ask Me Anything
• Cloudwatch and Systems Manager Workshop
• Containers Immersion Day
• Redshift Immersion Day
• Innovation in Ambiguity
• AWS Contingency Planning
• AWS CLI Examples
• Migrating to Cloud in 2023
• Chaos Engineering Workshop
• Chatgpt Friend or Foe