Course Goals

• Learn visual and computation techniques and tools, for typical data types
  • Learn how to complement each kind of methods
  • Gain a breadth of knowledge
• Work on real datasets and problems
• Learn practical know-how (useful for jobs, research) through significant hands-on programming assignments

Announcements and Discussion

The fastest way to get help with homework assignments is to post your questions on Ed Discussion. That way, you can get help from our TAs and instructor can help, as well as your peers.

If you prefer that your question is addressed to only our TAs and the instructor, you can use the private post feature (i.e., check the "Individual Students(s) / Instructors(s)" radio box).

While we welcome everyone to share their experiences in tackling issues and helping each other out, you must not post your answers, as that may affect the learning experience of your fellow classmates.

For special cases such as failed submissions due to system errors, missing grades, failed file uploads, emergencies that prevent you from submitting, or personal issues, you can contact the staff using a private Ed Discussion post.

Course Staff & Office Hours

TAs plan to hold office hours starting week 2, except on Georgia Tech holidays (e.g., thanksgiving, MLK day, spring break). Each office hour session will be run by at least one TA, and is 1 hour long. See GT’s academic calendar for the full list of holidays (https://registrar.gatech.edu/calendar). We will spread the office hours across weekdays, and across time of the day. We will announce the office hour times.

We will hold office hours via Ed Discussion threads, where the TA running the office hour will be responding to comments within the thread live. We will share more information in our office hour announcement post on Ed Discussion.

Information about Polo's weekly office hour will be shared on Ed Discussion too. All questions are welcome, except homework assignment related questions, which are best addressed via Ed Discussion and TA office hours.

Please note that you are always welcome to ask questions on Ed Discussion. Office hours supplement Ed Discussion, and do not replace it.

This course can be very tough for many!

The amounts of time students spend on this class greatly vary, based on their backgrounds, and what they may already know. Some former students told us they spent about 40-60 hours on each homework assignment (we have 4 big assignments, and no exams), and some reported much less. For example, for the homework assignment about D3 visualization programming, students who are completely new to javascript, css, and html likely will spend significantly more time than their peers who have already tried them before. Some former students who do not have a computer science background found the homework assignments challenging, would take significant time and effort, but were rewarding, fun, and "do-able."

Students have at least 3 weeks to complete each homework assignment. In the past, some students have waited until the last week to begin, and could not finish. It is critical to plan ahead and prepare for the significant time required to complete the homework assignments.

Some programming assignments involve high-level languages or scripting (e.g., Python, Java, SQL etc.). Some assignments involve web programming and D3 (e.g., Javascript, CSS, HTML). For example, an assignment on Hadoop and Spark may require you to learn some basic Java and Scala quickly, which should not be too challenging if you already know another high-level language like Python or C++. It is unlikely that you know all of the tools/skills needed in every programing task, so you are expected to learn many of them on the fly.

Basic linear algebra, probability and statistics knowledge is also expected.

Minimum Computer Requirements

• 8GB RAM (16GB recommended)
• 512GB disk (SSD recommended). Some assignments use data files that are more than a few GBs, and some uses virtual machines that can easily take up more than tens of GBs. It is typical for some project teams to use large datasets that are more than a few or tens of GBs.
• Dual-core Core i5 (8th generation or better recommended), or Mac with M1 processor or better

Accessing Course Materials Outside of US

Homework

• [10%] HW1: Collecting & visualizing data, SQLite, D3 warmup, OpenRefine
• [15%] HW2: D3 Graphs and Visualization (Canvas may show a weight greater than 15% because historically HW2 offers bonus points.)
• [15%] HW3: Spark, Docker, DataBricks, Cloud Services (AWS, Azure, GCP)
• [10%] HW4: PageRank, Random Forest, Scikit-Learn

Project

Distance Learning Students

1. [Not applicable this semester] Present in class without 3-day lag; or
2. Submit a video presentation with 3-day lag (e.g., screen capture)

Grading Policy

1. Proposal (7.5% of course grade)
2. Proposal presentation (5%) (video recording)
3. Progress report (5%)
4. Final poster presentation (7.5%) (video recording)
5. Final report (25%)

Plagiarism, Collaboration Policy, and Student Honor Code

• All course participants (myself, teaching assistants, and learners) are expected to know and abide by the Georgia Tech Academic Honor Code.
• Ethical behavior is extremely important in all facets of life.

1. Plagiarism is a serious offense. You are responsible for completing your own work. You are not allowed to copy and paste, or paraphrase, or submit materials created or published by others, as if you created the materials. All materials submitted must be your own.
2. You may discuss high-level ideas with other students at the "whiteboard" level (e.g., how cross validation works, use hashmap instead of array) and review any relevant materials online. However, each student must write up and submit his or her own answers.
3. You must not put your code on public domain (e.g., public GitHub), because a (future) student could copy your code. That student obviously violates the honor code, and you may also be implicated.
4. All incidents of suspected dishonesty, plagiarism, or violations of the Georgia Tech Honor Code will be subject to the institute’s Academic Integrity procedures (e.g., reported to and directly handled by the Office of Student Integrity (OSI)). Consequences can be severe, e.g., academic probation or dismissal, grade penalties, a 0 grade for assignments concerned, and prohibition from withdrawing from the class.

Late Policy and Due Dates

1. All homework and project deliverables are due at the times shown in the Course Schedule. These times are subject to change so please check back often. Convert the times to your local times using a Time Zone Converter.
2. Every homework assignment deliverable and every project deliverable comes with a generous 48-hour "grace period".
   1. The grace period allows students to address unexpected, minor issues without facing penalties.
   2. Students can use the grace period without asking.
   3. Course staff support is not guaranteed during the grace period.
   4. Submissions during the grace period will display as "late" but will not incur a penalty.
   5. If a student decides to make a submission during the grace period, they are responsible for all issues associated with that submission (e.g., any Gradescope errors, including those triggered by student’s syntax errors that crash Gradescope). Thus, a submisson that does not complete by the end of the grace period will receive 0 point.
3. For Canvas, a submission made during the grace period will be marked as "late", without point deduction. Canvas automatically appends a "version number" to files that you re-submit. You do not need to worry about these version numbers, and there is no need to delete old submissions. We will only grade the most recent submission.
4. For Gradescope, a submission made during the grace period will be marked as "late", without point deduction. Each submission and its score will be recorded and saved by Gradescope. By default, Gradescope uses your last submission for grading. To use a different submission, you must "activate" it prior to the end of the grace period (click “Submission History” button at bottom toolbar, then “Activate”).
5. We will not consider late submission of any missing parts of a deliverable. To make sure you have submitted everything, download your submitted files to double check. If your submitting large files, you are responsible for making sure they get uploaded to the system in time. You have 48 hours to verify your submissions!
6. No penalties for medical reasons or emergencies. And should they arise, you must contact the Dean of Students office. Any sensitive information, doctor's notes, medical documentation, explanation of emergencies, etc. should be submitted to the Dean’s office. After their office receives the information, they will notify us on your behalf. Do not share any sensitive information with us.

Timing Policy

• The course videos follow a logical sequence that includes knowledge-building and experience-building (assignments).
• Assignments should be completed by their due dates, in order for timely peer assessment. Peer assessments should also be completed by their due dates, to give timely feedback.
• You will have access to the course content for the scheduled duration of the course.

Attendance and COVID-19 Policy

• This semester, this course runs as an in-person class. Class attendance is not mandatory, except for project proposal presentation days (see course schedule for the exact dates).
• To address some of the requests from students for flexibility, especially if they are not able to come to class due to illness or quarantine, pre-recorded lecture videos from the OMS section of this course are available in the course schedule table above, and can be downloaded in the Media Gallery on Canvas.
• Back to Class: Covid-19 Prevention Tips
• Covid-19 Information and Resources

Netiquette

• Netiquette refers to etiquette that is used when communicating on the Internet. Review the Ground Rules for Online Discussions. When you are communicating via email, discussion forums or synchronously (real-time), please use correct spelling, punctuation and grammar consistent with the academic environment and scholarship.
• We expect all participants (learners, faculty, teaching assistants, staff) to interact respectfully. Learners who do not adhere to this guideline may be removed from the course.

Dataset Ideas (may need API, or scraping)

• DiffusionDB: A large-scale text-to-image prompt gallery dataset based on Stable Diffusion
• FiveThirtyEight data
• MalNet graphs (300GB) and MalNet images (80GB): 1262024 unique function call graphs across 696 families and 47 types of malware.
• Google Dataset Search
• Google Dataset Search
• Google public datasets. Thanks Revant!
• Kaggle public datasets
• Data science competitions for Africa. Thanks Krupa!
• Awesome Public Datasets. Thanks Marcel Gwerder!
• NYC Taxi data for 2013 (suggested by Chris Wong). 2013 Trip Data (11.0GB). 2013 Fare Data (7.7GB). Visualization for a days trip. Thanks Jitesh.
• Large datasets publicly available. Thanks Gopi!
• Yahoo WebScope
• Data.gov: U.S. Government's open data
• IPEDS data: Postsecondary education data from National Centre for Education Statistics
• Bureau of Labor Statistics data
• Uber pickups in NYU
• Freebase
• Yelp
• Microsoft Academic Graph
• Numerous APIs from Google (e.g., Maps, Freebase, YouTube, etc.)
• Zillow: real estate listing site
• Numerous graph datasets (large and small): SNAP, Konect
• Movies data: IMDB
• List of lists of datasets for recommendations.
  Thanks Jon!
• Million Song Dataset.
  It contains not only the basic information of songs (artist, genre, year, length etc), but also some musical features(like tempo, pitch, key, brightness).
  Thanks Minwei!
• Dataset about soccer games, players, clubs.
  No API, but easy to scrape.
  For a soccer player: transfer history, performance, nationality, birth date, etc.
  For a soccer club: performance, squad, etc.
  Thanks Ding!
• The Free 'Big Data' Sources Everyone Should Know
• Quandl - a dataset search engine for time-series data.
  Thanks Henry!
• UCI also has a collection of links to various datasets sorted for various tasks (Classification, Regression, etc)
  Thanks Vinodh!
• Amazon AWS Public Data Sets (Thanks Jonathan!)
• KDD Cup: annual competition in data mining, like Kaggle
• Academic domain: Microsoft Academic Search, DBLP
• Retrosheet: MLB statistics (Game/Play logs)
• Classification datasets
  Thanks Amish!
• Various geophysical datasets for the oceans (magnetism, gravity, seismology, etc).
  Thanks Ryan!
• Social trends (Thanks Jonathan!)
• Beer data. (Thanks Jonathan!). Website offline :( . Older version at web.archive.org
• Academic torrents (terabytes) (Thanks Vaibhav!)
• Article Search API from the New York Times (all the way back to 1851!) (Thanks Guido!)
• (Kayak: flight, hotel, car, etc.)
• Data Science Initiative - Microsoft Research has various datasets and access to tools that can aid in data science research

Resources

Office of Disability Services

The Office of Disability Services offers accommodations for students with disabilities. Please contact the office should you need help.

Support Services

Graduate Student Resources and academic and personal support services : Office of the Dean of Students, Counseling Center, Health Serivces, Women's Resource Center, LGBTQIA Resource Center, Veteran's Resource Center, Georgia Tech Police.

Recommended Reading

All content and course materials can be accessed online. There is no textbook for this course.

All Georgia Tech students have FREE access to https://www.oreilly.com, where you can find a huge number of highly rated and classic books (e.g., the "animal" books) from O'Reilly and Pearson covering a wide variety of computer science topics, including some of those listed below. Just log in with your official GT email address, e.g., jdoe3@gatech.edu.

Software engineering; become a better programmer and developer

• Debugging
• Clean Code
• Refactoring
• Design Patterns: Elements of Reusable Object-Oriented Software
• The Pragmatic Programmer: From Journeyman to Master

D3 Visualization; Javascript

• Interactive Data Visualization for the Web, 2nd Edition
• JavaScript: The Good Parts

Big Data

• Hadoop: The Definitive Guide, 2nd Edition
• HBase: The Definitive Guide, 2nd Edition

Python

• Python Bootcamp, for campus MS Analytics students. By Chris Simpkins

Data science, machine learning, data mining

• Data Science for Business by Foster Provost and Tom Fawcett
• The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani, and Jerome Friedman

Visualization

• A nice D3 tutorial
• The Wall Street Journal Guide to Information Graphics: The Dos and Don'ts of Presenting Data, Facts, and Figures by Dona Wong
• Information Dashboard Design: Displaying Data for At-a-Glance Monitoring by Stephen Few
• Show Me the Numbers: Designing Tables and Graphs to Enlighten by Stephen Few

SQL

• Video courses and lectures: (1) Coursera, (2) lynda.gatech.edu and search for SQL courses
• Interactive SQL tutorials
• Books: (1) SQL Cookbook (recipes to solve specific problems), Visual QuickStart Guide (succinct topic-by-topic), SQL Pocket Guide (covers syntax variations of MySQL, Oracle, etc.)
• Introductory tutorial

Probability

• FREE probability book, by Prof. Guy Lebanon. (From Amazon.)

Human Computation

• Human Computation book by Edith Law and Luis von Ahn

How to manage multiple versions of Python packages?

Prerequisites

Additional formal prerequisites for CSE 6242

None, but you should have taken courses similar to those listed in the next section, at Georgia Tech or at another school.

If you are an Analytics (OMS or campus) degree student, you should first take CSE 6040 and do very well in it; if necessary, please also first take CS 1301.

Additional formal prerequisites for CX 4242

Course offerings and Registration

Auditing & Pass/Fail

Previous offerings

Acknowledgment & Related Classes

We thank Intel's support in curriculum development for the memory mapping module (scaling up algorithms with virtual memory).

We thank Amazon Educate for providing free cloud credit for Amazon Web Services. We are excited to be an AWS partner university and part of AWS Educate's private beta.

We thank Google Cloud Platformfor providing free cloud credit for GCP. We are excited to be a GCP partner education.

We thank Microsoft Azure's special grant for providing free cloud credit.

We thank Tableau for Teaching program's data visualization software.

• Prof. John Stasko - Information Visualization - Fall 2012
• Prof. Jeff Heer - Research Topics in Interactive Data Analysis - Spring 2011
• Prof. Christos Faloutsos - Multimedia Databases and Data Mining - Fall 2012