Popularity of Programming Languages for Data Science

Below is a link to a study I found on kdnuggets.com. This is a useful website if you are interested in data science. The study is from October, 2016 so is fairly current. It is based on data from indeed.com as my earlier post was. It shows once again that Python, Java and R are the most popular programming languages for data science. Keep in mind , however, that R is a domain specific language for statistical computing. Unfortunately this study did not include commercial software, only open source. Thus SAS, etc. are not included.

KDNuggets - most popular languages for data science

Project Libre

We are preparing a new Project course based on Project Libre. This is free, open-source project management software that can read and write Microsoft Project files. Our course will parallel our Microsoft Project courses as much as possible. I will try to post our slides and/or comparison slides as we go along. This should prove interesting to those interested to see how compatible they are. Click the image below to see an expanded version on box.com. It shows the basic Gantt Chart view of Project Libre compared to Microsoft Project.

Using Microsoft Project for Agile Projects

You CAN use use Microsoft Project for Agile projects. While it may not be the ideal tool it can be done if you use custom fields and filters, etc. Below is a link from mpug.com that shows how. Lynda.com also has a video by Bonnie Biafore that shows how. Again making use of custom fields.

Agile Project Tutorial on mpug.com

Anna Forss has a different take. Check out the link below. She works with the calendars and options.

Anna Forss Agile with Microsoft Project

Please note this is not an argument for or against the Agile methodology versus waterfall. I am just pointing out a couple of useful links. Most of our Microsoft Project clients have been in the construction industry which uses waterfall. On the other hand we see little interest in waterfall from software development, especially web, groups. They much prefer Agile methodologies, with Scrum seeming the most popular. In my opinion the choice of a methodology depends on the project and the domain. There is no one answer as to which is "best".

Popularity of some programming languages/IT skills

What follows is a little survey I did on indeed.com on 4/28/17. I started off just looking for the popularity of the R language for data science but wound up broadening the search terms. I limited the search to my home state of California. It is not in  any way a scientific survey but it did prove interesting as far as it went. R ranked pretty high. Right behind Python and ahead of C++. C and SQL were the top two languages. Java was 3rd. We already offer a short coding class in Java and are developing one in Python. Perhaps we will do R next. A couple other observations are that the open languages are firmly in the lead. SAS had the most mentions of the commercial statistical packages.

I often say that SQL is a foundation skill for data analysis. This survey seems to bear that out. To that I would add Python and/or R for those interested in a data science career.

skill/term	indeed mentions
microsoft excel	28,420
c	23,962
sql	15,825
linux	14,165
java	13,769
python	12,835
r	9,826
c++	8,771
javascript	8,619
microsoft access	8,199
bi	7,778
hadoop	4,461
ruby	3,715
sas	3,602
mysql	3,601
tableau	3,413
business intelligence	3,390
spark	3,284
rails	2,677
epic	2,618
matlab	2,141
etl	1,726
scala	1,685
django	668
oracle	542
spss	538
minitab	273
stata	236
cobol	135
fortran	132
medisoft	20
statistica	7

Students are often confused by the MPx and xPM Extreme Project management approaches. The following two links may shed some light on this. The first is a link to an article by Wysocki on the Extreme and Emertxe Project Management Models. These are two models used when solution paths are unclear. The second is a good blog post which summarizes the 5 life cycle approaches. It is based on experiences in the construction industry. As such the Extreme models are not used but the article does discuss them including the xPM and MPx approaches.

xPM and MPx Project Management Models

Blog Summary of 5 Project Management Life Cycles

La Palma STEM Symposium

Here are the slides for a presentation I gave on Big Data and Data Science at the La Palma, California STEM Symposium. This is a booming field for people interested in data analysis.

La Palma STEM Symposium

FYI - I will be presenting a brief talk on Big Data and Data Science at this symposium. It is mainly aimed at Community college students and the local community. See link below.

La Palma STEM Symposium

---Dan

jGrasp IDE

We used jGRASP in a recent Introduction to Coding with Java class. It is not a production type IDE such as Netbeans or Eclipse but we have found it a good tool for introductory classes aimed more at casual users, beginning computer science students, etc. It has a lot of interesting features such as generation of UML diagrams, an interactions pane like the Visual Basic immediate window, object viewers, side by side columns for editing multiple class files, etc. It is lightweight and very quick to load. It will also handle Python, C, etc. But we have not yet explored that in any depth. Although we have used it for simple Python scripts. On the downside it does not have a "smart editor" with intellisense type capabilities beyond simple syntax highlighting.

Instantiate a subclass in Java

The following code actually instantiates the project manager subclass of the employee detail superclass. It inputs the manager's id, name, etc. and then instantiates a project manager passing the variables just input to the subclass constructor which in turn invokes the superclass constructor.

This concludes the initial version of the programs and screenshots for the JVA101 Introduction to coding class. There is another lesson dealing with error handling (exceptions) which we may add in the future.

 Click to view expanded image on box.com

Java subclasses

A class can inherit from a parent or superclass. This is done in Java with the extends keyword. The following snippets create a ProjectManager subclass from the EmployeeDetails superclass. Note that the subclass invokes code from the superclass using the super keyword. It also overrides the compute salary method.

Click image to view expanded image on box.com

Using Java Constructors

Here is the code that instantiates an object form the prior EmployeeDetail2 class. It inputs an employee id. If the id is 0 it invokes the no argument constructor, otherwise it inputs the name and salary and invokes the constructor that takes arguments. On another note I forgot to mention that having multiple constructor with the same name is called overloading. We will see this same term later with methods.

Java constructor

This is the Java EmployeeDetails class with a constructor, Constructors are used to initialize the instance variables in an object when an object is instantiated form a class. If no arguments are passed in to the constructor then a no args constructor is applied. If the programmer does not supply a no args constructor then Java initializes the instance variables to default settings defined by their data type. You can have multiple constructors with the same name but their signatures must differ in the number of arguments passed.

Click on the image for an expanded view on box.com.

Java Object

Here is the code that instantiates an employee detail object and uses it to compute the employee's net salary.  Click on the image for an expanded view.

Java Classes and Objects

Here is the code for creating a Java class. This will create an employee detail class. Note the class has a private method which computes the allowance. This is used by the method which computes and returns the salary. There are also two methods to set and retrieve instance variables. These are sometimes called setters and getters.

In a later post we will instantiate an object from this class. We will resume our parallel Python files when we return to working on the Python course. Click on the image for an expanded view of the Java class on box.com.

Builtins

Just finished the section on using builtins. I fudged a little as in Python I used string slicing instead of a substring function as in Java. This is kind of a cool feature in Python array handling. For example to pickup the first character in a person's name in Java I used

      String empEMail = empFName.substring(0,1);

Whereas in Python the code was

       empEMail = empFName[:1]

Which means everything before the character at index 1.

Click on the image below to see an expanded view of the three columns on box.com with pseudocode, Java and Python.

JAV101 Java Course Link

Here is the link to our Java course. I should have posted it before,

JVA101 Introduction to Programming using Java

Python Course Development

As some of you know we are developing an Introduction to Programming with Python course. It will more or less parallel our similar class based on Java. Below is the outline as we have it so far. We just finished topic 3A and will be starting 3B shortly. Our first version is being built using Python 3.5.

Lesson 1: Introduction to Programming

Topic 1A: Overview of Programming

Topic 1B: Introduction to the Software Development Life Cycle

Lesson 2: Creating Simple Programs

Topic 2A: Work with Variables

Topic 2B: Work with Operators

Topic 2C: Control Program Execution

Topic 2D: Work with Lists

Lesson 3: Creating Programs Using Functions

Topic 3A: Create Functions

==>Topic 3B: Work with Built-ins

Lesson 4: Implementing the Object-Oriented Methodology

Topic 4A: Create a Class

Topic 4B: Create an Object

Topic 4C: Create a Constructor

Topic 4D: Create a Subclass

Lesson 5: Handling Programming Errors

Topic 5A: Fix Syntax Errors

Topic 5B: Fix Logical Errors

Topic 5C: Fix Runtime Errors

Java Programming Courseware

Below is the outline for our Introduction to Programming with Java Course (JVA101). This is a one day class designed to give the student an introduction to programming concepts using the Java language as the teaching vehicle. The courseware is from Logical Operations. Each student in our classes will receive one of these manuals.

Lesson 1: Introduction to Programming

Topic 1A: Overview of Programming

Topic 1B: Introduction to the Software Development Life Cycle

Lesson 2: Creating Simple Programs

Topic 2A: Work with Variables

Topic 2B: Work with Operators

Topic 2C: Control Program Execution

Topic 2D: Work with Arrays

Lesson 3: Creating Programs Using Methods

Topic 3A: Create Methods

Topic 3B: Work with Built-in Methods

Lesson 4: Implementing the Object-Oriented Methodology

Topic 4A: Create a Class

Topic 4B: Create an Object

Topic 4C: Create a Constructor

Topic 4D: Create a Subclass

Lesson 5: Handling Programming Errors

Topic 5A: Fix Syntax Errors

Topic 5B: Fix Logical Errors

Topic 5C: Fix Runtime Errors

The following is the link to the manual in their course catalog.

Logical Operations Courseware

Links to IT Certifications

Here is an excellent link I received form the University that lists many certifications available. It is a good summary so I thought I would pass it along.

Tom's IT Pro Certification Links

Methods in Java and functions in Python

We were bound to hit a few bumps in the road in developing our Python class using the Java manual as an outline. We are at the start of chapter three. There are five chapters in all. At this point the Java manual discusses using methods. Since we have not yet gotten to classes (which aren't formally discussed until chapter 4) we are substituting functions in our Python material at this point. We can discuss methods later when we get to chapter four. Below is the thumbnail of the three column (pseudocode, java, python) screenshot on box.com.

Changing Timescale in Project Libre

We are going to develop a new Project course based on the open source Project Libre. As with Python we did not find any suitable courseware manuals so we are developing our own. I downloaded Libre the other day and opened up one of our little Microsoft Project exercise files. As advertised it opened perfectly in Project Libre. Right away I noticed little differences: no auto-schedule/manual schedule, no dropdowns for duration, no question mark in the duration column, etc. The one that that was a little annoying was the lack of an obvious way to change the timescale in the Gantt chart. After a little searching on the internet I found one way is to use the zoom in and out commands on the resource tab. By the way, notice I said tab. Project Libre uses the same now familiar Icon Bar user interface that Microsoft Office uses. Below is a screenshot for changing the timescale.

Python Tutorial

Below is the link to a Python tutorial we have found useful. There are others we have used for matplotlib and numpy which we can post later.

https://www.tutorialspoint.com/python/

Java arrays and Python lists

We used lists for our Python class to correspond to Java arrays. Python does have arrays but lists are actually closer to Java arrays. Our examples show both single dimensional and two dimensional arrays and lists. We processed the arrays using for loops to compute sums. Again, note that the Python syntax iterates over a range. Click on the image below to see an expanded image on box.com.

matplotlib tutorial

We are using a lot of tutorials to help us prepare our Python materials. I'll post some here in case others are interested. Below is a simple starter one for matplotlib, the Python plotting library.

PythonProgramming.net

Anaconda

Recently installed the Anaconda Python IDE. I was pleasantly surprised to find that it does, indeed, install many of the libraries such as pandas for data manipulation, numpy for arrays and matplotlib for plotting.  These are available from the regular command line as well iPython and Jupyter.The install went smoothly and only took a few minutes on my Windows 10 computer. The distribution was about 420 MB. This is what it installed. More about these in later posts.

Loops in Java and Python

Below are the basic looping statements in Java and Python. Java has three: while, do...while and for, Python has two: while and for. But not do...while, although there are ways to accomplish the same thing. Also Python has an interesting approach to the for statement, sometimes called a for each. It iterates over a sequence selecting each element in turn. In the example below if counter is 15 it will iterate over i from 1 through 15.

   for i in range(1,counter+1):
      sum = sum + i;

On another note I decided I would I put the code for the Python scripts in a function and then called the function. This is more similar to Java which requires the existence of a class with a main() method. See the example below. It basically checks to see if the script being run is the main program, not an import.

def Loops():
   counter = 10
   i = 1
   while i <= counter:
      print("i = {}".format(i))
      i = i + 1

if __name__ == "__main__":

   Loops()

Click on the image below to see the code for the various loops.

Switch statement in Java and equivalent in Python

Here is another Java/Python comparison. This one for the switch statement in Java. I used elif in Python to keep it simple for beginners. Click on the image to see an expanded version on box.com.

Java and Python if statements comparison

Here is a comparison of Java and Python for the if statement. We are preparing our Python scripts to parallel (at least as much as possible) the Java class from Logical Operations. So we may as well publish them for our reference as well as anyone else who might be interested. It also includes the pseudocode. See screenshots on box.com.

https://app.box.com/s/ljmqkh0hgr6g5kc6gog8ljex124b4pcc

jGrasp and Python

In the course of working on our new Python class we discovered you can run Python as well as Java from jGrasp. While I am not recommending this one way or another it did have a nice side effect. You can develop and test Java programs in parallel with their Python versions right in the same session. By the time we are done developing our new Python training material we should have a lot of side-by-side comparisons of the code for the same tasks. So far it has been running about 1/2 the code in Python vs. Java. While that is a substantial reduction it is not as dramatic as I thought it would be. I may do another parallel stream in pseudocode, mostly for use in college classes.